Depth-guided video inpainting for autonomous driving

Systems and methods of video inpainting for autonomous driving are disclosed. For example, the method stitches a multiplicity of depth frames into a 3D map, where one or more objects in the depth frames have previously been removed. The method further projects the 3D map onto a first image frame to generate a corresponding depth map, where the first image frame includes a target inpainting region. For each target pixel within the target inpainting region of the first image frame, based on the corresponding depth map, the method further maps the target pixel within the target inpainting region of the first image frame to a candidate pixel in a second image frame. The method further determines a candidate color to fill the target pixel. The method further performs Poisson image editing on the first image frame to achieve color consistency at a boundary and between inside and outside of the target inpainting region of the first image frame. For each pixel in the target inpainting region of the first image frame, the method further traces the pixel into neighboring frames and replacing an original color of the pixel with an average of colors sampled from the neighboring frames.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase application under 37 U.S.C. § 371 of International Application No. PCT/CN2020/092390, filed May 26, 2020, entitled “DEPTH-GUIDED VIDEO INPAINTING FOR AUTONOMOUS DRIVING”, the disclosure of which is incorporated herein by reference.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to autonomous driving. More particularly, embodiments of the disclosure relate to systems and methods for depth-guided video inpainting for autonomous driving.

BACKGROUND

As computational power increases, multi-modality sensing has become more and more popular in recent years. In particular, in the area of autonomous driving (AD), multiple sensors are combined to overcome the drawbacks of individual ones, which can provide redundancy for safety. Nowadays, most self-driving vehicles are equipped with light detection and range (LIDAR) and cameras for both perception and mapping. Moreover, simulation systems have become essential to the development and validation of AD technologies. Instead of using computer graphics to create virtual driving scenarios, augmented real-world pictures with simulated traffic flow have been previously proposed to create photorealistic simulation images and renderings. One key component in the pipeline of such photorealistic simulation images and renderings is to remove moving agents on the road to generate clean background street images. Those kinds of data can be generated using an augmented platform and a video inpainting method based on the deep learning techniques.

Map service companies, which display street-level panoramic views in their map applications (Apps), also choose to place depth sensors in addition to image sensors on their capture vehicles. Due to privacy protection, however, those street view images have to be post-processed to blur human faces and vehicle license plates before posted for public access. As such, is a strong desire to completely remove those agents on the road for better privacy protection and more clear street images.

Significant progress has been made in image inpainting in recent years. The mainstream approaches adopt a patch-based method to complete missing regions by sampling and pasting similar patches from known regions or other source images. The method has been naturally extended to video inpainting, where not only spatial coherence but also temporal coherence is preserved.

The basic idea behind video inpainting is that missing regions/pixels within a frame are observed in some other frames of the same video. Under this observation, some prior art methods use optical flow as guidance to fill the missing pixels either explicitly or implicitly. They are successfully applied in different scenarios with seamless inpainting results. However, flow computation suffers from textureless areas, regardless if it is learning based or not. Furthermore, perspective changes in the video can also degrade the quality of optical flow estimation. These frame-wise flow errors are accumulated when missing pixels are filled from a temporally distant frame, thereby resulting in distorted inpainting results.

The emergence of deep learning, particularly generative adversarial networks (GAN), has provided a powerful tool for inpainting. For images, some conventional methods have formulated inpainting as a conditional image generation problem. Although formulated differently, GAN-based inpainting approaches are essentially identical as the patch-based approach, since the spirit is still looking for similar textures in the training data and fill the holes. Therefore, those conventional methods have to delicately choose their training data to match the domain of the input images, though domain adaptation is not an easy task once the input images come from different scenarios. Moreover, GAN-based approaches share the same problem as the patch-based methods that they are poor at handling perspective changes in images.

With respect to inpainting, the principle of inpainting is filling the target holes by borrowing appearance information from known sources. The sources could be regions other than the hole in the same image, images from the same video or images/videos of similar scenarios. It is critical to reduce the search space for the right pixels. Following different cues, prior works can be categorized into 3 major classes: propagation-based inpainting, patch-based inpainting, and learning-based inpainting.

Propagation-based methods extrapolate boundary pixels around the holes for image completion. These approaches are successfully applied to regions of uniform colors. However, it has difficulties to fill large holes with rich texture variations. Thus, propagation-based approaches usually repair small holes and scratches in an image.

Patch-based methods, on the other hand, not only look at the boundary pixels but also search in the other regions/images for similar appearance in order to complete missing regions. This kind of approach has been extended to the temporal domain for video inpainting. For example, optical flow and color in the missing regions are jointly estimated to address the temporal consistency problem. In general, patch-based methods can better handle non-stationary visual data. As suggested by its name, patch-based methods depend on reliable pixel matches to copy and paste image patches to missing regions. When a pixel match cannot be robustly obtained, for example in cases of big perspective changes or illumination changes, the inpainting results are problematic.

Regarding learning-based inpainting, the success of deep learning techniques inspires recent works on applying it for image inpainting. For example, one prior art method adds a few feature maps in the new Shepard layers to achieve stronger results than a much deeper network architecture. GAN was first introduced to generate novel photos. It is straightforward to extend it to inpainting by formulating inpainting as a conditional image generation problem. Another prior art method proposed context encoders—a convolutional neural network trained to generate contents of an arbitrary image region conditioned on its surroundings. The context encoders are trained to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing parts. Still another prior art method used global and local context discriminators to distinguish real images from fake ones. The global discriminator looks at the entire image to ensure it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. More recently, a contextual attention mechanism in a generative inpainting framework was presented, which further improves the inpainting quality. For video inpainting, an effective framework that is specially designed to exploit redundant information across video frames was formulated. They first synthesize a spatially and temporally coherent optical flow field across video frames, then the synthesized flow field is used to guide the propagation of pixels to fill up the missing regions in the video.

SUMMARY

Embodiments of the present disclosure provide a computer-implemented method of video inpainting, a non-transitory machine-readable medium, and a data processing system.

In a first aspect, some embodiments of the present disclosure provide a computer-implemented method of video inpainting, the method includes: receiving a plurality of sensor data sets comprising depth frames and image frames; for each depth frame, removing one or more objects from the depth frame thereby producing a plurality of resulting depth frames without the one or more objects; stitching the plurality of resulting depth frames into a three-dimensional (3D) map; refining a camera pose of a first image frame having a target inpainting region; and projecting the 3D map onto the first image frame to generate a corresponding depth map.

In a second aspect, some embodiments of the present disclosure provide a non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations include: receiving a plurality of sensor data sets comprising depth frames and image frames; for each depth frame, removing one or more objects from the depth frame thereby producing a plurality of resulting depth frames without the one or more objects; stitching the plurality of resulting depth frames into a three-dimensional (3D) map; refining a camera pose of a first image frame having a target inpainting region; and projecting the 3D map onto the first image frame to generate a corresponding depth map.

In a third aspect, some embodiments of the present disclosure provide a data processing system, the method includes: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including: receiving a plurality of sensor data sets comprising depth frames and image frames; for each depth frame, removing one or more objects from the depth frame thereby producing a plurality of resulting depth frames without the one or more objects; stitching the plurality of resulting depth frames into a three-dimensional (3D) map; refining a camera pose of a first image frame having a target inpainting region; projecting the 3D map onto the first image frame to generate a corresponding depth map; and for each target pixel within the target inpainting region of the first image frame, based on the corresponding depth map, mapping the target pixel within the target inpainting region of the first image frame to a candidate pixel in a second image frame included in the image frames, and determining a candidate color to fill the target pixel.

DETAILED DESCRIPTION

According to some embodiments, to get clear street view and photo realistic simulation in autonomous driving, described herein is an automatic video inpainting algorithm that can remove traffic agents from videos and synthesize missing regions with the guidance of depth/point cloud. By building a dense 3D map from stitched point clouds, frames within a video are geometrically correlated via this common 3D map. In order to fill a target inpainting area in a frame, pixels from other frames can be transformed into the current one with correct occlusion. Furthermore, multiple videos can be fused through 3D point cloud registration, thereby making it possible to inpaint a target video with multiple source videos. The motivation of the embodiments of the disclosure described herein is to solve the long-time occlusion problem where an occluded area has never been visible in the entire video. Therefore, the embodiments of the disclosure are novel and are the first to enable fusing of multiple videos for video inpainting in order to solve such long time occlusion problem.

According to some embodiments, video inpainting method and system with the guidance of three-dimensional (3D) maps in AD scenarios are described herein. In one embodiment, the use of deep learning-based methods can be avoided so that the entire pipeline may only run on central processing units (CPUs). This makes it easy to be generalized to different platforms and different use cases because it does not require graphics processing units (GPUs) and domain adaptation of training data. In some embodiments, 3D map guided inpainting is a new direction for the inpainting community to explore, given that more and more videos are accompanied with depth data.

According to some embodiments, described herein is a method to inpaint street-view videos with the guidance of depth, as image+depth sensors become standard for AD vehicles. For example, depending on the tasks, target objects are either manually labeled or automatically detected in color images, and then removed from their depth counterpart. A 3D map may be built by stitching all point clouds together and projected back onto individual frames. Most of the frame pixels are assigned with a depth value via 3D projection and those remaining pixels obtain their depth by interpolation. With a dense depth map and known extrinsic camera parameters, colors can be sampled from other frames to fill holes within the current frame. These colors serve as an initial guess for those missing pixels, followed by regularization enforcing spatial and photometric smoothness. After that, color harmonization can be applied to generate smooth and seamless blending boundaries. In the end, a moving average is applied along the optical flow to make the final inpainted video look smooth temporally.

Unlike learning-based methods, the approach described above cannot inpaint occluded areas if they are never visible in the video. To solve this problem, fusion inpainting may be utilized, which makes use of multiple video clips to inpaint a target region. Compared to state-of-the-art inpainting approaches, embodiments of the disclosure described herein are able to preserve better details in the missing region with correct perspective distortion. Temporal coherence is implicitly enforced since the 3D map is consistent across all frames. Moreover, multiple video clips captured at different times can be inpainted by registering all frames into a common 3D point map. In one embodiment, sensor datasets captured or collected from an AD vehicle can serve as inputs to the embodiments of the disclosure described herein, though other suitable datasets can be utilized, such as datasets involving both indoor and outdoor scenarios, as long as synchronized image+depth data are utilized.

In one aspect, a computer-implemented method of video inpainting is described. The method may receive a plurality of sensor data sets comprising depth frames and image frames. For each depth frame, the method may further remove one or more objects from the depth frame thereby producing a plurality of resulting depth frames without the one or more objects. The method may further stitch the plurality of resulting depth frames into a 3D map. The method may further refine a camera pose of a first image frame having a target inpainting region. The method may further project the 3D map onto the first image frame to generate a corresponding depth map.

In one embodiment, for each target pixel within the target inpainting region of the first image frame, based on the corresponding depth map, the method may further map the target pixel within the target inpainting region of the first image frame to a candidate pixel in a second image frame included in the image frames. The method may further determine a candidate color to fill the target pixel. The method may further perform Poisson image editing on the first image frame to achieve color consistency between inside and outside of the target inpainting region of the first image frame. The method may further use video fusion inpainting to inpaint occluded areas within the target inpainting region. For each pixel in the target inpainting region of the first image frame, the method may trace the pixel into neighboring frames and replacing an original color of the pixel with an average of colors sampled from the neighboring frames.

Referring now toFIG. 2, in one embodiment, sensor system115includes, but it is not limited to, one or more cameras211, global positioning system (GPS) unit212, inertial measurement unit (IMU)213, radar unit214, and a light detection and range (LIDAR) unit215. GPS system212may include a transceiver operable to provide information regarding the position of the ADV. IMU unit213may sense position and orientation changes of the ADV based on inertial acceleration. Radar unit214may represent a system that utilizes radio signals to sense objects within the local environment of the ADV. In some embodiments, in addition to sensing objects, radar unit214may additionally sense the speed and/or heading of the objects. LIDAR unit215may sense objects in the environment in which the ADV is located using lasers. LIDAR unit215could include one or more laser sources, a laser scanner, and one or more detectors, among other system components. Cameras211may include one or more devices to capture images of the environment surrounding the ADV. Cameras211may be still cameras and/or video cameras. A camera may be mechanically movable, for example, by mounting the camera on a rotating and/or tilting a platform.

Some or all of the functions of ADV101may be controlled or managed by ADS110, especially when operating in an autonomous driving mode. ADS110includes the necessary hardware (e.g., processor(s), memory, storage) and software (e.g., operating system, planning and routing programs) to receive information from sensor system115, control system111, wireless communication system112, and/or user interface system113, process the received information, plan a route or path from a starting point to a destination point, and then drive vehicle101based on the planning and control information. Alternatively, ADS110may be integrated with vehicle control system111.

While ADV101is moving along the route, ADS110may also obtain real-time traffic information from a traffic information system or server (TIS). Note that servers103-104may be operated by a third party entity. Alternatively, the functionalities of servers103-104may be integrated with ADS110. Based on the real-time traffic information, MPOI information, and location information, as well as real-time local environment data detected or sensed by sensor system115(e.g., obstacles, objects, nearby vehicles), ADS110can plan an optimal route and drive vehicle101, for example, via control system111, according to the planned route to reach the specified destination safely and efficiently.

FIGS. 3A and 3Bare block diagrams illustrating an example of an autonomous driving system used with an ADV according to one embodiment. System300may be implemented as a part of ADV101ofFIG. 1including, but is not limited to, ADS110, control system111, and sensor system115. Referring toFIGS. 3A-3B, ADS110includes, but is not limited to, localization module301, perception module302, prediction module303, decision module304, planning module305, control module306, routing module307and sensor data collection308.

Based on the sensor data provided by sensor system115and localization information obtained by localization module301, a perception of the surrounding environment is determined by perception module302. The perception information may represent what an ordinary driver would perceive surrounding a vehicle in which the driver is driving. The perception can include the lane configuration, traffic light signals, a relative position of another vehicle, a pedestrian, a building, crosswalk, or other traffic related signs (e.g., stop signs, yield signs), etc., for example, in a form of an object. The lane configuration includes information describing a lane or lanes, such as, for example, a shape of the lane (e.g., straight or curvature), a width of the lane, how many lanes in a road, one-way or two-way lane, merging or splitting lanes, exiting lane, etc.

For each of the objects, prediction module303predicts what the object will behave under the circumstances. The prediction is performed based on the perception data perceiving the driving environment at the point in time in view of a set of map/rout information311and traffic rules312. For example, if the object is a vehicle at an opposing direction and the current driving environment includes an intersection, prediction module303will predict whether the vehicle will likely move straight forward or make a turn. If the perception data indicates that the intersection has no traffic light, prediction module303may predict that the vehicle may have to fully stop prior to enter the intersection. If the perception data indicates that the vehicle is currently at a left-turn only lane or a right-turn only lane, prediction module303may predict that the vehicle will more likely make a left turn or right turn respectively.

For each of the objects, decision module304makes a decision regarding how to handle the object. For example, for a particular object (e.g., another vehicle in a crossing route) as well as its metadata describing the object (e.g., a speed, direction, turning angle), decision module304decides how to encounter the object (e.g., overtake, yield, stop, pass). Decision module304may make such decisions according to a set of rules such as traffic rules or driving rules312, which may be stored in persistent storage device352.

Routing module307is configured to provide one or more routes or paths from a starting point to a destination point. For a given trip from a start location to a destination location, for example, received from a user, routing module307obtains route and map information311and determines all possible routes or paths from the starting location to reach the destination location. Routing module307may generate a reference line in a form of a topographic map for each of the routes it determines from the starting location to reach the destination location. A reference line refers to an ideal route or path without any interference from others such as other vehicles, obstacles, or traffic condition. That is, if there is no other vehicle, pedestrians, or obstacles on the road, an ADV should exactly or closely follows the reference line. The topographic maps are then provided to decision module304and/or planning module305. Decision module304and/or planning module305examine all of the possible routes to select and modify one of the most optimal routes in view of other data provided by other modules such as traffic conditions from localization module301, driving environment perceived by perception module302, and traffic condition predicted by prediction module303. The actual path or route for controlling the ADV may be close to or different from the reference line provided by routing module307dependent upon the specific driving environment at the point in time.

Based on the planning and control data, control module306controls and drives the ADV, by sending proper commands or signals to vehicle control system111, according to a route or path defined by the planning and control data. The planning and control data include sufficient information to drive the vehicle from a first point to a second point of a route or path using appropriate vehicle settings or driving parameters (e.g., throttle, braking, steering commands) at different points in time along the path or route.

Note that decision module304and planning module305may be integrated as an integrated module. Decision module304/planning module305may include a navigation system or functionalities of a navigation system to determine a driving path for the ADV. For example, the navigation system may determine a series of speeds and directional headings to affect movement of the ADV along a path that substantially avoids perceived obstacles while generally advancing the ADV along a roadway-based path leading to an ultimate destination. The destination may be set according to user inputs via user interface system113. The navigation system may update the driving path dynamically while the ADV is in operation. The navigation system can incorporate data from a GPS system and one or more maps so as to determine the driving path for the ADV.

In one embodiment, while the ADV is moving along a route, sensor data collection module308may capture or collect sensor data, such as camera data, LIDAR data or frames, radar data, etc., generated from sensor system115(e.g., cameras211, radar unit214, LIDAR unit215, etc.) and store the sensor data as part of sensor data sets313, which may be stored in persistent storage device352, or alternatively on a server (e.g., server103-104), for subsequent processing. In one embodiment, sensor data sets313may include large-scale datasets of videos recorded over a period of time. The videos, for example, may include synchronized images, point clouds of depth images, etc. In some embodiments, data sets313may include challenging scenes, for example background occluded by a large vehicle (such as a bus, shuttle or truck) in an intersection, a vehicle in front blocking a front view at all time, etc. For those lengthy time occlusion scenarios, the background may be missing in the entire video sequence. Accordingly, those challenging scenarios may be captured more than once to provide data for video fusion inpainting, as described in more detail herein below.

FIG. 4Ais a block diagram illustrating an example video inpainting system according to one embodiment. Referring toFIG. 4A(andFIGS. 4B-4C), video inpainting system400includes a depth map construction module401, a candidate color sampling module402, a regularization module403, a color harmonization module404, and a temporal smoothing module405. In one embodiment, depth map construction module401receives (or retrieves) the sensor data sets313, for example from persistent storage device352or a server (e.g., server103-104). Sensor data sets313may include depth frames510(e.g., LIDAR frames), where each depth image includes frame-wise point clouds representing one or more objects and a background of a scene. Sensor data sets313may also include image frames560captured, for example, from cameras211of system115.

As shown inFIG. 4B, depth map construction module401may include a dynamic object removal module421, a map stitching module422, and a pose refinement module423. For each depth frame of the depth frames510, dynamic object removal module421may remove moving objects from the point clouds and only maintain background points in a final 3D map. This can be relatively straight-forward to perform once the calibration between a depth sensor (e.g., LIDAR unit215) and an image sensor (e.g., cameras211) is performed. For example, to remove each moving object, all points that are projected in one or more bounding boxes (or target regions) of an image surrounding their respective objects (e.g., bounding boxes531-532of image530) can be removed. The bounding boxes can be automatically detected or manually labeled. Alternatively, machine learning on point clouds (e.g., PointNet++) may be utilized to detect and remove those moving objects directly from the point clouds.

Using the resulting point clouds of depth frames510, with the moving objects being removed, map stitching module422may stitch those resulting point clouds in a 3D map520. For example, in one embodiment, module422may invoke an odometry and mapping method or tool in real-time, such as a LIDAR odometry and mapping (LOAM) method, to fuse the resulting depth frames to build 3D map520. In another embodiment, a real-time 3D reconstruction and interaction method can be used to reconstruct 3D map520based on the resulting depth frames. This method can be further down-sampled to generate final point clouds with a reasonable resolution.

In one embodiment, pose refinement module423is configured to refine a camera pose of each image (e.g., image530). For example, relative poses between a depth sensor and an image sensor can be calibrated in advance. However, there are still some misalignments between the point clouds and image pixels. Vibrations, inaccurate synchronization, and/or accumulative errors from point cloud stitching can cause pose offset between the image sensor and depth sensor. As an example, referring toFIG. 4D, the point cloud is projected into a target region610with colors. The left image shows projection by calibration result. Obviously misalignment can be seen at the boundaries. The right image shows projection by optimized rotation R, where points match more effectively with surrounding pixels. The region between target region620and target region630is where the colors of projected 3D points (e.g., point cloud in 3D map520) are compared to image pixels (e.g., pixels in image530) to optimize camera rotation matrix R.

In order to produce seamless inpainting results, such offset should be compensated even if it is minor. From the initial extrinsic calibration between the image sensor and depth sensor, their relative rotation R and translation T are optimized by minimizing the photometric projection error. The error is defined as:
E=Σp∈Ω|c(p)−c(q)|2,
where p is a pixel projection from the 3D map, Ω is an area surrounding the target inpainting region, which is illustrated inFIG. 4Das the region between target regions620and630, and q is original pixel in the image overlaid by p. The function c returns the value of a pixel.

Note that the colors and locations of a pixel are discrete values, thereby making the error function E non-continuous on R and T. The equation above cannot be solved directly using standard solvers, such as Levenberg-Marquardt algorithm or Gauss-Newton algorithm. Instead, discrete spaces of R and T are searched to minimize E. However, R and T may have six degrees of freedom (DoF) in total, thereby making the searching space extremely large. Accordingly, T can be fixed and only R is optimized because R is dominant at determining projection location when the majority of the 3D map are distant points. Moreover, in some embodiments, only projection pixels need to be moved slightly in vertical and horizontal directions in the image space, which is determined by pitch and yaw angles of a camera (e.g., camera(s)211ofFIG. 2). Finally, the search space may be reduced to 2 DoF, which significantly speeds up the optimization process.

Referring back toFIGS. 4B-4C, once the camera pose refined by module423, the 3D map520is projected onto each image frame (e.g., image530) to generate a corresponding depth map540. Note that some point clouds are captured far from the current image, which can be occluded and de-occluded during the projection process. Hence, a z-buffer may be employed to obtain the nearest depth.

To obtain a fully dense depth map, a depth estimation method can be employed to learn and produce dense depth maps from sparse ones, though a linear interpolation may be sufficient to generate a dense 3D map. In some embodiments, a median filter may be applied to remove some individual noise points, and a final example dense depth map is shown inFIG. 5. InFIG. 5, an image and its corresponding dense depth map are shown. Note that the depth is only rendered for background points and all moving objects have been removed.

With continued reference toFIG. 4A, in one embodiment, candidate color sampling module402is configured to map a pixel from one image to pixels from other images. As every pixel is assigned a depth value, it is possible to map a pixel from one image to other images. There are multiple choices of colors to fill in the pixels of a target inpainting region, and a guideline should be followed to find the best candidate color. For example, there are two principles to choose the best color candidate: 1) choose from the frame that is closer to the current frame temporally and 2) choose from the frame where the 3D background is closer to the camera. The first requirement ensures the video inpainting approach described herein suffers less from perspective distortion and occlusion. The second requirement is because image records more texture details when they are closer to objects, more details can be retained in the inpainting regions.

Under this guideline and the fact that sensors only move forwards during capture, module402is configure to first search forward temporally to the end of video and then backward until the beginning. The first valid pixel is chosen as a candidate pixel, and the valid pixel means its location does not fall into target inpainting regions. Referring now toFIG. 6, which illustrates an example of a candidate selection criteria according to one embodiment, in the top row, a pixel812of frame810may find its candidate colors in two subsequent frames820and830, where road texture appears clearly in both frames820and830. In this case, the frame that is temporally close to the current frame (e.g., frame810) is chosen in order to minimize the impact of perspective change and potential occlusion or de-occlusion. In the bottom row, a pixel852of frame850may find its candidate colors in a previous frame840and a subsequent frame860. In this case, the subsequent frame860is chosen over the previous frame840, since the road texture is lost in frame840.

At this point, every pixel gets a color value individually. If the camera pose and depth value are 100% correct, perfect inpainting results can be generated with smooth boundaries and neighbors. However, such is not the case in the real world, particularly a depth map often carries errors. Therefore, regularization module403is invoked to enforce smoothness constraints. In one embodiment, the color selection may be formulated as a discrete global optimization problem and solve it using, for example, belief propagation (BP). Before explaining the formulation, however, the color space and neighbors of a target pixel are first defined. Referring now toFIG. 7, as shown in the left image pair (images910-920), a target pixel912in frame910finds its candidate pixel922from a source image920. Due to depth inaccuracy, the true color may not lie exactly on the candidate pixel, but a small window921around. Then, all pixel colors from n by n window921may be collected to form a color space for the target pixel912, where n is a positive integer.

Still referring toFIG. 7, the right image pair (images930-940) illustrates how to find out the expected colors of neighbors. Due to perspective changes, the four neighbors933of a target pixel932are not necessarily neighbors in source image940. Thus, neighboring pixels943may be warped into the source image940by their depth value to sample the expected colors.

In more detail, in one embodiment, let P be a set of pixels in a target inpainting region (e.g., region911/931ofFIG. 7) and L be a set of labels. The labels correspond to indices of potential colors in the color space. A labeling function l assigns a lpϵL to each pixel pϵP. It is assumed that the labels should vary smoothly almost everywhere, but may change dramatically at some places such as object boundaries. The quality of labeling is given by an energy function as:

E=∑(p,q)∈N⁢V⁡(lp,lq)+∑p∈P⁢Dp⁡(lp),
where N represents the number of edges in a four-connected image grid graph. V(lp,lq) is the cost of assigning labels lp and lq to two neighboring pixels, and may be referred to as a discontinuity cost. Dp(lp) is the cost of assigning label lp to pixel p, which may be referred to as a data cost. Determining a labeling with minimum energy corresponds to a Maximum A Posteriori (MAP) estimation problem.

Accordingly, a boundary smoothness constraint can be incorporated into the data cost as follows:

Here, Cp(·) and Cq(·) fetch colors for p and q at label lpand lq. L, R, T, B stand for q is on the left, on the right, on the top and on the bottom, respectively. For a pair of two neighboring pixels p and q, the differences between p's color and q's expected color of p can be computed, and vice versa.

In one embodiment, color harmonization module404is configured to use Poisson image editing to generate smooth and seamless blending boundaries. As an example, pixels from different frames may have different colors due to changing camera exposure time and white balance, thereby causing color discontinuities (as can be seen inFIG. 8). These problems may be solved using Poisson image editing. Poisson image editing is originally proposed to clone an image patch from source image into a destination image with seamless boundary and original texture. It achieves this by solving the following minimization problem:

Ω is the inpainting region with boundary ∂Ω. f*is a color function of a destination image and f is the color function of the target inpainting region within the destination image. Δ.=[∂./∂x, ∂./∂y] is a gradient operator. v is the desired color gradient defined over Ω.

Here, v is computed using the output from the belief propagation step, with one exception. If two neighboring pixels within Ω are from different frames, their gradient may be set to 0. This allows color consistency within the inpainting regions. The effectiveness of this solution is demonstrated inFIG. 8. InFIG. 8, left image1010is an input image. Middle image1020is an inpainting result. Note the color discontinuity in region1022and the blank pixels in region1021. Note that blank-pixel region1021is also filled up. Since blank pixels have 0 gradient values, solving the Poisson equation on this part is equivalent to smooth color interpolation. Lastly, the right image1030is the result after color harmonization.

Herein, it is assumed that the inpainting regions are visible in some other frames. Otherwise, some pixels would remain blank, as can be seen fromFIG. 8in region1021. Learning-based methods can hallucinate inpainting colors from their training data. In contrast, embodiments of the disclosure described herein cannot inpaint occluded areas if they are never visible in the video, thereby leaving blank pixels.

For small areas of blank pixels, a smooth interpolation is sufficient to fill the holes. However, in some embodiments, a vehicle in front, for example, can block a wide field of view for the entire video duration, thereby leaving large blank holes. A simple interpolation is not capable of handling this problem. This problem can be addressed by capturing another video of the same scene, where the occluded parts become visible. Fortunately, newly captured frames can be registered into an existing 3D map using a LOAM method. Once new frames are registered and merged into the existing 3D map, inpainting is performed exactly the same way.

With continued reference toFIG. 4A, temporal smoothing module405may compute forward and backward optical flows for all result frames. That is, for every pixel in the target inpainting areas, module405may trace the pixel into neighboring frames using the optical flow and replace the original color of the pixel with an average of colors sampled from neighboring frames.

FIG. 9is a flow diagram illustrating an example method of constructing a depth map according to one embodiment. Method900may be performed by hardware, software, or a combination of both. For example, method900may be performed by depth map construction module401ofFIG. 4B.

Referring theFIG. 9, at block901, the method receives a multiplicity of sensor data sets including depth frames and image frames. At block902, for each depth frame, the method removes one or more objects from the depth frame thereby producing a multiplicity of resulting depth frames without the one or more objects. At block903, the method stitches the resulting depth frames into a 3D map. At block904, the method refines a camera pose of a first image frame having a target inpainting region. At block905, the method projects the 3D map onto the first image frame to generate a corresponding depth map.

FIG. 10is a flow diagram illustrating an example of a method of video inpainting according to one embodiment. Method1000may be performed by hardware, software, or a combination of both. For example, method1000may be performed by system400ofFIG. 4A.

Referring toFIG. 10, at block1001, the method stitches a multiplicity of depth frames into a 3D map, where one or more objects in the depth frames have previously been removed. At block1002, the method projects the 3D map onto a first image frame to generate a corresponding depth map, where the first image frame includes a target inpainting region. At block1003, for each target pixel within the target inpainting region of the first image frame, based on the corresponding depth map, the method maps the target pixel within the target inpainting region of the first image frame to a candidate pixel in a second image frame. At block1004, the method determines a candidate color to fill the target pixel. At block1005, the method performs Poisson image editing on the first image frame to achieve color consistency between inside and outside of the target inpainting region of the first image frame. At block1006, for each pixel in the target inpainting region of the first image frame, the method traces the pixel into neighboring frames and replacing an original color of the pixel with an average of colors sampled from the neighboring frames.