Patent Description:
Augmented Reality (AR) relates to technology that provides a composite view of a real-world environment together with a virtual-world environment (e.g., computer generated input). Correct perception of depth is often needed to deliver a realistic and seamless AR experience. For example, in AR-assisted maintenance or manufacturing tasks, the user tends to interact frequently with both real and virtual objects. However, without correct depth perception, it is difficult to provide a seamless interaction experience with the appropriate occlusion handling between the real-world scene and the virtual-world scene.

In general, real-time 3D sensing is computationally expensive and requires high-end sensors. To reduce this overhead, some early work relies on 2D contour tracking to infer an occlusion relationship, which is typically assumed to be fixed. Alternatively, some other work includes building 3D models of the scene offline and using these 3D models online for depth testing, assuming the scene is static and remains unchanged. Although these methods can achieve some occlusion handling effects, they cannot accommodate the dynamic nature of user interactions which are very common in AR applications.

Also, the recent arrival of lightweight RGB-Depth (RGB-D) cameras provide some 3D sensing capabilities for AR applications. However, these RGB-D cameras typically have low cost consumer depth sensors, which usually suffer from various types of noises, especially around object boundaries. Such limitations typically cause unsuitable visual artifacts when these lightweight RGB-D cameras are used for AR applications, thereby prohibiting decent AR experiences. Plenty of research has been done for depth map enhancement to improve the quality of sensor data provided by these lightweight RGB-D cameras. An example of such a depth map enhancement is disclosed in <NPL>. However, the majority of these approaches cannot be directly applied to AR use cases due to their high computational cost.

In addition, filtering is often used for image enhancement. For instance, some examples include a joint bilateral filtering process or a guided image filtering process. Also, other examples include a domain transform process, an adaptive manifolds process, or an inpainting process. However, these processes are typically computationally expensive and often result in edge blurring, thereby causing interpolation artifacts around boundaries.

The following is a summary of certain embodiments described in detail below. The described aspects are presented merely to provide the reader with a brief summary of these certain embodiments and the description of these aspects is not intended to limit the scope of this disclosure.

The features, aspects, and advantages of the present invention are further clarified by the following detailed description of certain exemplary embodiments in view of the accompanying drawings throughout which like characters represent like parts.

The embodiments described above, which have been shown and described by way of example, and many of their advantages will be understood by the foregoing description, and it will be apparent that various changes can be made in the form, construction, and arrangement of the components without departing from the disclosed subject matter or without sacrificing one or more of its advantages. Indeed, the described forms of these embodiments are merely explanatory. These embodiments are susceptible to various modifications and alternative forms, and the following claims define the scope of the invention.

<FIG> illustrates a block diagram of a system <NUM> for dynamic occlusion handling in AR according to an example embodiment. In an example embodiment, the system <NUM> includes a head mounted display <NUM> and a dynamic occlusion handling system <NUM>. In addition, the system <NUM> includes communication technology <NUM> that connects the head mounted display <NUM> to the dynamic occlusion handling system <NUM>. In an example embodiment, the communication technology <NUM> is configured to provide at least data transfer between the head mounted display <NUM> and the dynamic occlusion system <NUM>. In an example embodiment, the communication technology <NUM> includes wired technology, wireless technology, or a combination thereof. As non-limiting examples, the communication technology <NUM> includes HDMI technology, WiFi technology, or any suitable communication link.

In an example embodiment, the head mounted display <NUM> is an optical head mounted display, which is enabled to reflect projected images while allowing a user to see through it. In an example embodiment, the head mounted display <NUM> includes at least a depth sensor <NUM> and a video camera <NUM>. In <FIG>, for instance, the head mounted display <NUM> includes an RGB-D camera <NUM>, which includes a depth sensor <NUM> and a video camera <NUM>. In an example embodiment, the RGB-D camera <NUM> can be near-range.

In an example embodiment, the depth sensor <NUM> is configured to provide depth data, as well as geometry information for dynamic occlusion handling. In this regard, for instance, the depth sensor <NUM> is a structured-light sensor or Time-of-Flight sensor. Alternatively, a stereo sensor can be used to obtain dynamic depth information. In an example embodiment, depending upon the application, the depth sensor <NUM> can have any suitable sensing range. For instance, in <FIG>, the RGB-D camera <NUM> includes a depth sensor <NUM> with a sensing range of <NUM> to <NUM>, which is sufficient to cover an area of AR interactions involving the user's hands <NUM>.

In an example embodiment, the video camera <NUM> is configured to provide video or a recorded series of color images. In an example embodiment, the video camera <NUM> is configured to provide scene tracking (e.g., visual SLAM). In addition, since the glasses view <NUM>, provided by the head mounted display <NUM>, is unable to provide information for dynamic occlusion handling, the system <NUM> uses the video data from the video view <NUM> and adopts the video view <NUM> as the glasses view <NUM> to provide dynamic occlusion handling.

In an example embodiment, the system <NUM> includes the dynamic occlusion handling system <NUM>. In an example embodiment, the dynamic occlusion handling system <NUM> is any suitable computing system that includes a dynamic occlusion handling module <NUM> and that can implement the functions disclosed herein. As non-limiting examples, the computing system is a personal computer, a laptop, a tablet, or any suitable computer technology that is enabled to implement the functions of the dynamic occlusion handling module <NUM>.

In an example embodiment, the computing system includes at least input/output (I/O) devices <NUM>, a communication system <NUM>, computer readable media <NUM>, other functional modules <NUM>, and a processing system <NUM>. In an example embodiment, the I/O devices can include any suitable device or combination of devices, such as a keyboard, a speaker, a microphone, a display, etc. In an example embodiment, the communication system <NUM> includes any suitable communication means that enables the components of the dynamic occlusion handling system <NUM> to communicate with each other and also enables the dynamic occlusion handling system <NUM> to communicate with the head mounted display <NUM> via the communication technology <NUM>. Also, in an example embodiment, the communication system <NUM> includes any suitable communication means that enables the computing the dynamic occlusion handling system <NUM> to connect to the Internet, as well as with other computing systems and/or devices on a computer network or any suitable network. In an example embodiment, the computer readable media <NUM> is a computer or electronic storage system that is configured to store and provide access to various data to enable the functions disclosed herein. In an example embodiment, the computer readable media <NUM> can include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable memory technology. In an example embodiment, the computer readable media <NUM> is local, remote, or a combination thereof (e.g., partly local and partly remote). In an example embodiment, the other functional modules <NUM> can include hardware, software, or a combination thereof. For instance, the other functional modules <NUM> can include an operating system, logic circuitry, any hardware computing components, any software computing components, or any combination thereof. In an example embodiment, the processing system <NUM> includes at least one processing unit to perform and implement the dynamic occlusion handling in accordance with the dynamic occlusion handling module <NUM>. In <FIG>, for instance, the processing system <NUM> includes at least a central processing unit (CPU) and a graphics processing unit (GPU).

As discussed above, the dynamic occlusion handling system <NUM> includes a dynamic occlusion handling module <NUM>. In an example embodiment, the dynamic occlusion handling module <NUM> includes hardware, software, or a combination thereof. In an example embodiment, the dynamic occlusion handling module <NUM> is configured to provide the requisite data and support to the processing system <NUM> such that a process <NUM> (e.g. <FIG>) is enabled to execute and provide enhanced depth data and dynamic occlusion handling, thereby providing a realistic AR experience.

<FIG> illustrate non-limiting examples in which virtual objects <NUM> are rendered in video view <NUM>, as the acquisition sensor space. Specifically, <FIG> illustrates a virtual object <NUM> rendering without dynamic occlusion handling while <FIG> illustrates a virtual object <NUM> rendering with dynamic occlusion handling. In this regard, in each of <FIG>, the virtual object <NUM> rendering includes a treasure chest as the virtual object <NUM>. Also, in each of <FIG>, the remaining parts of this video view include the user's hand <NUM> in a real-world environment. However, without dynamic occlusion handling, the user's hand <NUM> is improperly occluded by the virtual object <NUM>, as shown in the circled region <NUM> of <FIG>. That is, the circled region <NUM> of <FIG> does not provide a realistic portrayal of the user's hand <NUM> interacting with the virtual object <NUM>. In contrast, with dynamic occlusion handling, the user's hand <NUM> is not occluded by the virtual object <NUM>, as shown in the circled region <NUM> of <FIG>. As such, with dynamic occlusion handling, the circled region <NUM> of <FIG> is able to provide a realistic portrayal of the user's hand <NUM> interacting with the virtual object <NUM>.

<FIG> relate to renderings of the virtual objects <NUM> of <FIG>, respectively. More specifically, <FIG> illustrate non-limiting examples of the rendering of the virtual objects <NUM> without dynamic occlusion handling. In this regard, <FIG> represents a left eye view of the rendering of the virtual object <NUM> and <FIG> represents a right eye view of the rendering of the virtual object <NUM>. In contrast, <FIG> illustrate non-limiting examples of the rendering of the virtual objects <NUM> with dynamic occlusion handling. More specifically, <FIG> represents a left eye view of the rendering of the virtual object <NUM> and <FIG> represents a right eye view of the rendering of the virtual object <NUM>. As shown in <FIG>, with dynamic occlusion handling, the virtual object <NUM> is modified, as highlighted in each circled region <NUM>, such that the virtual object <NUM> does not occlude the user's hand <NUM>. Accordingly, with dynamic occlusion handling, the interaction between the virtual object <NUM> and the user's hand <NUM> is presented in a proper and realistic manner, as shown in at least the circled region <NUM> of <FIG> and <FIG>.

<FIG> illustrate non-limiting examples of optical, see-through images of the virtual objects <NUM> in glasses view <NUM> via the optical head mounted display <NUM>. <FIG> illustrate examples without dynamic occlusion handling. Specifically, <FIG> represents a left eye view of the virtual object <NUM> in the glasses view <NUM> and <FIG> represents a right eye view of the virtual object <NUM> in the glasses view <NUM>. In contrast, <FIG> illustrate examples with dynamic occlusion handling. Specifically, <FIG> represents a left eye view of the virtual object <NUM> in the glasses view <NUM> and <FIG> represents a right eye view of the virtual object <NUM> in the glasses view <NUM>. As evidenced by a comparison of <FIG> with that of <FIG>, the inclusion of the dynamic occlusion handling provides a more realistic and immersive experience, as the parts of the virtual objects <NUM> that should be occluded by the user's hands <NUM> are removed from view.

<FIG> provide non-limiting examples of mismatches between boundaries of objects taken from depth maps compared to corresponding boundaries of objects taken from color images. Specifically, <FIG> illustrates an example of a depth map <NUM> and <FIG> illustrates a corresponding example of a color image <NUM>. In addition, <FIG> illustrates an example of the depth map <NUM> of <FIG> overlaying the color image <NUM> of <FIG>. Meanwhile, <FIG> illustrates an enlarged view of the boxed region <NUM> of <FIG> illustrates an enlarged view of the boxed region <NUM> of <FIG>. As shown in <FIG>, the boundary of the user's hand <NUM> in the depth map <NUM> does not match the corresponding boundary of the user's hand <NUM> in the color image <NUM>.

<FIG> are example results based on dynamic occlusion handling with raw depth data from the depth map <NUM>. Specifically, <FIG> includes dynamic occlusion handling, particularly with regard to a rendering of a virtual object <NUM> (e.g., smartphone) in relation to a user's hand <NUM>. Meanwhile, <FIG> illustrates an enlarged view of the boxed region <NUM> of <FIG>. In addition, <FIG> illustrates an enlarged view of the boxed region <NUM> of <FIG>. As shown, when dynamic occlusion handling is performed with raw depth data, <FIG> include visual artifacts due to the mismatches in boundaries of at least the user's hand <NUM> between the raw depth map <NUM> and the color image <NUM>. However, the system <NUM> includes a process <NUM>, which is enabled to overcome this issue by improving the consistency of object boundaries, for instance, between depth data and RGB data.

<FIG> is a block diagram of a process <NUM> of the system <NUM> according to an example embodiment. In an example embodiment, upon receiving depth data and video data from the RGB-D camera <NUM>, the process <NUM> includes at least a video view process <NUM> and a glasses view rendering process <NUM>. In this regard, the process <NUM> is performed when the processing system132 executes computer-readable data (e.g., computer-executable data), which is stored on non-transitory computer readable media via the dynamic occlusion handling module <NUM>, the computer readable media <NUM>, or a combination thereof. Generally, the computer executable data can include various instructions, data structures, applications, routines, programs, modules, procedures, other software components, or any combination thereof.

In an example embodiment, the process <NUM> leverages instances in which boundaries in raw depth maps are normally reasonably close to their counterparts in the corresponding color images, where the image gradients are typically high. In an example embodiment, the process <NUM> includes snapping at least one depth edge point towards its desired target location. In this regard, based on the above, the process <NUM> includes discretizing the solution space by constraining the target position of the depth edge point to be on a local line segment and then find an optimal solution for the entire set of depth edge points via discrete energy minimization.

In an example embodiment, the process <NUM> includes a video view process <NUM> and a glasses view rendering process <NUM>, as shown in <FIG>. In an example embodiment, the video view process <NUM> includes a depth edge point process <NUM>, a candidate search process <NUM>, an optimization process <NUM>, and a depth map enhancement process <NUM>. In an example embodiment, the depth edge point process <NUM> includes depth edge point extraction <NUM>, grouping and ordering <NUM>, and 2D normal computations <NUM>. More specifically, in an example embodiment, the process <NUM> includes extracting depth edge points from a depth map and computing smooth 2D normal directions with respect to the extracted depth edge points. In this regard, each 2D normal segment or line defines a solution space for a corresponding edge point, i.e. candidates for each edge point are searched only along this normal direction. In an example embodiment, after the candidate search process <NUM>, the process <NUM> includes an optimization process <NUM> based on the results of the candidate search <NUM> to locate and utilize optimal snapping targets. In this regard, for instance, the optimization process <NUM> includes defining energy functions in a solution space that includes at least a data term and a smoothness term. Also, in this case, the optimization process <NUM> includes performing energy minimization efficiently via dynamic programming to identify the optimal target position for each edge point. In an example embodiment, the process <NUM> includes a depth map enhancement process <NUM>, which is based on an output of edge-snapping. Upon enhancing the depth map, the process <NUM> switches from the video view process <NUM> to the glasses view rendering process <NUM>.

<FIG> is a flow diagram of a depth edge point process <NUM> according to an example embodiment. In an example embodiment, the depth edge point process <NUM> is configured to extract depth edge points from depth points (or pixels) with valid depth values. In addition, the depth edge point process is configured to perform a number of operations in preparation for the candidate search process <NUM> and optimization process <NUM>. More specifically, an example implementation <NUM> of the depth edge point process <NUM> is discussed below.

At step <NUM>, the processing system <NUM> is configured to extract depth edge points. In an example embodiment, for instance, the depth edge points are those points whose local neighborhood exhibits large depth discontinuity. In this regard, for instance, the processing system <NUM> primarily or only considers depth points (or pixels) with valid depth values. For each of these pixels, a <NUM>×<NUM> local patch is examined. If any of the four-neighbor pixels either has an invalid depth value or has a valid depth value that differs from the center pixel beyond a certain value, then this center pixel is considered to be a depth edge point. As an example, the raw depth map normally could contain some outliers as isolated points or a very small patch. To remove the effect of these outliers, the processing system <NUM> is configured to apply a morphological opening, i.e. erosion followed by dilation, to the depth map mask before extracting the depth edge points.

At step <NUM>, the processing system <NUM> is configured to perform a depth first search on each image group to group the extracted depth edge points. During the depth first search, two depth edge points are considered connected only when one is in the <NUM>×<NUM> neighborhood of the other and the depth difference between these two depth points (or pixels) is less than a certain threshold τmax.

At step <NUM>, the processing system <NUM> is configured to order the depth edge points of each group so that they traverse from one end of the edge contour towards the other, as required by some of other processes (e.g., the optimization process <NUM>). In some cases, such as when an edge contour is a cyclic contour, the processing system <NUM> is configured to select one of the depth edge points as the starting point, wherein the selection can be performed at random or by any suitable selection method. In an example embodiment, the following operations in the remainder of this discussion of <FIG> are performed for each group of depth edge points separately. Meanwhile, <FIG> shows an example containing multiple groups of depth edge points.

At step <NUM>, the processing system <NUM> is configured to perform low pass filtering on the raw depth edge points to smooth the 2D positions of these depth edge points. More specifically, due to zigzag pattern or unevenness of the raw depth edges, the normal directly computed from these raw depth edge points may suffer from substantial artifacts. In contrast, with low pass filtering, the processing system <NUM> is configured to reduce noise and artifacts by utilizing these smoothed depth edge points at step <NUM>.

At step <NUM>, the processing system <NUM> is configured to compute the 2D normal of these depth edge points. In an example embodiment, the processing system <NUM> is configured to compute the 2D normal of each depth edge point using two neighboring points. In an example embodiment, the processing system <NUM> utilizes the smoothed depth edge points only for the 2D normal computation, while relying on the raw depth edge points for all (or most) of the later processing.

<FIG> illustrate certain aspects of the example implementation <NUM> of the depth edge point processing according to an example embodiment. Specifically, <FIG> illustrates an example of a color image <NUM> from the RGB-D camera <NUM>. <FIG> illustrates an example of a raw depth map <NUM> from the RGB-D camera <NUM>. <FIG> illustrates examples of raw depth edge points <NUM>, which overlay a gray-scale image <NUM>. Meanwhile, <FIG> illustrate enlarged views of portions of <FIG> that correspond to the boxed region <NUM> of <FIG>. In this regard, <FIG> illustrates the raw depth edge points <NUM> associated with a boundary of a thumb of the user's hand <NUM> while <FIG> illustrates smoothed depth edge points <NUM>. In addition, <FIG> illustrates 2D normals <NUM> that are generated based on the raw depth edge points <NUM>. In contrast, <FIG> illustrates 2D normals <NUM> that are generated based on the smoothed depth edge points. As shown, the 2D-normals of the smoothed depth edge points in <FIG> carry less noise than that of the raw depth edge points <FIG>.

<FIG>, <FIG>, <FIG> relate to the candidate search process <NUM> and the optimization process <NUM>. More specifically, <FIG> is a flow diagram of an example implementation <NUM> of the candidate search process <NUM> and the optimization process <NUM> according to an example embodiment. Meanwhile, <FIG>, <FIG>, <FIG> illustrate various aspects of the candidate search process <NUM> and the optimization process <NUM>.

At step <NUM>, in an example embodiment, the processing system <NUM> searches for candidates for each depth edge point. In this regard, for instance, the solution space of snapping each depth edge point is constrained to the line of its 2D normal. Since there is no prior information as to which direction is the target direction, the processing system <NUM> is configured to search in both the positive and negative normal directions to a certain range rs, resulting in 2rs +<NUM> candidates. Also, in an example embodiment, the processing system <NUM> is configured to denote a depth edge point as pi and its corresponding candidate set as Li = {ci,k | k = <NUM>,. ,2rs +<NUM>}.

At step <NUM>, in an example embodiment, the processing system <NUM> obtains the image gradients using a Sobel operator in multiple color spaces. In an example embodiment, the first part of the image gradients is computed directly in the RGB color space by the following equation:
<MAT>.

As indicated above, this equation contains image gradients along both x and y directions. However, in some cases, the image gradients in the RGB color space are not necessarily high along some object boundaries. Thus, in an example embodiment, the processing system <NUM> is configured to enhance the discriminant power by incorporating image gradients from the YCbCr space as indicated by the following equation:
<MAT> At step <NUM>, in an example embodiment, the processing system <NUM> combines these image gradients and defines the cost of snapping a point pi towards a candidate ci,k as follows:
<MAT>
where wrgb and wcbcr are the weights of different color space gradients.

As indicated above, encoding image gradients from multiple color spaces provides a number of advantages. For example, combining different color spaces generally provides more discriminating power for this edge-snapping framework. For instance, in some cases, the RGB color space alone might not be sufficient. In this regard, turning to <FIG>, as an example, the boundaries of the fingertips, as shown in the circled regions <NUM>, do not have a strong image gradient in the RGB space. In this case, when involving only the RGB color space, there are some edge points associated with these fingertips that cannot be snapped to the desired locations. In contrast, when the YCbCr space is incorporated with the RGB space, the processing system <NUM> is configured to achieve snapping results, which are improved compared to those snapping results, which involve only the RGB space. In many AR use cases, there are scenes in which a user interacts with at least one virtual object <NUM>. In such cases, the incorporation of the YCbCr color space is particularly suitable for differentiating skin color associated with a user from other colors (e.g., non-skin colors). Also, in other example embodiments, other color spaces can be used. For instance, the hue channel of an HSV color space can be used. Moreover, although this example embodiment uses RGB and YCbCr spaces, other example embodiments include various combinations of various color spaces.

At step <NUM>, in an example embodiment, the processing system <NUM> defines a smoothness term to penalize a large deviation between neighboring depth edge points (or depth edge pixels). In this regard, to achieve smooth snapping, the processing system <NUM> snaps neighboring depth edge points to locations that are relative close to each other and/or not far away from each other. For instance, in an example embodiment, for a pair of consecutive depth edge points pi and pj, the processing system <NUM> computes the cost of snapping pi onto ci,k and pj onto cj,l via the following equation:
<MAT>.

In this equation, the parameter dmax defines the maximal discrepancy allowed for two consecutive depth edge points.

At step <NUM>, in an example embodiment, the processing system <NUM> determines or finds a candidate for each depth edge point to minimize the following energy function:
<MAT>
where λs leverages the importance of the smoothness constraint. In an example embodiment, this class of discrete optimization problem is solved in an efficient manner via dynamic programming, which identifies an optimal path in the solution space.

At step <NUM>, the processing system <NUM> determines an optimal path by solving the discrete optimization problem considering the data costs and smoothness costs. Specifically, the processing system <NUM> constructs a matrix H of dimension N × (2rs+<NUM>) where N is the number of depth edge points. The entries are initialized with the data term H(i,k) = Ed(i,k). The processing system <NUM> then traverses from the first depth edge point toward the last depth edge point, and updates the matrix via the following equation:
<MAT>.

In an example embodiment, as discussed above, the processing system <NUM> provides this update to find the optimal path from point i to point i+<NUM>, considering both the data costs and the smoothness costs. In an example embodiment, this operation is performed for all the candidates l = <NUM>,. , 2rs+<NUM> and for all depth edge points sequentially. Generally, the k that gives the minimum of the second term is the best candidate that connects pi with pi+<NUM> if candidate l is selected for pi+<NUM> and is recorded during the update. When the update is finished, i.e. reaching the last edge point, the processing system <NUM> selects the candidate that gives minimal cost for the last point. In an example embodiment, the processing system <NUM> then traverses back to locate the best candidates for a previous point given the decision for the current point, which was recorded earlier during the update. In an example embodiment, the processing system <NUM> continues this procedure until the first point is reached where the optimal path is found. In this regard, the optimal path provides a target position to snap for each edge point.

<FIG> illustrate at least one benefit associated with using image gradients from multiple color spaces. In this regard, <FIG> is a non-limiting example of a color image <NUM>. Meanwhile, each of <FIG> illustrate an enlarged view of the boxed region <NUM> of <FIG> include the raw depth edge points <NUM> and their target positions <NUM> after the optimization process <NUM>. In addition, <FIG> also include the paths <NUM>, which show the movements of the raw depth edge points <NUM> to their corresponding target positions <NUM>. More specifically, <FIG> shows the results obtained by only using the image gradient in the RGB space. In contrast, <FIG> shows the result obtained by combining image gradients from both the RGB space and the YCbCr space. In this regard, the fusion of multiple color spaces, as shown in <FIG>, improves the robustness of the edge-snapping framework compared to that of a single color space, as shown in <FIG>.

<FIG> illustrate at least one other benefit associated with using image gradients from multiple color spaces. Specifically, <FIG> illustrates a non-limiting example of a color image (or raw RGB data), as obtained from the RGB-D camera <NUM>. <FIG> is a non-limiting example of the magnitude of image gradients from the red channel <NUM>. In this example, the circled regions <NUM> highlight instances in which the image gradients of the object boundary <NUM> of the user's hand <NUM> are relatively low in the RGB space. <FIG> is a non-limiting example of the magnitude of image gradients in a converted CR channel <NUM>, where the object boundary <NUM> of the user's hand <NUM>, particularly at the fingertips, are more visible.

<FIG> illustrate a number of benefits associated with applying smoothness terms. <FIG> illustrates a non-limiting example of a color image <NUM>. Meanwhile, <FIG> are enlarged views of the boxed region <NUM> of <FIG>. More specifically, <FIG> illustrates the edge-snapping results without a smoothness constraint. In contrast, <FIG> illustrates the edge-snapping results with at least one smoothness constraint. <FIG> include the raw depth edge points <NUM> and their target positions <NUM>. In addition, <FIG> also include the paths <NUM>, which show the movements of the raw depth edge points <NUM> to their corresponding target positions <NUM>. In this regard, the results provided with smoothness constraints, as shown in <FIG>, provide greater edge-snapping accuracy compared to that without smoothness constraints, as shown in <FIG>.

<FIG> illustrate a number of benefits of applying smoothness terms. <FIG> illustrates a non-limiting example of a color image <NUM>. Meanwhile, <FIG> are enlarged views of the boxed region <NUM> of <FIG> illustrates the edge-snapping results without a smoothness constraint. In contrast, <FIG> illustrates the edge-snapping results with at least one smoothness constraint. <FIG> include the raw depth edge points <NUM> and their target positions <NUM>. In addition, <FIG> also include the paths <NUM>, which show the movements of the raw depth edge points <NUM> to their corresponding target positions <NUM>. In this regard, the results provided with smoothness constraints, as shown in <FIG>, provides better edge-snapping accuracy compared to that without smoothness constraints, as shown in <FIG>.

Without the smoothness term, the process <NUM> will basically use the "winner takes all" strategy in that the candidate with the highest image gradient is selected as the target position for each depth edge point. However, when a background scene has some strong edges, this "winner takes all" strategy for selecting target positions will result in various artifacts. In this regard, for instance, <FIG> illustrate examples in which some depth edge points were snapped to undesirable positions having high image gradients. In contrast, the inclusion of the smoothness term within the process <NUM> can effectively prevent such artifacts from occurring, as shown in <FIG>.

<FIG> and <FIG> relate to the depth map enhancement process <NUM>. Specifically, <FIG> is a flow diagram of an example implementation <NUM> of the depth map enhancement process <NUM> according to an example embodiment. In addition, <FIG> illustrate depth map enhancement based on edge-snapping. More specifically, each of <FIG> illustrate a depth map <NUM> overlaying a color image <NUM>. Also, in each of <FIG>, the curve <NUM> represents a boundary of a thumb from the user's hand <NUM>, as taken from the depth map <NUM>. In this example, the shaded region <NUM>, bounded by at least the curve <NUM>, has valid depth measurements while the remaining regions have zero depths. <FIG> also illustrate depth edge points 320A and 320B (as taken from curve <NUM>) and their corresponding target positions 342A and 342B. Also, the points 344A and 344B, illustrated as triangles in <FIG>, represent the depth points (or pixels), which are used for retrieving reference depth values. That is, <FIG> illustrate examples of certain aspects of the example implementation <NUM>, as discussed below.

At step <NUM>, in an example embodiment, the processing system <NUM> considers two consecutive depth edge points 320A and 320B as well as their target positions 342A and 342B, which form a quadrilateral as illustrated by the shaded region <NUM> in each of <FIG>. In an example embodiment, the processing system <NUM> processes all of the depth points (or pixels) inside this quadrilateral (or the shaded region <NUM>) for enhancement. In an example embodiment, this processing is performed for each pair of consecutive depth edge points 320A and 320B. Essentially, each depth point (or pixel) inside the quadrilateral (or the shaded region <NUM>) has incorrect depth measurements due to sensor noises. In an example embodiment, the true depth of each of these points (or pixels) is recovered. However, such an example embodiment could involve significantly complicated operations and be computationally expensive, but might not be necessary for achieving visually pleasing dynamic occlusion effects. Therefore, in another example embodiment, the processing system <NUM> is configured to perform an approximation to estimate reasonable depth values for these depth points (or pixels) that are generally sufficient.

In general, there are typically two types of errors for the depth points (or pixels) in the regions, as shown in <FIG>. For example, the first type of error ("case <NUM>") includes at least one missing value, where the object boundaries of the depth map <NUM> are generally inside the object, as shown within the boxed region <NUM> of <FIG>. Another type of error ("case <NUM>") occurs when depth points (or pixels) belonging to an object further away are labeled with depth values from the occluding object, as shown within the boxed region <NUM> of <FIG>. In both of these cases, the processing system <NUM> implements the following same methodology to modify the depth values.

At step <NUM>, in an example embodiment, for each depth edge point (or pixel) of the pair of consecutive depth edge points 320A and 320B, the processing system <NUM> traverses one step back along the direction from the target to this pixel and retrieves the depth value as a reference depth value. Examples of these reference pixels are represented by the black triangles <NUM> in <FIG>, respectively.

At step <NUM>, in an example embodiment, the processing system <NUM> then takes the average of the reference depth values from the pair and assigns it to all of the depth points (or pixels) inside the region. As illustrated in <FIG>, for case <NUM>, the reference values are taken from a region inside the finger. Therefore, the target region <NUM> will be filled in with some depth from the finger, resulting in filling effects for the missing values. For case <NUM> in <FIG>, the reference values will be zero and the target region will be replaced with zero depth resulting in this piece being removed. With this single procedure, the processing system <NUM> achieves both effects, as desired. When considering speed, this approximation is sufficient for dynamic occlusion handling. However, in an alternative example embodiment, the processing system <NUM> is configured to implement an extrapolation process to estimate the depth values.

In an example embodiment, the depth map enhancement process <NUM> is highly parallel. Accordingly, with regard to the processing system <NUM>, the CPU, the GPU, or a combination thereof can perform the depth map enhancement process <NUM>. In an example embodiment, the edge-snapping moves the depth edge points 320A and 320B in directions towards their target positions 342A and 342B. In an example embodiment, the processing system <NUM> is configured to process all or substantially all of the depth points (or pixels) that fall within the regions of the edge-snapping. After the depth map enhancement process <NUM>, the process <NUM> includes a glasses view rendering process <NUM>.

<FIG> and <FIG> relate to the glasses view rendering process <NUM>, which is configured to achieve dynamic occlusion effects in the glasses view <NUM>. Specifically, <FIG> is a flow diagram of an example implementation <NUM> of the glasses view rendering process <NUM> according to an example embodiment. In an example embodiment, the CPU, the GPU, or a combination thereof can perform this example implementation <NUM>. For instance, in an example embodiment, for speed, the GPU of the processing system <NUM> is configured to perform the glasses view rendering process <NUM>. In addition, <FIG> illustrate examples of certain aspects of the example implementation <NUM>.

At step <NUM>, in an example embodiment, the processing system <NUM> transforms the depth data from the video view <NUM> to the glasses view <NUM>. In an example embodiment, for instance, the transformation is obtained via calibration using software technology for AR applications, such as ARToolKit or other similar software programs. Due to the differences between the video view <NUM> and the glasses view <NUM>, empty regions (holes) might be created as illustrated in <FIG>. Here the curve <NUM> represents the object surface. Also in <FIG>, point p1 and point p2 are on the surface that projects to nearby points in video view <NUM>, and p1 is further than p2. In glasses view <NUM>, due to this view change, the point (or pixel) nearby p2 follows the ray R, for which there is no direct depth measurement in this case. One way to obtain the depth is via interpolation between point p1 and point p2, ending up with the point p4. However, this interpolation might be problematic for occlusion handling. In this regard, for instance, when a virtual object <NUM> is placed in a position, as shown in <FIG>, point p4 will occlude the virtual object <NUM>. Essentially, in this case, there is no information regarding the true depth along the ray R without any prior information. A safer way, which is also used for view synthesis, is to take the larger depth between point p1 and point p2 as the estimation, resulting in point p3 as shown in <FIG>. Accordingly, guided by this strategy, the processing system <NUM> performs a number of operations when transforming the scene depth from video view <NUM> to the glasses view <NUM> before depth testing in glasses view <NUM>.

At step <NUM>, in an example embodiment, the processing system <NUM> triangulates all or substantially all of the points (or pixels) on the image grid and renders the enhanced depth map as a triangular mesh to a depth texture.

At step <NUM>, in an example embodiment, during this rendering, the processing system <NUM> identifies the triangles with an edge longer than a certain threshold. As one non-limiting example, the threshold is <NUM>. In this regard, the points (or pixels) within these triangles correspond to the case illustrated in <FIG>.

At step <NUM>, in an example embodiment, the processing system <NUM> assigns these points (or pixels) with the maximum depth among the three end points of this triangle.

At step <NUM>, in an example embodiment, the processing system <NUM> renders the depths for dynamic occlusion handling. In this regard, for instance, the processing system <NUM> is configured to implement this process via appropriate software technology, such as OpenGL Shader or any other software program, and apply this process to both the left and right view of the glasses.

As discussed above, the process <NUM> is configured to leverage the data provided by RGB-D camera <NUM>. More specifically, the dynamic occlusion handling system <NUM> includes an edge-snapping algorithm that snaps (or moves) an object boundary of the raw depth data towards the corresponding color image and then enhances the object boundary of the depth map based on the edge-snapping results. This edge-snapping is particularly beneficial as the use of raw depth data may include holes, low resolutions, and significant noises around the boundaries, thereby introducing visual artifacts that are undesirable in various applications including AR. The enhanced depth maps are then used for depth testing with the virtual objects <NUM> for dynamic occlusion handling. Further, there are several AR applications that can benefit from this dynamic occlusion handling. As non-limiting examples, this dynamic occlusion handling can be applied to at least the following two AR use cases.

As a non-limiting example, a first AR use case involves an automotive repair application, where a user uses an AR system for guidance. In this example, the automotive repair application includes an AR scene <NUM> with a 3D printed dashboard as an example. In addition, the AR scene <NUM> includes virtual objects <NUM>, specifically a virtual touch screen and a windshield. For evaluation purposes, the following discussion includes positioning a user's hand <NUM> in different locations of the AR scene <NUM>. In some cases, the user's hand <NUM> should be occluded by the touch screen but not the windshield; while in others, the user's hand <NUM> should occlude both virtual objects <NUM>. Some of the example results are shown in <FIG>.

<FIG> illustrate visual results of different occlusion handling strategies in an AR-assisted automotive repair scenario. Specifically, <FIG> illustrate an instance in which the user's hand <NUM> should reside between two virtual objects <NUM> (e.g., the virtual touchscreen and the virtual windshield). <FIG> illustrates the visual results of virtual objects <NUM> in relation to the user's hand <NUM> without any occlusion handling. As shown in <FIG>, instead of residing between two virtual objects <NUM>, as desired, the user's hand <NUM> is improperly occluded by both virtual objects <NUM>. <FIG> illustrates the visual results with occlusion handling using raw depth data. As shown in <FIG>, the AR scene <NUM> suffers from defects, such as various visual artifacts, as indicated by the arrows <NUM>. In contrast, <FIG> illustrate the visual results of virtual objects <NUM> with dynamic occlusion handling using an enhanced depth map, as disclosed herein. As shown in <FIG>, the AR scene <NUM> includes a boundary for the user's hand <NUM>, which is better preserved and properly positioned in relation to the virtual objects <NUM>, when dynamic occlusion handling is performed with an enhanced depth map.

<FIG> illustrate an instance in which the user's hand <NUM> should occlude both virtual objects <NUM> (e.g., the virtual touchscreen and the virtual windshield). <FIG> illustrates the visual results of virtual objects <NUM> in relation to the user's hand <NUM>, without any occlusion handling. As shown in <FIG>, instead of residing in front of the virtual objects <NUM>, as desired, the user's hand <NUM> is improperly occluded by both virtual objects <NUM>. <FIG> illustrates the visual results of the virtual objects <NUM> in relation to the user's hand <NUM> with occlusion handling using raw depth data. As shown in <FIG>, the AR scene <NUM> suffers from defects, such as various visual artifacts, as indicated by the arrows <NUM>. In contrast, <FIG> illustrate the visual results of the virtual objects <NUM> in relation to the user's hand <NUM> with dynamic occlusion handling using an enhanced depth map, as disclosed herein. As shown in <FIG>, the AR scene <NUM> includes a boundary for the user's hand <NUM>, which is better preserved and properly positioned in relation to the virtual objects <NUM>, when dynamic occlusion handling is performed with an enhanced depth map.

<FIG> illustrate instances in which a user's hand <NUM> should occlude at least two virtual objects <NUM> (e.g., the virtual touchscreen and the virtual windshield). <FIG> illustrates the visual results of virtual objects <NUM> in relation to the user's hand <NUM>, without any occlusion handling. As shown in <FIG>, instead of residing in front of the virtual objects <NUM>, as desired, the finger of the user's hand <NUM> is improperly occluded by both virtual objects <NUM>. <FIG> illustrates the visual results of the virtual objects <NUM> in relation to the user's hand <NUM> with occlusion handling using raw depth data. In contrast, <FIG> illustrate the visual results of the virtual objects <NUM> in relation to the user's hand <NUM> with dynamic occlusion handling using an enhanced depth map, as disclosed herein. As shown in <FIG>, the AR scene <NUM> includes a boundary for the user's hand <NUM>, which is better preserved and properly positioned in relation to the virtual objects <NUM>, when dynamic occlusion handling is performed with an enhanced depth map.

As another non-limiting example, a second AR use case involves AR gaming. For instance, in a treasure hunting game with an AR system, the real scene serves as the playground while the virtual treasure chest is a virtual object <NUM> hidden somewhere in the real scene. More specifically, in this example, the virtual treasure chest is hidden behind a closet door <NUM> and behind a box <NUM>. Therefore, in this AR scene <NUM>, to be able to find the hidden virtual treasure chest, the user should open the closet door <NUM> and remove the box <NUM>.

However, in this treasure hunting game, without the appropriate dynamic occlusion handling, the virtual treasure chest will be visible to the user, ruining the entire gaming experience of finding the hidden virtual treasure chest. Using the raw depth data from the depth sensor, reasonable occlusion handling effects can be achieved. However, visual artifacts can also be observed in this AR scene <NUM> when raw depth data is used. Due to the occlusion between the closet door <NUM> and the box <NUM>, there are normally missing depth values along the boundaries. As the user opens the closet door <NUM>, visual artifacts can be observed. In contrast, by using dynamic occlusion handling with enhanced depth maps via the process <NUM>, the boundaries of the closet door <NUM> and the box <NUM> are snapped to their desired locations and the visual artifacts are removed.

<FIG> illustrate visual results of different occlusion handling strategies in an AR treasure hunting scenario. In this example, the virtual object <NUM> (e.g., treasure chest) should be positioned behind the box <NUM> in this AR scene <NUM>. More specifically, <FIG> illustrate visual effects without occlusion handling. In <FIG>, without any occlusion handling, the virtual object <NUM> occludes the box <NUM> and the closet door <NUM> and is therefore not positioned behind the box <NUM>, as intended. Meanwhile, in <FIG>, when applying occlusion handling on raw depth data, the virtual object <NUM> is correctly positioned between the closet door <NUM>, as intended, but incorrectly occludes the box <NUM>. That is, the raw depth data of the raw depth map <NUM> clearly misses some values in the circled regions <NUM> and thus the virtual object <NUM> incorrectly occludes the box in these circled regions <NUM>. In this case, as evidenced by <FIG>, when the virtual object <NUM> occludes the real scene in an unintended manner, then the AR scene <NUM> suffers from undesired artifacts. In contrast, <FIG> illustrate the visual effects with dynamic occlusion handling, as discussed herein, in which enhanced depth maps are used and contribute to the AR scene <NUM>. As shown, in each of the AR scenes <NUM> of <FIG>, the virtual object <NUM> is rendered behind both the box <NUM> and the closet door <NUM> in an appropriate manner and without any visual artifacts. That is, with dynamic occlusion handling, the user is provided with a proper and realistic AR experience.

<FIG> illustrate object boundaries in color images, raw depth maps, and enhanced depth maps. Specifically, each of <FIG> is a color image <NUM> with a ground-truth boundary <NUM> of the user's hand <NUM>. In this regard, each of <FIG> presents different hand gestures and/or background scene. Turning to <FIG>, these illustrations utilize, for instance, a standard JET color scheme, with the corresponding color images <NUM> overlaying their corresponding depth maps <NUM>. More specifically, <FIG> include the raw depth maps <NUM> while <FIG> include the enhanced depth maps <NUM>. As evidenced by a comparison of <FIG> with <FIG>, the object boundaries <NUM> of the hands <NUM> in the enhanced depth maps <NUM> correspond more closely to the ground-truth boundary <NUM> than the object boundaries <NUM> of the hands <NUM> in the raw depth maps <NUM>. That is, the enhanced depth maps <NUM> provide improved object boundaries <NUM>, thereby achieving dynamic occlusion handling that results in an improved AR experience.

Furthermore, <FIG> visualize the desired ground-truth boundary <NUM> of the hand <NUM> over the original color image <NUM>. Ideally, the object boundary in the depth maps should match this curve. However, as shown in <FIG>, the raw depth maps <NUM> suffer from various types of noises and missing values, resulting in mismatches with the ground-truth boundary <NUM>. For instance, in <FIG>, there is a hole in the palm region, creating a false object boundary. Meanwhile, <FIG> represent the results of example embodiments after depth map enhancement. As shown by the results of <FIG>, the process <NUM> improves the consistency of boundaries of objects between image data (e.g., RGB data) and depth data.

As discussed above, the system <NUM> provides dynamic occlusion handling, which enables accurate depth perception in AR applications. Dynamic occlusion handling therefore ensures a realistic and immersive AR experience. In general, existing solutions typically suffer from various limitations, e.g. static scene assumption or high computational complexity. In contrast, this system <NUM> is configured to implement a process <NUM> that includes a depth map enhancement process <NUM> for dynamic occlusion handling in AR applications. Advantageously, this system <NUM> implements an edge-snapping approach, formulated as discrete optimization, that improves the consistency of object boundaries between RGB data and depth data. In an example embodiment, the system <NUM> solves the optimization problem efficiently via dynamic programming. In addition, the system <NUM> is configured to run at an interactive rate on a computing platform, (e.g., tablet platform). Also, the system <NUM> provides a rendering strategy for glasses view <NUM> to avoid holes and artifacts due to interpolation that originate from differences between the video view <NUM> (data acquisition sensor) and the glasses view <NUM>. Furthermore, experimental evaluations demonstrate that this edge-snapping approach largely enhances the raw sensor data and is particularly suitable compared to several related approaches in terms of both speed and quality. Also, unlike other approaches that focus on the entire image, this process <NUM> advantageously focuses on the edge regions. Moreover, the system <NUM> delivers visually pleasing dynamic occlusion effects during user interactions.

As aforementioned, in an example embodiment, the system <NUM> is configured to perform edge-snapping between the depth maps and color images, primarily based on image gradients. Additionally or alternatively, when the characteristics of the sensor data from the depth sensor <NUM> provides raw depth edges that are close to the corresponding desired color edges, the system <NUM> is configured to model the color characteristic of individual objects for segmentation. Additionally or alternatively, the system <NUM> is configured to further enhance the above-mentioned energy function by taking into account other information besides image gradients, such as that of color distributions or other relevant data, to better accommodate complicated scenarios such as a cluttered scene. Additionally or alternatively, the system <NUM> can consider and include temporal information. Additionally or alternatively, the system <NUM> can include explicit tracking of moving objects to enhance the robustness of the edge-snapping framework.

Claim 1:
A computing system comprising:
a processing system (<NUM>) including at least one processing unit, the processing system (<NUM>) being configured to implement a method that includes:
receiving a depth map with a first boundary of an object ;
receiving a color image that corresponds to the depth map, the color image including a second boundary of the object;
extracting depth edge points of the first boundary from the depth map;
identifying candidates for the depth edge points along two-dimensional (2D) normal line segments in which each 2D normal line segment is perpendicular to a respective smoothed position of the depth edge point using smoothed positions of two neighboring depth edge points;
identifying target depth edge points on the depth map from among the corresponding candidates by minimizing an energy function based at least on a data term involving image gradient data from the color image and a smoothness term involving a smoothness constraint, and wherein the target depth edge points correspond to color edge points of the second boundary of the object in the color image; and
snapping the depth edge points towards the target depth edge points,
wherein the smoothness constraint for a pair of consecutive edge points pi and pj is defined by computing the cost of snapping pi onto ci,k and pj onto cj,l via following equation: <MAT>
wherein the candidate set of a depth edge point pi is denoted as Li = {ci,k | k = <NUM>,...,2rs +<NUM>}, where rs is a range in both the positive and negative direction of the 2D normal line segment of edge point pi and dmax defines the maximal discrepancy allowed for two consecutive depth edge points.