Patent Description:
Stereo matching is one of various methods of acquiring depth information from two-dimensional (2D) images. In stereo matching, corresponding points are detected from at least two images, and a depth of an object in the images is calculated based on the corresponding points. A stereo image includes images of the same object captured from different viewpoints, for example, from a left viewpoint corresponding to a left eye and a right viewpoint corresponding to a right eye. Corresponding points are points in at least two images of the stereo image that correspond to a same point of the object. Due to a characteristic of a binocular disparity, a great displacement between images in the stereo image occurs when the object is located close to a camera, and a small displacement between the images occurs when the object is located far away from the camera. A depth of the object in the image, that is, a distance from the camera to the object, is calculated based on a disparity, which is a difference in position, between points in an image of one viewpoint and corresponding points in an image of another viewpoint. The disparity is obtained, and the depth of the object is calculated based on the disparity.

Known from the art is for example the method disclosed in <NPL>.

In its abstract, this publication discloses that, leveraging on the recent developments in convolutional neural networks (CNNs), matching dense correspondence from a stereo pair has been cast as a learning problem, with performance exceeding traditional approaches. The authors propose a cascade CNN architecture composing of two stages. The first stage advances the recently proposed DispNet by equipping it with extra up-convolution modules, leading to disparity images with more details. The second stage explicitly rectifies the disparity initialized by the first stage; it couples with the first-stage and generates residual signals across multiple scales. The summation of the outputs from the two stages gives the final disparity. As opposed to directly learning the disparity at the second stage, the authors state that they show that residual learning provides more effective refinement. The authors also state that this benefits the training of the overall cascade network. They also state that experimentation shows that their cascade residual learning scheme provides state-of-the-art performance for matching stereo correspondence.

Also known from the art is the method to predict the correctness of stereo correspondences, also called confidence, and a confidence fusion method for dense disparity estimation, as disclosed in <NPL>.

In its abstract, this publication discloses that the input of this method consists in a two channels local window (disparity patch) which is designed by taking into account ideas of conventional confidence features. 1st channel is coming from the idea that neighboring pixels which have consistent disparities are more likely to be correct matching. In 2nd channel, a disparity from another image is considered such that the matches from left to right image should be consistent with those from right to left. The disparity patches are used as inputs of Convolutional Neural Networks so that the features and classifiers are simultaneously trained unlike what is done by existing methods. Moreover, the confidence is incorporated into Semi-Global Matching(SGM) by adjusting its parameters directly. According to the authors, they show the prominent performance of both confidence prediction and dense disparity estimation on KITTI datasets which are real world scenery.

In its abstract, this publication discloses that a traditional solution of area-based stereo uses some kind of windowed pixel intensity correlation. This approach suffers from discretization artifacts which corrupt the correlation value. According to the authors, they introduce a new correlation statistic, which is completely invariant to image sampling, moreover it naturally provides a position of the correlation maximum between pixels.

Also known from the art is the method disclosed in<NPL>.

In its abstract, this publication discloses that match points between stereo image pairs can be used to derive terrain elevation data from aerial photography and to determine obstacle and target ranges for camera guided vehicles. These measurements are made possible by the relative image offsets, or parallax, produced when objects at different ranges are imaged from different angles. In other image comparison applications, such as change detection, parallax may constitute a significant nuisance, producing undesired relative image distortions. Parallax removal through pixel by pixel image matching is then necessary before image comparison can be performed.

The summary below is provided to introduce a selection of concepts in a simplified form that are further described below.

In one aspect, a disparity estimation method performed by a processor according to accompanying claim <NUM> is provided. Further embodiments are provided in the dependent method claims.

A disparity estimation method performed by a processor includes extracting a first image patch including a reference pixel from a first image; extracting a second image patch including a target pixel corresponding to the reference pixel from a second image; and estimating a residual of an initial disparity between the reference pixel and the target pixel from the first image patch and the second image patch using a residual model, the residual being an estimated difference between the initial disparity and an actual disparity between the reference pixel and the target pixel.

The disparity estimation method may further include determining a final disparity between the reference pixel and the target pixel by correcting the initial disparity based on the estimated residual.

The disparity estimation method may further include calculating a depth corresponding to each of pixels of the first image and the second image based on the final disparity.

The disparity estimation method may further include estimating the initial disparity between the reference pixel of the first image and the target pixel of the second image.

The estimating of the initial disparity may include determining a search range in the second image; comparing a reference image patch including the reference pixel to each of candidate image patches respectively corresponding to pixels included in the search range; and determining the target pixel in the second image based on a result of the comparing.

The estimating of the residual may include extracting feature data from the first image patch and the second image patch using a feature model; and estimating the residual from the feature data using the residual model.

The disparity estimation method may further include estimating false disparity information of the initial disparity from the first image patch and the second image patch using a false disparity detection model.

The disparity estimation method may further include excluding the initial disparity from further use in response to determining that the initial disparity is false based on the estimated false disparity information.

The disparity estimation method may further include estimating the initial disparity in an integer-pixel unit, and the estimating of the residual may include estimating the residual in a sub-pixel unit.

The extracting of the first image patch may include extracting a feature point from the first image; and determining a pixel corresponding to the feature point in the first image as the reference pixel.

In another aspect, a non-transitory computer-readable medium according to accompanying claim <NUM> is provided, which stores instructions that, when executed by a processor, cause the processor to perform the disparity estimation method described above.

In another aspect, a disparity estimation apparatus according to accompanying claim <NUM> is provided. Further embodiments are provided in the dependent apparatus claims.

A disparity estimation apparatus includes an image acquirer configured to acquire a first image and a second image; and a processor configured to extract a first image patch including a reference pixel from the first image, extract a second image patch including a target pixel corresponding to the reference pixel from the second image, and estimate a residual of an initial disparity between the reference pixel and the target pixel from the first image patch and the second image patch using a residual model, the residual being an estimated difference between the initial disparity and an actual disparity between the reference pixel and the target pixel.

The processor is further configured to determine a final disparity between the reference pixel and the target pixel by correcting the initial disparity based on the estimated residual.

The processor may be further configured to calculate a depth corresponding to each of pixels of the first image and the second image based on the final disparity.

The processor may be further configured to estimate the initial disparity between the reference pixel of the first image and the target pixel of the second image.

The processor may be further configured to determine a search range in the second image, compare a reference image patch including the reference pixel to each of candidate image patches respectively corresponding to pixels included in the search range, and determine the target pixel in the second image based on a result of the comparing.

The processor may be further configured to extract feature data from the first image patch and the second image patch using a feature model, and estimate the residual from the feature data using the residual model.

The processor may be further configured to estimate false disparity information of the initial disparity from the first image patch and the second image patch using a false disparity detection model.

The processor may be further configured to exclude the initial disparity from further use in response to determining that the initial disparity is false based on the estimated false disparity information.

The processor may be further configured to estimate the initial disparity in an integer-pixel unit, and estimate the residual in a sub-pixel unit.

A disparity estimation method performed by a processor may include estimating an initial disparity having a first resolution between a reference pixel in a first image and a target pixel in a second image, the target pixel corresponding to the reference pixel; estimating a residual having a second resolution smaller than the first resolution from the first image and the second image using a residual model, the residual being an estimated difference between the initial disparity and an actual disparity between the reference pixel and the reference pixel; and correcting the initial disparity based on the residual to obtain a final disparity.

The estimating of the residual includes extracting a first image patch including the reference pixel from the first image; extracting a second image patch including the target pixel from the second image; and estimating the residual from the first image patch and the second image patch using the residual model.

The extracting of the first image patch includes extracting an image patch centered on the reference pixel from the first image as the first image patch, and the extracting of the second image patch includes extracting an image patch centered on the target pixel from the second image as the second image patch;.

The estimating of the initial disparity may include estimating the initial disparity in a single-pixel unit, and the estimating of the residual may include estimating the residual in a sub-pixel unit.

The disparity estimation method may further include estimating false disparity information indicating a probability that the initial disparity is false from the first image and the second image; excluding the initial disparity from further use in response to the false disparity information exceeding a false threshold level; and estimating the residual in response to the false disparity information not exceeding the false threshold level.

A method of training a residual model performed by a processor may include estimating an initial disparity from a first reference image of a stereo reference image and a second reference image of the stereo reference image; extracting a first reference image patch from the first reference image; extracting a second reference image patch from the second reference image; estimating a residual from the first reference image patch and the second reference image patch using a residual model implemented as a neural network; correcting the initial disparity based on the residual to obtain an estimated disparity; calculating a value of a loss function that is a function of a difference between a ground truth disparity for the first reference image and the second reference image and the estimated disparity; and training the neural network of the residual model to minimize the value of the loss function.

The method of training a residual model may further include extracting first feature data corresponding to a first feature point from the first reference image patch using a first feature model; extracting second feature data corresponding to a second feature point from the second reference image patch using a second feature model, the second feature point corresponding to the first feature point; and concatenating the first feature data and the second feature date to obtain concatenated feature data, wherein the estimating of the residual may include inputting the concatenated feature data into the residual model to cause the residual model to output the residual.

The method of training a residual model may further include estimating false disparity information indicating a probability that the initial disparity is false from the first reference image and the second reference image using a false disparity detection model, wherein the loss function is a single loss function that is a function of both the difference between the ground truth disparity and the initial disparity, and a cross-entropy error of reference false disparity information indicating a probability that the initial disparity is false and the estimated false disparity information, the calculating of the value of the loss function may include calculating a value of the single loss function, and the training of the neural network of the residual model may include training both the neural network of the residual model and the false disparity detection model to minimize the value of the single loss function.

The method of training a residual model may further include extracting first feature data corresponding to a first feature point from the first reference image patch using a first feature model; extracting second feature data corresponding to a second feature point from the second reference image patch using a second feature model, the second feature point corresponding to the first feature point; and concatenating the first feature data and the second feature date to obtain concatenated feature data, wherein the estimating of the residual may include inputting the concatenated feature data into the residual model to cause the residual model to output the residual, and the estimating of the false disparity information may include inputting the concatenated feature data into the false disparity detection model to cause the false disparity detection model to output the estimated false disparity information.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosure of this application pertains based on an understanding of the disclosure of this application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of this application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

As will become apparent from the following description, this application describes examples of a disparity estimation method performed by a processor, and a processor-implemented disparity estimation apparatus, having an improved performance resulting at least in part from estimating an initial disparity by performing stereo matching between two images of a stereo image captured by a stereo camera, estimating a residual that is an estimated difference between the initial disparity and an actual disparity between the two images using a residual model, and correcting the initial disparity based on the residual to obtain a final disparity between the two images having an increased accuracy. The residual model may be a neural network trained by machine learning based on reference images and reference residuals corresponding to the reference images. The initial disparity is estimated in a single-pixel unit, and the residual is estimated in a sub-pixel unit. The increased accuracy of the final disparity enables calculation of a more accurate depth of a point in the stereo image corresponding to the reference pixel and the target pixel.

<FIG> is a block diagram illustrating an example of a configuration of a disparity estimation apparatus.

Referring to <FIG>, a disparity estimation apparatus <NUM> includes an image acquirer <NUM> and a processor <NUM>.

The image acquirer <NUM> acquires an image of an environment of the disparity estimation apparatus <NUM>. For example, the image acquirer <NUM> acquires a first image and a second image of the environment. In one example, the image acquirer <NUM> is a stereo camera, and the first image and the second image are a left image and a right image, respectively, but the image acquirer <NUM>, the first image, and the second image are not limited thereto.

The processor <NUM> estimates an initial disparity between the first image and the second image, and estimates a residual of the estimated initial disparity.

In the following description, a disparity is a difference in position between an arbitrary point (for example, a reference pixel) of a reference image and a corresponding point (for example, a target pixel) of a target image corresponding to the reference image. For example, the processor <NUM> selects the first image as a reference image and selects the second image as a target image. A reference pixel is a pixel selected from the first image for use in determining a disparity. A target pixel is a pixel selected from the second image that corresponds to the reference pixel. In other words, the reference pixel and the target pixel correspond to the same point viewed from different viewpoints.

The residual is a value obtained by estimating an error of the initial disparity. That is, the residual is an estimated difference between the initial disparity and an actual disparity. In one example, the processor <NUM> calculates a final disparity by adding the residual to the initial disparity.

<FIG> is a block diagram illustrating another example of a configuration of a disparity estimation apparatus.

Referring to <FIG>, a disparity estimation apparatus <NUM> includes a memory <NUM> in addition to an image acquirer <NUM> and a processor <NUM>.

The image acquirer <NUM> performs the same operation described with respect to <FIG>. In one example, the image acquirer <NUM> includes a first camera <NUM> and a second camera <NUM>. The first camera <NUM> and the second camera <NUM> are spaced apart from each other by a predetermined distance known as a baseline. The first camera <NUM> generates a first image by capturing an image of an environment of the disparity estimation apparatus <NUM>, and the second camera <NUM> generates a second image by capturing an image of the environment of the disparity estimation apparatus <NUM> at a position spaced apart from the first camera <NUM> by the predetermined distance or baseline. That is, the first camera <NUM> and the second camera <NUM> capture the first image and the second image from different viewpoints. In one example, when the first camera <NUM> and the second camera <NUM> are horizontally spaced apart from each other by a predetermined distance, the first camera <NUM> generates a left image and the second camera <NUM> generates a right image, but the first camera <NUM> and the second camera <NUM> are not limited thereto. In another example, when the first camera <NUM> and the second camera <NUM> are vertically spaced apart from each other, the first image is an upper image and the second image is a lower image.

The processor <NUM> extracts a first image patch including a reference pixel from the first image, and extracts a second image patch including a target pixel corresponding to the reference pixel from the second image. For example, the processor <NUM> extracts a rectangular portion of the first image as the first image patch. Also, the processor <NUM> extracts a rectangular portion of the second image as the second image patch. The first image patch and the second image patch have the same size and the same resolution.

In the following description, an image patch is a partial image obtained by cropping an entire image. For example, the first image patch is a partial image obtained by cropping the first image, and the second image patch is a partial image obtained by cropping the second image.

Also, the processor <NUM> estimates an initial disparity between the reference pixel and the target pixel from the first image and the second image, and then estimates a residual of the initial disparity from the first image patch and the second image patch using a residual model.

In the following description, the residual model is a model that outputs an error of an initial disparity estimated from two images. The residual model is, for example, a model that has been trained by machine learning. The residual model includes parameters of a machine learning structure. For example, when a neural network is used as the machine learning structure, the residual model includes connection weights between nodes in the neural network.

For example, the residual model includes parameters of a machine learning structure (for example, a neural network) that has been trained to output a reference residual (that is, a difference between a reference disparity and an initial disparity estimated from a reference image pair) corresponding to a reference image patch pair extracted from the reference image pair in response to an input of the reference image patch pair. The reference disparity is an actual disparity between the two reference images of the reference image pair. Training data used to train the machine learning structure includes the reference image patch pair and the reference residual. The reference residual is a residual provided as a ground truth for the reference image patch pair. However, training of the residual model is not limited thereto. An example of a process of training a residual model together with a false disparity detection model using a single loss function will be described below with reference to <FIG>.

The memory <NUM> stores the residual model. Also, the memory <NUM> temporarily stores data used to estimate a residual using the residual model.

The disparity estimation apparatus <NUM> estimates an initial disparity from the first image and the second image acquired from the first camera <NUM> and the second camera <NUM> of a stereo camera, estimates a residual of the initial disparity, and corrects the initial disparity based on the residual to determine a final disparity having a relatively high accuracy. The disparity estimation apparatus <NUM> estimates the residual using the residual model, and thus it is possible to determine a more accurate final disparity than can be determined by a conventional method of estimating a final disparity based on a loss curve fitted between the two lowest values of a loss function without regard to a feature of an image.

<FIG> illustrates an example of a disparity estimation process.

Referring to <FIG>, a processor of a disparity estimation apparatus receives a first image <NUM> and a second image <NUM> from an image acquirer as described above. In the following description, for convenience of description, the first image <NUM> and the second image <NUM> are referred to as a left image and a right image, respectively, but are not limited thereto.

The processor estimates an initial disparity between the first image <NUM> and the second image <NUM>. The estimating of the initial disparity between the first image <NUM> and the second image <NUM> is referred to as "stereo matching" <NUM>. For example, stereo matching is an operation of comparing image information, for example, intensities or colors of pixels, to find corresponding points, or comparing image information, for example, intensities or colors, of image patches surrounding a central pixel. The processor uses a multi-block-matching (MBM) scheme to estimate the initial disparity between the first image <NUM> and the second image <NUM>. For example, the processor estimates an initial disparity between a reference pixel of the first image <NUM> and a target pixel of the second image <NUM>.

The processor compares a first image patch <NUM> including a reference pixel of the first image <NUM> to a search range <NUM> of the second image <NUM> to determine a second image patch <NUM> including a target pixel, and estimate an initial disparity corresponding to a difference in position between the first image patch <NUM> and the second image patch <NUM>. The processor estimates an initial disparity for pixels corresponding to at least one object represented in the first image <NUM>. For example, the processor extracts feature points from the first image <NUM>. The feature points are points corresponding to features of the first image <NUM>, and are pixels corresponding to a portion of an object (for example, a vehicle of <FIG>) represented in the first image <NUM>. The processor determines, as a reference pixel, a pixel corresponding to a feature point extracted from the first image <NUM>. For example, in <FIG>, the processor extracts, as a feature point, a central point of an object area corresponding to the vehicle detected from the first image <NUM>, but is not limited thereto. Accordingly, in another example, the processor also extracts, as feature points, in addition to the central point of the object area, at least some points or all points of the object area from the first image <NUM>.

Also, the processor determines the search range <NUM> in the second image <NUM>. The processor determines the search range <NUM> based on the first image patch <NUM> including the reference pixel of the first image <NUM>. For example, the processor determines, as the search range <NUM> in the second image <NUM>, an area having upper and lower boundaries at the same height as upper and lower boundaries of the reference image patch <NUM> of the first image <NUM>.

The processor compares a reference image patch including the reference pixel with each of candidate image patches respectively corresponding to candidate pixels included in the search range <NUM>. For example, a candidate pixel is a pixel at the same height as the reference pixel in the search range <NUM> of the second image <NUM>. Although some of the pixels at the same height as the reference pixel are determined as candidate pixels in the second image <NUM> as shown in <FIG>, in another example, all of the pixels at the same height as the reference pixel are determined to be candidate pixels. A candidate image patch is a partial image obtained by cropping the search range <NUM> using a candidate pixel as a central point of the candidate image patch. The reference image patch and the candidate image patch have the same size and the same resolution. In the following description, the reference image patch is an image patch used as a criterion of the stereo matching <NUM>, and the candidate image patches are image patches selected for comparison to the reference image patch.

The processor determines a target pixel in the second image <NUM> based on a result of a comparison between the candidate image patch and the reference image patch. For example, the processor compares each of the candidate image patches to the reference image patch, and calculates a similarity between each of the candidate image patches and the reference image patch based on a result of the comparison. The processor determines, as a target pixel, a pixel (for example, a central point of a candidate image patch) corresponding to the candidate image patch having a highest similarity to the reference image patch in the search range <NUM>.

The processor crops each of the first image <NUM> and the second image <NUM> into image patches, which is referred to as "image patch cropping" <NUM>. For example, the processor extracts the first image patch <NUM> including the reference pixel from the first image <NUM>. Also, the processor extracts the second image patch <NUM> including the target pixel from the second image <NUM>.

The processor estimates a residual from the first image patch <NUM> and the second image patch <NUM> using a residual model <NUM>. The processor inputs the first image patch <NUM> and the second image patch <NUM> to the residual model <NUM>. The residual model <NUM> is trained to output a reference residual in response to an input of a reference image patch pair. The processor inputs the first image patch <NUM> and the second image patch <NUM> in a form of feature data to the residual model <NUM>, and estimates a residual from the feature data using the residual model <NUM>. For example, the processor extracts feature data from a first image patch and a second image patch using a feature model. An example of a feature model will be described below with reference to <FIG>.

The processor inputs the first image patch <NUM> and the second image patch <NUM> to the residual model <NUM>, and the residual model <NUM> outputs the residual in a sub-pixel unit. An example of a sub-pixel unit will be described below with reference to <FIG>.

The processor corrects the initial disparity based on the residual to determine a final disparity in operation <NUM>. For example, the processor calculates a final disparity <NUM> by adding the residual to the initial disparity.

<FIG> illustrates an example of a process of estimating false disparity information in addition to the disparity estimation process of <FIG>.

The processor calculates the final disparity <NUM> as shown in <FIG>, and estimates false disparity information <NUM> based on the first image patch <NUM> and the second image patch <NUM>.

For example, the processor estimates the false disparity information <NUM> of the initial disparity from the first image patch <NUM> and the second image patch <NUM> using a false disparity detection model <NUM>. The false disparity information <NUM> is information associated with a false initial disparity, and includes, for example, a probability that an initial disparity is false.

The false disparity detection model <NUM> is a model trained to output reference false disparity information in response to an input of a reference image patch pair. The reference false disparity information is information generated in advance, and includes, for example, a probability that an initial disparity estimated from the reference image patch pair is false.

The processor excludes the initial disparity from further use in response to determining that the initial disparity is false based on the estimated false disparity information <NUM>.

In one example, when the estimated false disparity information <NUM> exceeds a threshold false level, the processor excludes the initial disparity from further use. The threshold false level is a level used as a criterion of falseness, and includes, for example, a probability used as a criterion to determine whether an initial disparity is false.

In another example, when the estimated false disparity information <NUM> is less than or equal to the threshold false level, the processor continues to estimate a residual based on the initial disparity.

<FIG> illustrates an example of a process of training a residual model and a false disparity detection model.

Referring to <FIG>, a residual model <NUM> and a false disparity detection model <NUM> are designed with a machine learning structure to share feature models <NUM> and <NUM> (for example, feature extraction networks, which may be neural networks). The machine learning structure is described below.

A processor of a disparity estimation apparatus estimates an initial disparity Dinit from an input image pair of a first reference image <NUM> and a second reference image <NUM> in operation <NUM>, and performs image patch cropping <NUM> to crop each of the first reference image <NUM> and the second reference image <NUM> into image patches based on the initial disparity Dinit.

The processor extracts feature data from each of the image patches using the feature models <NUM> and <NUM>, and performs a concatenate operation <NUM> to generate concatenated feature data by concatenating the extracted feature data. For example, a the concatenate operation <NUM> may be implemented by a concatenated model, which may be a neural network including, for example, a fully connected layer including nodes connected to output nodes of the feature models <NUM> and <NUM>. For example, the feature models <NUM> and <NUM> are trained to extract low-level features from the image patches. The low-level features are represented by image feature data expressed by, for example, any one or any combination of any two or more of a point, a line, a texture, and a color.

The processor inputs the concatenated feature data to the residual model <NUM> and the false disparity detection model <NUM>, and estimates a residual Δd using the residual model <NUM> and false disparity information lfalse using the false disparity detection model <NUM>.

A training apparatus trains the machine learning structure configured as described above. For example, the training apparatus trains the machine learning structure of <FIG> to output a reference disparity DGT corresponding to a first reference image <NUM> and a second reference image <NUM> in response to an input of a pair of the first reference image <NUM> and the second reference image <NUM> (hereinafter referred to as a "reference image pair"). The training apparatus trains the feature models <NUM> and <NUM>, the residual model <NUM>, and the false disparity detection model <NUM> based on a single loss function <NUM>. For example, when "n" feature points are extracted from the first reference image <NUM>, the training apparatus expresses the single loss function <NUM> as L using Equations <NUM> and <NUM> below. In this example, n is an integer greater than or equal to "<NUM>. " <MAT> <MAT>.

PiEST denotes a probability that an initial disparity Dinit for an i-th feature point is true, and i is an integer greater than or equal to "<NUM>" and less than or equal to "n. " LFID denotes a false initial disparity label. LFID is given for an initial disparity Dinit in a training set that the training apparatus uses to train the machine learning structure. More specifically, LFID indicates whether a given initial disparity is true or false. Therefore, Prob(LFID) denotes a probability that an initial disparity Dinit is false, and <NUM>-Prob(LFID) denotes a probability that the initial disparity Dinit is true. ∥ ∥<NUM> denotes the L<NUM> norm, also called the Euclidean norm. Also, CE denotes a cross-entropy loss between LGT and LEST, and α denotes a constant. DiGT denotes a reference disparity for the i-th feature point. DiEST denotes an estimated disparity obtained by correcting the initial disparity Dinit for the i-th feature point using the residual Δd in an arbitrary cycle during training. LGT denotes reference false disparity information, and LEST denotes false disparity information estimated in an arbitrary cycle during training.

In one example in which there is a relatively high probability that the initial disparity Dinit is true, the training apparatus trains the feature models <NUM> and <NUM>, the concatenated model implementing the concatenate operation <NUM>, and the residual model <NUM> so that the reference disparity DiGT is equal to the estimated disparity DiEST based on <MAT> of Equation <NUM>. In another example in which there is a relatively high probability that the initial disparity Dinit is false, the training apparatus performs training by excluding the estimated disparity DiEST from further use based on <MAT>. The training apparatus uses the estimated disparity DiEST for a training process only when a probability that the initial disparity Dinit is true is high, and thus it is possible to reduce an estimation error when the reference disparity DiGT and the estimated disparity DiEST are similar to each other.

The training apparatus trains the false disparity detection model <NUM> to output a probability that the initial disparity Dinit is false based on α·CE(LGT,LEST) of Equation <NUM> corresponding to a cross-entropy.

Also, the training apparatus uses the constant α to correct for a difference in a range of values between two losses, for example, <MAT> and α·CE(LGT,LEST). In other words, the training apparatus uses the constant α to balance the two losses.

<FIG> is a flowchart illustrating an example of a disparity estimation method.

Referring to <FIG>, in operation <NUM>, a processor of a disparity estimation apparatus extracts a first image patch including a reference pixel from a first image. For example, the processor extracts a feature point from the first image and determines a pixel corresponding to the feature point as a reference pixel. The feature point is, for example, at least a portion of an area corresponding to an object in the first image.

In operation <NUM>, the processor extracts a second image patch including a target pixel corresponding to the reference pixel from a second image. For example, the processor determines the target pixel of the second image corresponding to the reference pixel of the first image through stereo matching. Also, the processor extracts the second image patch based on the target pixel. For example, the processor extracts the second image patch using the target pixel as a central point of the second image patch.

In operation <NUM>, the processor estimates a residual of an initial disparity between the reference pixel and the target pixel from the first image patch and the second image patch using a residual model. For example, the processor extracts feature data from the first image patch and the second image patch and inputs the extracted feature data to the residual model, and the residual model outputs the residual of the initial disparity based on the input feature data.

<FIG> illustrates an example of a process of calculating a depth based on a corrected disparity.

In <FIG>, a first image and a second image are a left image <NUM> and a right image <NUM>, respectively, but are not limited thereto. A design may vary depending on an arrangement of cameras included in a disparity estimation apparatus.

In operation <NUM>, the disparity estimation apparatus estimates an initial disparity in an integer-pixel unit. For example, the disparity estimation apparatus estimates the initial disparity by performing stereo matching based on the left image <NUM> and the right image <NUM> that are output from a stereo camera.

In operation <NUM>, the disparity estimation apparatus crops the left image <NUM> into left image patches and crops the right image <NUM> into right image patches based on the initial disparity.

In operation <NUM>, the disparity estimation apparatus estimates a residual in a sub-pixel unit from the left image patches and the right image patches using a residual model loaded from a database (DB) <NUM>. The residual model has, for example, a deep neural network structure.

In operation <NUM>, the disparity estimation apparatus corrects the initial disparity based on the residual. For example, the disparity estimation apparatus calculates a final disparity by adding the residual to the initial disparity.

In operation <NUM>, a processor of the disparity estimation apparatus calculates a depth <NUM> corresponding to each of pixels in the left image <NUM> and the right image <NUM> based on the final disparity.

The disparity estimation apparatus repeatedly performs operations <NUM> through <NUM> for all pixels extracted as feature points from the left image <NUM>. The disparity estimation apparatus calculates a depth <NUM> of each of all of the pixels extracted from the left image <NUM>. The calculated depth <NUM> is used for modeling an object in each of the left image <NUM> and the right image <NUM> as a three-dimensional (3D) shape.

<FIG> illustrates an example of detecting a false disparity in addition to the process of <FIG>.

When an initial disparity is similar to a ground truth disparity, the disparity estimation apparatus accurately calculates a final disparity and a depth in <FIG>. When the initial disparity is greatly different from the ground truth disparity, an accuracy of estimation of a residual decreases. To prevent a decrease in the accuracy, the disparity estimation apparatus estimates false disparity information <NUM> associated with the initial disparity based on a difference between the initial disparity and the ground truth disparity.

In operation <NUM>, the disparity estimation apparatus estimates the false disparity information <NUM> from the left image patches and the right image patches using a false disparity detection model. The false disparity detection model has, for example, a neural network structure. Because the false disparity information <NUM> indicates a probability that the initial disparity is false, the false disparity information <NUM> corresponds to a reliability level of the calculated depth <NUM>. For example, when the false disparity information <NUM> exceeds a threshold false level, the disparity estimation apparatus excludes the initial disparity from further use. In this example, when the left image <NUM> and the right image <NUM> are in a current frame of consecutive frames, the processor suspends calculating a depth in the current frame and resumes calculating a depth in a next frame.

<FIG> illustrates an example in which a disparity estimation apparatus estimates a disparity in a sub-pixel unit.

A processor of the disparity estimation apparatus estimates an initial disparity in an integer-pixel unit, and estimates a residual in a sub-pixel unit. In the following description, the integer-pixel unit is a unit defined by a single pixel, and the sub-pixel unit is a unit smaller than a pixel, and is, for example, a real number corresponding to a ratio of an arbitrary distance to a single pixel. That is, the initial disparity is expressed as an integer number of pixels, and the residual is expressed as a real number of pixels. The residual may be expressed as a decimal number, and may be positive or negative, and may be less than <NUM>, equal to <NUM>, or greater than <NUM>.

Referring to <FIG>, for convenience of description, each of a first image <NUM> and a second image <NUM> includes <NUM> × <NUM> pixels. Also, objects <NUM> and <NUM> are represented in the first image <NUM> and the second image <NUM>, respectively. The processor estimates an initial disparity between a reference point <NUM> of the first image <NUM> and a target point <NUM> of the second image <NUM> through stereo matching <NUM>. The initial disparity between the reference point <NUM> and the target point <NUM> is, for example, "<NUM>" in an integer-pixel unit. Also, a residual <NUM> estimated from a first image patch cropped from the first image <NUM> and a second image patch cropped from the second image <NUM> is, for example, "-<NUM>" in a sub-pixel unit. The processor determines a final disparity to be "<NUM>" by correcting the initial disparity based on the residual <NUM>, for example, by adding the residual to the initial disparity to obtain the final disparity, i.e., <NUM> + (-<NUM>) = <NUM>. Thus, the disparity estimation apparatus determines a final point <NUM> of the second image <NUM> corresponding to the reference point <NUM>. However, the above-described values of the initial disparity, the final disparity, and the residual <NUM> are merely examples.

According to the examples described above, a disparity estimation apparatus performs stereo matching to find corresponding points in a left image and a right image for use in measuring a depth using a stereo camera. A processor of the disparity estimation apparatus estimates an initial disparity in an integer-pixel unit through the stereo matching. However, when a depth is calculated based on the initial disparity estimated in an integer-pixel unit, a depth resolution depends on a physical size of a pixel in a stereo image. The disparity estimation apparatus estimates a residual in a sub-pixel unit using a residual model, and thus it is possible to more precisely estimate a final disparity and a depth because the greater the resolution of the final disparity, the greater the resolution of the depth. As described above, a precisely estimated depth is used for more precise modeling of a 3D shape.

Also, the disparity estimation apparatus described above efficiently and precisely estimates a depth of each of pixels in a stereo image. The disparity estimation apparatus is implemented as, for example, a depth sensor for vehicles or a mobile depth sensor.

The disparity estimation apparatus <NUM>, the image acquirer <NUM>, and the processor <NUM> in <FIG>, the disparity estimation apparatus <NUM>, the image acquirer <NUM>, the first camera <NUM>, the second camera <NUM>, the processor <NUM>, and the memory <NUM> in <FIG>, the residual model <NUM> in <FIG>, the residual model <NUM> and the false disparity detection model <NUM> in <FIG>, the feature models <NUM> and <NUM>, the concatenated model, the residual model <NUM>, and the false disparity detection model <NUM> in <FIG>, the database (DB) <NUM> in <FIG>, and the database (DB) <NUM> in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A disparity estimation method performed by a processor, the disparity estimation method comprising:
estimating (<NUM>) an initial disparity between a reference pixel in a first image (<NUM>) and a target pixel in a second image (<NUM>), wherein the initial disparity is expressed as an integer number of pixels, comprising:
determining a search range in the second image (<NUM>);
comparing a reference image patch comprising the reference pixel to each of candidate image patches respectively corresponding to pixels included in the search range; and
determining the target pixel in the second image based on a result of the comparing;
extracting (<NUM>) a first image patch comprising the reference pixel as a central point from the first image (<NUM>);
extracting (<NUM>) a second image patch comprising the target pixel as a central point from the second image (<NUM>), wherein the target pixel corresponds to the reference pixel; and
estimating (<NUM>, <NUM>) a residual of the initial disparity between the reference pixel and the target pixel from the first image patch and the second image patch using a residual model trained by machine learning based on reference images and reference residuals corresponding to the reference images, the residual being an estimated difference between the initial disparity and an actual disparity between the reference pixel and the target pixel, wherein the residual has a sub-pixel unit being a unit having a size smaller than a pixel;
correcting (<NUM>) the initial disparity based on the residual to obtain a final disparity.