Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detection of pixel correspondences in stereo imaging through the use of polarization data. One of the methods include: obtaining a first image from a first viewpoint of a stereo pair and a second image from a second viewpoint of the stereo pair; obtaining an angle of linear polarization (AOLP) map and a degree of linear polarization (DOLP) map having a field of view overlapping the first image and the second image; computing a surface normal map based on the AOLP map and the DOLP map; and detecting corresponding pixels between the first image and the second image by computing a disparity map that minimizes an energy function comprising a pixel matching cost term and a polarized smoothness regularization term computed based on pixels in a local neighborhood and based on the surface normal map.

FIELD

Aspects of embodiments of the present disclosure relate to the field of image processing and computer vision.

BACKGROUND

Detecting corresponding pixels in pairs of images is useful in computer vision for performing, for example, three-dimensional (3D) reconstruction of a scene. In this context, a first pixel in a first image and a second pixel in a second image are considered to be corresponding pixels when they depict the same point or portion of a real-world surface. As a concrete example, in a stereo pair of left and right images captured of a scene that includes a pencil, a first pixel in the left image depicting the tip of the pencil is said to correspond to a second pixel in the right image that also depicts the tip of the pencil.

Three-dimensional reconstruction can be performed using this stereo pair based on finding corresponding pixels between the first and second scene and through triangulation. This technique is related to depth perception by humans using stereo vision and binocular disparity or parallax, where the two eyes provide different views on a scene. Closer objects exhibit greater parallax shift than distant objects, hence the distance to (or depth of) an object is inferred based on the degree of parallax shift between the views provided by the two eyes. In a similar manner, detecting the disparity in corresponding pixels between images captured by a stereo pair of cameras provides a computer vision system with a manner for detecting the depth of an object or a surface in a scene imaged by the stereo pair.

SUMMARY

This specification describes a system implemented by one or more computers that improves the detection of pixel correspondences in stereo imaging through the use of polarization data. Leveraging polarization, the system can more accurately detect correspondences, e.g., in terms of disparities, between pixels in a first image and pixels in a second image in a pair of stereo images.

Given a pair of polarized stereo images, the techniques described in this specification can more accurately detect pixel correspondences between the two images in the pair by considering polarization cues. By way of example, considering polarization cues can improve the accuracy of detected pixel correspondence for non-fronto-parallel scenes as well as the robustness of the detected pixel correspondence to noise. A computer vision system implementing the techniques described in this specification can more reliably generate a three-dimensional reconstruction or another digital representation of a scene in an environment from polarized stereo images, while being easily configurable to additionally generate outputs for a variety of perception tasks, e.g., object detection and/or classification, pose estimation, semantic segmentation, image reconstruction, and the like. As a result, improved performance of robots on various tasks can thus be achieved by virtue of the quality improvement in the vision data that includes such 3-D reconstructions and/or perception task outputs to be processed by a control system of the robots when controlling the robots.

DETAILED DESCRIPTION

This specification describes a system implemented by one or more computers that receives a pair of polarized stereo images and generates a disparity map for a first polarized stereo image and a second polarized stereo image in the pair. The disparity map can then be used for a variety of purposes or applications. In some examples, the disparity map can be used to find the correspondence between pixels in the first image and the second image in the pair; the disparity map can be used to estimate a depth map of an environment; the disparity map can also be used to reconstruct a three-dimensional (3-D) scene of the environment.

Stereo camera systems are one subset of digital camera devices for capturing 3-D content. Stereo camera systems capture images of a scene from two or more camera modules that may be spaced apart from each other along a baseline. The baseline is typically horizontal, similar to binocular vision in a human, such that images captured by a stereo pair are often referred to as a left image and right image. However, stereo camera systems are not limited thereto and the baseline may be oriented vertically or along any other direction.

Some stereo camera systems are polarized stereo camera systems. Each camera module in a polarized stereo camera may include a polarizer or polarizing filter or polarization mask placed in the optical path between a scene and an image sensor of the camera module. Polarization imaging provides significant additional information about the scene that would not otherwise be available using a standard camera system (i.e., a camera system lacking a polarizing filter). As one example, shape from polarization (SfP) provides techniques for using polarization signatures to compute the surface normals (e.g., the directions in which surfaces of objects are facing) using the Fresnel equations. Some commercially available sensors, such as the IMX250MZR polarization image sensor available from Sony® Corporation, are examples of the polarized stereo camera systems.

FIG.1shows an example computer vision system100. The computer vision system100is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The computer vision system100obtains a first polarized stereo image102and a second polarized stereo image104and generates a disparity map122for the images102and104in the pair by using polarization cues. The first polarized stereo image102and a second polarized stereo image104may be taken from slightly different viewpoints by a polarized stereo camera system. In some cases, each polarized stereo image is composed of multiple (e.g., three or more) polarization raw frames captured by the polarized stereo camera system that has differently oriented polarizing filters in their optical paths. That is, the multiple polarization raw frames corresponding to the polarized stereo image are captured of a same scene with polarizing filters set at different angles of linear polarization (e.g., 0°, 60°, and 120°; or 0°, 45°, 90°, and 135°),

The computer vision system100can obtain the images102and104from the polarized stereo camera system in any of a variety of ways. In some cases, the computer vision system100and the polarized stereo camera system may both be part of the same stand-alone system (that is, a self-contained device capable of operating without further hardware). For example, an FPGA or other processor may be integrated as an embedded device into a virtual-reality headset or robot equipped with a polarized stereo camera. Similarly, a laptop, tablet, smart phone, or other personal computing device may have an embedded polarized stereo camera. In these cases, an internal device bus or another interface/connector may transport the camera output signals from the camera(s) to the processor(s) of the stand-alone system. In some other cases, the computer vision system100may be provided as one or more devices separate from the polarized stereo camera system, and the image data may be transmitted via a wired or wireless communication link. For example, a polarized stereo camera system may send its camera output signals via a cable (e.g., a USB cable or an Ethernet cable) or via Bluetooth to a nearby (e.g., desktop, laptop, or tablet) computer. As another example, polarized stereo image pairs may be uploaded from the polarized stereo camera system to the Internet, from which the computer vision system100may download them via a wired or wireless network connection.

The computer vision system100also obtains one or more angle of linear polarization (AOLP) maps106and one or more degree of linear polarization (DOLP) maps108. As used in this specification, an AOLP (or DOLP) map refers to a digital representation of features of an image in a polarization representation space. In some cases, the computer vision system100can similarly obtain the AOLP map(s)106and the DOLP map(s)108from the polarized stereo camera system, e.g., as metadata that accompanies the first and second images102and104, while in other cases, the computer vision system100can receive the AOLP map(s)106and the DOLP(s) map108from a different source, e.g., from a polarization signature computation engine that may be implemented either local to or remote from the computer vision system100.

The AOLP map(s)106and the DOLP map(s)108may each have a field of view overlapping the fields of view of the first polarized stereo image102and the second polarized stereo image104. For example, the computer vision system100can obtain a first AOLP map and a first DOLP map that are computed from the viewpoint of the first polarized stereo image102, and can additionally or alternatively obtain a second AOLP map and a second DOLP map that are computed from the viewpoint of the second polarized stereo image104.

Angle of linear polarization (AOLP) and degree of linear polarization (DOLP), which describe the physical information of polarization, may be defined in terms of Stokes parameters. The Stokes parameters are a set of values that describe the polarization state of electromagnetic radiation well-known in physics. The Stokes parameters I, Q, U, and/or V may be determined from measurements of the scattered light, where I is the total radiance of light, Q and U describe linearly polarized radiation, and V describes the circularly polarized radiation. The degree of linear polarization (DOLP) indicates the ratio of the intensity of the polarized to the intensity of the unpolarized part of a light ray hitting the image sensor of a camera module. The angle of linear polarization (AOLP) indicates the direction of that linear polarization.

A surface normal engine110of the computer vision system100uses the polarization features represented by the AOLP map(s)106and the DOLP map(s)108and, optionally, additional feature information extracted from the images102and104to generate a surface normal map112. The surface normal engine110can for example compute the surface normals from the polarization features by using the Fresnel equations. The surface normal map112is a digital representation that indicates the directions in which surfaces of objects in an image are facing. For example, the surface normal map112may include, for each of one or more pixels in the first polarized stereo image102or the second polarized stereo image104, the coordinates of the surface normal, or vector which is perpendicular to the surface. Because the AOLP map(s)106and the DOLP map(s)108provide additional polarization feature information such as information about the object plane (surface normal) that reflects the incoming light, using the AOLP map(s) and the DOLP map(s) to generate the surface normals is generally more accurate than interpolating such normal vectors merely from images.

From (i) the first image102and the second image104, (ii) the AOLP map106and the DOLP map108, (iii) the surface normal map112, or some combinations of (i)-(iii), the computer vision system100uses a polarized semi-global matching (SGM) engine120to generate a disparity map122. The disparity map122is a digital representation that indicates a disparity between the first polarized stereo image102and the second polarized stereo image104. Disparity refers to the difference in location (e.g., horizontal coordinates) of corresponding pixels in a pair of images. In other words, disparity indicates the displacement of a pixel (or pixel block) in the second image with respect to its location in the first image. For example, the disparity map122may indicate a horizontal offset between pixels of the first image102and pixels of the second image104. The disparity map122may be stored in known graphics formats, such as GIF or TIFF format.

As traditional computer vision algorithms for estimation of a disparity map based on two input images captured by a rectified stereo image pair (e.g., a vertically aligned stereo image pair), semi-global matching (SGM) and its improvements are described in more detail in Hirschmuller, Heiko. “Accurate and efficient stereo processing by semi-global matching and mutual information.” 2005IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR'05). Vol. 2. IEEE, 2005, in Hirschmuller, Heiko. “Stereo processing by semiglobal matching and mutual information.”IEEE Transactions on pattern analysis and machine intelligence30.2 (2007): 328-341, and in Hirschmüller, Heiko. “Semi-global matching-motivation, developments and applications.”Photogrammetric Week11 (2011): 173-184.

In standard semi-global matching (SGM), a disparity map D122is computed by minimizing an objective function (or energy function) composed of a pixel matching cost term Cp(dp) and a smoothness regularization term R(p, dp, q, dq), as in the form below:

where Cp(dp) is the pixel matching cost term at pixel p with disparity dpand R(p, dp, q, dq) is the smoothness regularization term that penalizes disparity differences between adjacent pixels in a neighborhood. As will be explained below, standard SGM is not well adapted for stereo matching that involves non-fronto-parallel scenes due to the implementation of this smoothness regularization term.

In principle the pixel matching cost term Cp(dp) can use any pixel-wise dissimilarity measure, e.g., the absolute or squared difference between intensity values and/or color values, Birchfield-Tomasi dissimilarity, Hamming distance of the census transform, or the like. Given a rectified stereo image pair, for a pixel p with coordinates (x, y) in a first image, the set of pixel candidates q in a second image is usually selected within a neighborhood (a subset) in the second image. For example, the set of pixel candidates q with coordinates (x′, y) may be selected as {(x′, y)|x′≥x, x′≤x+d}, where d refers to the maximum allowed disparity (or displacement) level that defines the neighborhood. The pixel matching cost term Cp(dp) for d disparity levels at a given pixel p, in the first image may then be determined by computing such a pixel-wise dissimilarity measure with (x to x+d) pixel coordinates in the second image.

The smoothness regularization term which has the form as below

measures the regularization cost between pixels p and q with disparities dpand dqrespectively, for all pairs of neighboring pixels N, where c1and c2are two constant parameters, with c1<c2. The three-way comparison allows to assign a smaller penalty for unitary changes in disparity, thus allowing smooth transitions corresponding, e.g., to slanted surfaces, and penalizing larger jumps while preserving discontinuities due to the constant penalty term.

The implicit assumption made by the standard SGM or other area-based matching algorithms is that objects have fronto-parallel surfaces (i.e. depth is constant within the neighborhood). This assumption is violated by sloping surfaces and/or creased, dented, or otherwise buckled surfaces. Because the currently used smoothness regularization term does not account well for non-fronto-parallel scenes, these algorithms often produce less satisfactory results for scenes involving non-fronto-parallel surfaces. Non-fronto-parallel surfaces may result in “ghost match,” i.e. a local match that is not consistent with a global solution of the stereo correspondence problem.

The polarized semi-global matching (SGM) engine120thus uses slope (or gradient) information captured from polarization to modify the energy function in standard SGM by adding another smoothness regularization term Rslopeto generate an augmented energy function that has the form below:

This smoothness regularization term Rslopeis dependent on the polarization cues and, in particular, the slopes of the surfaces in the scene, which can be computed from the polarization cues. The slopes of the surfaces can be computed with reference to surface normals (or a gradient map, because surface normals are the derivatives of 3-D shape), e.g., as the orientation of the surface normals defined in the surface normal map112. Hence the smoothness regularization term Rslopemay also be referred to as the polarized smoothness regularization term. For example, the slopes can be computed by obtaining a surface normal. As another example, the slopes can be computed by differentiating the 3-D shape.

Unlike the smoothness regularization term used in standard SGM (which may be referred to as the unpolarized smoothness regularization term), which always penalizes jumps in disparity between adjacent pixels in a neighborhood, the smoothness regularization term Rslopeaccommodates the jumps in disparity between adjacent pixels by taking into consideration the slope in the neighborhood, for example by adaptively computing a reduced regularization cost to account for the fact that a particular pixel is further away from the polarized stereo camera system than its neighboring pixels, e.g., when a real-world point corresponding to the particular pixel is located in a dent in the surface on which the other points corresponding to the neighboring pixels reside.

For example, the smoothness regularization term Rslopecan be evaluated as a least squares distance between dpand dpor a binary indicator function of dpand dp.

As another example, when using a single polarization camera, the smoothness regularization term Rslopecan be evaluated as:

where w1and w2are weighting factors, d is the stereo disparity, and α is surface azimuth angle.

As yet another example, when using a multiple polarization camera, the smoothness regularization term Rslopecan be evaluated by adding the following equation to the equation mentioned above with reference to the single polarization camera example:

where α1and α2are the surface azimuth angles (with respect to the same reference frame) given by the polarization camera images from2different points of view.

In some implementations, the polarized SGM engine120can compute the slopes to feed in to the smoothness regularization term Rslopeby computing a differentiation of the 3D shapes captured in the first image102and second image104. In some implementations, the polarized SGM engine120can use shape from polarization (SfP) theory to compute the slopes to feed in to the smoothness regularization term Rslope. Suitable techniques for computing slopes by applying SfP to the AOLP map106and DOLP map108are described in U.S. Pat. No. 10,260,866 B2, the entire contents of which are hereby incorporated by reference herein in their entirety. In some implementations, the polarized SGM engine120can use machine learning techniques to compute the slopes to feed in to the smoothness regularization term Rslope. For example, the polarized SGM engine120can use a trained neural network to compute the slopes from the first image102and second image104. Suitable techniques for implementing and training such a neural network are described in U.S. patent application Ser. No. 17/359,326, the entire contents of which are hereby incorporated by reference herein in their entirety.

To generate the disparity map D122, the polarized SGM engine120can use any suitable optimization algorithms including, e.g., dynamic programming (DP), tree based DP, graph cuts, belief propagation, and so on, to determine, e.g. in an iterative manner, the value of the disparity dpfor each pixel p in the first image (or disparity dqfor each pixel q in the second image) that minimizes the energy function E(D). Collectively, the disparities dpor dqhaving the final determined values define the disparity map D122.

The computer vision system100can also include one or more components that make use of the disparity map122for a variety of purposes or applications. One example of such component is a 3-D reconstruction engine130, which uses information about the correspondences between stereo pairs of first and second images102and104defined (e.g., in terms of horizontal offset) by the disparity map122to generate a reconstructed 3-D scene132or some other digital representation of the environment. The 3-D reconstruction engine130can implement program logic that returns an array of 3-D world point coordinates that reconstruct the scene of the environment from the disparity map122. For example, the 3-D reconstruction engine130can reconstruct the 3-D world coordinates of points corresponding to each pixel from the disparity map122, where the 3-D world coordinates are defined relative to the optical center of a particular camera module in the polarized stereo camera system.

Another example of such component is a depth map estimation engine, which uses the disparity map122to generate, e.g., through triangulation, a depth map that includes a depth (or distance) measurement indicative of a distance between the polarized stereo camera system and the surface depicted by the pixels in an image captured by the polarized stereo camera system. Additional examples of such component include components that implement program logic to generate outputs that identify objects to pick up by one or more robots, identify pick positions for objects by the robots, and/or define collision-free trajectories for the robots from processing the disparity map122, the reconstructed 3-D scene132, or both. The system can for example pass an instruction or other control signal to a control system for the one or more robots to cause the robots to perform one or more actions in accordance with the outputs.

FIG.2is a flow diagram of an example process200for detect corresponding pixels between the first image and the second image. For convenience, the process200will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing neural network system, e.g., the computer vision system100ofFIG.1, appropriately programmed, can perform the process200.

The system obtains a first polarized stereo image generated at a first position from a first viewpoint and a second polarized stereo image generated at a second position from a second viewpoint (step210). The first and second images may be captured by using a polarized stereo camera system. The first and second images may be viewed as a stereo pair.

The system obtains an angle of linear polarization (AOLP) map and a degree of linear polarization (DOLP) map each having a field of view overlapping the first image and the second image (step220). While the remainder of the explanation of process200assumes that the AOLP map and the DOLP map are computed from the first viewpoint of the first polarized stereo image, in some cases the system can additionally obtain a second AOLP map and a second DOLP map that are computed from the second viewpoint of the second polarized stereo image. In those cases, the system can additionally perform processes300or400or both described below with reference toFIG.3to enforce a left/right or symmetric consistency between the pair of images.

The system computes a surface normal map based on the AOLP map and the DOLP map and, optionally, additional feature information extracted from the first and second polarized stereo images (step230). As mentioned above, the system can do so by (i) applying machine learning techniques on the first and second polarized images; (b) applying physics-based Fresnel equations based on the AOLP and DOLP maps, (c) applying machine learning techniques on the AOLP and DOLP maps, or some combination of these.

The system detects corresponding pixels between the first image and the second image by computing a disparity map that optimizes, i.e., minimizes, an energy function (step240). The disparity map is a digital representation that indicates a disparity between the first polarized stereo image and the second polarized stereo image. Disparity refers to the difference in location (e.g., horizontal coordinates) of corresponding pixels in a pair of images.

In standard semi-global matching (SGM), the energy function contains mainly two terms. One is a pixel matching cost term which measures the pixel similarity and the other is an unpolarized smoothness regularization term that penalizes disparity variations in pixels in a local neighborhood. One difference with standard SGM is that the system additionally uses a polarized smoothness regularization term in an augmented energy function and then computes the disparity map that minimizes the augmented energy function. While the unpolarized smoothness regularization term may be computed based on just the pixels in a local neighborhood, the polarized smoothness regularization term, which is dependent on the slopes of the surfaces in the scene captured by the first and second images, will be computed by the system based on (i) pixels in the local neighborhood and (ii) the surface normal map, as described above.

When included in the augmented energy function, the polarized smoothness regularization term and the unpolarized smoothness regularization term can be differently weighted, e.g., based on whether a surface in the scene is fronto-parallel or not. In some implementations, the polarized smoothness regularization term and the unpolarized smoothness regularization term can be complementarily weighted based on the DOLP map. In other words, the weighting can be determined based on how strong the polarization cue is, so that the regularization term will be a dependent on the DOLP map. This weighting could be performed because the accuracy of the slope estimate depends on obliqueness of angle of surface to optical axis of the camera, and DOLP is a good proxy for the angle (obliqueness).

In some other implementations, the polarized smoothness regularization term and the unpolarized smoothness regularization term are complementarily weighted based on computing a difference in slope between the slope map and a fronto-parallel plane with respect to the first viewpoint. For example, the weighting term can be based on how “strong” the polarization cue is. Thus, the polarized smoothness the regularization term becomes dependent on DOLP. As another example, the weighting term can be defined by a hyperparameter the value of which can be predetermined based on the slope difference.

As a particular example of this, the augmented energy function can have the form below:

where λ is the weighting term.

FIG.3is a flow diagram of an example process300for performing a surface normal consistency check between surface normal maps computed from different viewpoints. For convenience, the process300will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing neural network system, e.g., the computer vision system100ofFIG.1, appropriately programmed, can perform the process300.

The system obtains a second angle of linear polarization (AOLP) map and a second degree of linear polarization (DOLP) map (step310). The second ALOP map and the second DOLP map are computed from the second viewpoint of the second polarized stereo image in the stereo pair of images obtained at step210of process200.

Like how the surface normal map is computed at step230of process200, the system computes a second surface normal map based on the second AOLP map and the second DOLP map (step320).

The system transforms the second surface normal map to the first viewpoint in accordance with (i) the second AOLP map and (ii) the disparity map obtained from process200to compute a transformed second surface normal map (step330). In some implementations, this can involve calculating a transformation matrix and using (an inverse transpose of) the matrix to transform the surface normals in the second surface normal map from a coordinate space of the second surface normal map to a different coordinate space of the first surface normal map. In accordance with geometric principles of camera projection, the transformation matrix can be a camera projection matrix of extrinsic parameters that define the rotation and translation between the first and second cameras.

The system performs a surface normals consistency check between the surface normal map and the second surface normal map by computing a difference between the surface normal map and the transformed second surface normal map (step340). In general, the surface normals consistency check can be performed to generate a refined surface normal map which when used to compute the polarized smoothness regularization term in the augmented energy function can result in a more accurate disparity map for pixels in the first image and the second image. In some implementations, this can involve computing:

where f represents Fresnel equations that are viewpoint dependent, nland nrrepresent the surface normals (n) computed from the first (left) and second (right) viewpoints, respectively, d is a disparity estimate, and θcalis an abstraction of physical parameters that map polarization to a normal. Such an abstraction includes, but is not limited to, refractive index n and ambiguity maps.

FIG.4is a flow diagram of an example process400for performing a polarization consistency check between images captured from different viewpoints. For convenience, the process400will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing neural network system, e.g., the computer vision system100ofFIG.1, appropriately programmed, can perform the process400.

The system obtains a second angle of linear polarization (AOLP) map and a second degree of linear polarization (DOLP) map (step410). The second AOLP map and the second DOLP map are computed from the second viewpoint of the second polarized stereo image in the stereo pair of images obtained at step210of process200.

The system transforms the second image to the first viewpoint in accordance with (i) the second AOLP map and (ii) the disparity map obtained from process200to compute a transformed second image (step420). Like how the second surface normal map can be transformed, in some implementations, this can involve calculating a transformation matrix and using the matrix to transform the pixels in the second image from a coordinate space of the second image to a different coordinate space of the first image.

The system performs a polarization consistency check between the first image and the second image by computing a difference between the first image and the transformed second image (step430). The system can then remove any outliers (i.e., any pixel having a disparity value greater than a disparity threshold or any other suitable metric) from the second AOLP map in accordance with the polarization consistency check. The second AOLP map could then be used in left-right consistency check, as an example. In some implementations, this can involve back-calculation of the second (or first) image based on knowledge of the disparities and polarization angles.

Embodiment 1 is a method comprising:obtaining a first image from a first viewpoint of a stereo pair and a second image from a second viewpoint of the stereo pair;obtaining an angle of linear polarization (AOLP) map and a degree of linear polarization (DOLP) map having a field of view overlapping the first image and the second image;computing a surface normal map based on the AOLP map and the DOLP map; anddetecting corresponding pixels between the first image and the second image by computing a disparity map that minimizes an energy function comprising a pixel matching cost term and a polarized smoothness regularization term computed based on pixels in a local neighborhood and based on the surface normal map.

Embodiment 2 is the method of embodiment 1, wherein the energy function further comprises an unpolarized smoothness regularization term computed based on the pixels in the local neighborhood.

Embodiment 3 is the method of any one of embodiments 1-2, wherein the polarized smoothness regularization term and the unpolarized smoothness regularization term are complementarily weighted based on the DOLP map.

Embodiment 4 is the method of any one of embodiments 1-2, wherein the polarized smoothness regularization term and the unpolarized smoothness regularization term are complementarily weighted based on computing difference in slope between the slope map and a fronto-parallel plane with respect to the first viewpoint

Embodiment 5 is the method of any one of embodiments 1-4, wherein the AOLP map and the DOLP map are computed from the first viewpoint, and wherein the method further comprises:obtaining a second AOLP map and a second DOLP map computed from the second viewpoint;computing a second surface normal map based on the second AOLP map and the second DOLP map;transforming the second surface normal map to the first viewpoint in accordance with the second AOLP map and the disparity map to compute a transformed second surface normal map; andperforming a consistency check between the surface normal map and the second surface normal map by computing a difference between the surface normal map and the transformed second surface normal map.

Embodiment 6 is the method of any one of embodiments 1-4, wherein the AOLP map and the DOLP map are computed from the first viewpoint, and wherein the method further comprises:obtaining a second AOLP map and a second DOLP map captured from the second viewpoint;transforming the second image to the first viewpoint in accordance with the second AOLP map and the disparity map to compute a transformed second image; andperforming a polarization consistency check between the first image and the second image by computing a difference between the first image and the transformed second image.

Embodiment 7 is the method of embodiment 6, further comprising removing outliers from the second AOLP map in accordance with the polarization consistency check.

Embodiment 8 is a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 7.

Embodiment 9 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 7.