An information processing apparatus includes a projection unit configured to project a projection pattern onto an object, an imaging unit configured to capture an image of the object on which the projection pattern is projected, and a derivation unit configured to derive a three-dimensional shape of the object based on the image captured by the imaging unit. The projection pattern projected on the object by the projection unit includes a first pattern including a continuous luminance variation repetitively arranged at certain distances in a predetermined direction, and a second pattern having information for identifying the position of the measurement pattern in the captured image in an area between peaks in the measurement pattern.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for measuring a three-dimensional shape of the surface of an object by projecting a projection pattern onto an object and capturing an image of the projected projection pattern.

2. Description of the Related Art

The conventional active three-dimensional shape measurement projects a fringe pattern onto an object, captures an image of the projected fringe pattern, and calculates distortions of the fringe pattern based on the captured image, thus measuring a three-dimensional shape and surface distortion conditions of the object. In particular, the phase shift method has been widely applied. This method projects a fringe pattern having a sinusoidal-wave-shaped luminance variation onto an object, and captures a plurality of images while shifting the phase of the fringe pattern, thus measuring a three-dimensional shape and surface distortion conditions of the object with high density and high accuracy.

However, the phase shift method premises that a plurality of images is captured in a state where the shape of the object remains unchanged, and phase calculation is performed. Therefore, the phase shift method has a problem that measurement cannot be accurately performed if the object moves or transforms during image capturing.

To solve this problem, there has been employed a technique for performing phase calculation by capturing only one image, instead of a plurality of images. A method discussed in Ryusuke Sagawa, Hiroshi Kawasaki, Ryo Furukawa, Shota Kiyota, “Dense One-shot 3D Reconstruction by Detecting Continuous Regions with Parallel Line Projection”, Collected Papers of Meeting on Image Recognition and Understanding (MIRU2011), pp. 416-423 (2011) projects one piece of vertical and horizontal line patterns uniformly colored in green and blue in a specific order onto an object, and captures only one image of the projected line patterns. More specifically, the method corresponds the order of the projected line patterns with the order of the captured line patterns, applies a Gabor filter to the captured line patterns as a fringe pattern having a luminance variation, and performs phase calculation based on the one captured image, thus measuring a three-dimensional shape with high density.

However, in the method discussed in the above-described nonpatent document, since two different colors are used for the line patterns in the projection pattern, the waveforms of the line patterns of respective colors are largely affected by the surface color of the object, resulting in failed correspondence between the order of the projected line patterns and the order of the captured line patterns. Further, the measurement accuracy is remarkably degraded because phase calculation cannot be accurately performed.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an information processing apparatus includes a projection unit configured to project a projection pattern onto an object, an imaging unit configured to capture an image of the object on which the projection pattern is projected, and a derivation unit configured to derive a three-dimensional shape of the object based on the image captured by the imaging unit. The projection pattern projected on the object by the projection unit includes a first pattern including a continuous luminance variation repetitively arranged at certain distances in a predetermined direction, and a second pattern having information for identifying the position of the measurement pattern in the captured image in an area between peaks in the measurement pattern.

DESCRIPTION OF THE EMBODIMENTS

A three-dimensional shape measuring apparatus according to a first exemplary embodiment is directed to measuring the surface shape of an object, and has a configuration illustrated inFIG. 1. However, the configuration illustrated inFIG. 1is to be considered as an example, and various modifications are possible. For example, some constituent elements may be integrated into one constituent element, or one constituent element may be divided into a plurality of constituent elements.

A projector102functions as a projection unit for projecting a projection pattern on an object101to be measured. This projection pattern may be prestored in a memory in the projector102, or generated and supplied by a projection pattern generation unit (described below), or supplied from an external device (not illustrated) to the projector102.

FIG. 2Aillustrates a configuration of a projection pattern201to be used for the present exemplary embodiment. The projection pattern201to be used for the present exemplary embodiment is a monochromatic pattern including a measurement-waveform (first pattern)202having a continuous sinusoidal-wave-shaped luminance variation repetitively arranged in the vertical direction (first direction), and a plurality of code symbols203(second pattern) superimposed between the waves of the measurement-waveform202. This means that the measurement-waveform202and the code symbols203have the same color. Each code symbol203is a waveform in which information used for measurement-waveform correspondence (described below) is coded. In the present exemplary embodiment and subsequent exemplary embodiments, the word “monochrome” includes monochromatic gradation.

The three-dimensional shape measuring apparatus includes a projection pattern generation units103, a projection pattern control unit109(not illustrated), a projection pattern extraction unit105, a measurement-waveform corresponding unit106, a phase calculation unit107, and a three-dimensional shape calculation unit108. These units are constituted by a general-purpose computer (hardware) including a central processing unit (CPU), a memory, a storage device, such as a hard disk, and various input/output interfaces. Each of the projection pattern generation unit103, the projection pattern control unit109, the projection pattern extraction unit105, the measurement-waveform corresponding unit106, the phase calculation unit107, and the three-dimensional shape calculation unit108is realized when the CPU executes a relevant program.

The projection pattern generation unit103functions to generate data of a projection pattern to be projected by the projector102according to certain rules (described below).

The projection pattern control unit109(not illustrated) transmits the projection pattern generated by the projection pattern generation unit103to the projector102. The data is prestored in the storage device.

The camera104functions as an imaging unit for capturing an image of the object101, on which the projection pattern is projected, to generate a captured image301, and transmitting the captured image301to the subsequent projection pattern extraction unit105. The optical axis of the camera104and the optical axis of the projector102are arranged in parallel, and almost perpendicularly to the object101.

Upon acquisition of the captured image301transmitted from the camera104, the projection pattern extraction unit105selectively extracts a measurement-waveform peak curve on the captured image301equivalent to a mountain-shaped portion of the measurement-waveform202in the projection pattern201exiting in the acquired captured image301.

The measurement-waveform corresponding unit106decodes the code symbols203from the captured image301to acquire code information to be used for measurement-waveform correspondence (described below). Based on this code information, the measurement-waveform corresponding unit106correlates the number of waves between a captured image waveform peak curve obtained by the projection pattern extraction unit105and each measurement-waveform peak curve in the captured image301. The measurement-waveform corresponding unit106identifies the position of the measurement-waveform202on the captured image301.

The phase calculation unit107calculates the phase of a sine wave for areas between measurement-waveform peak positions within the captured image301. In an area where the code symbol203exists, a waveform different from the sine wave is produced and therefore the phase calculation unit107does not perform phase calculation.

The three-dimensional shape calculation unit108calculates the depth from the camera104to the object101within the captured image301, i.e., a three-dimensional shape of the object101. Specifically, the three-dimensional shape calculation unit108calculates a three-dimensional shape based on the result of measurement-waveform correlation obtained by the measurement-waveform corresponding unit106, the phase calculated by the phase calculation unit107, and a positional relation obtained in advance between the projector102and the camera104.

FIG. 4is a flowchart illustrating processing according to the present exemplary embodiment. Processing in each step of the flowchart will be described below.

In step S401, the projection pattern generation unit103(seeFIG. 1) generates data of the projection pattern201(seeFIG. 2A) according to the following rules.

The projection pattern201is a monochromatic pattern including the measurement-waveform202having a continuous sinusoidal-wave-shaped luminance variation repetitively arranged in the vertical direction, and the plurality of code symbols203superimposed between the waves of the measurement-waveform202. Information (described below) to be used for corresponding the number of waves in the measurement-waveform202is coded in the code symbols203.FIG. 2Billustrates a variation in the luminance B of an area where the code symbol203exists, assuming that the peak position of the measurement-waveform202is a phase 0 on the horizontal axis, and the adjacent lower peak position of the measurement-waveform202is a phase 2π thereon.

As illustrated inFIG. 2C, the projection pattern201includes areas Aa, Ab, Ba, Bb, Ca, Cb, Da, and Db repetitively arranged in this order in the horizontal direction, and a gap area E inserted between areas A, B, C, and D. Each of the areas A to D is divided in the vertical direction by the code symbols203which intermittently appear. Each code symbol203is arranged in the middle between measurement-waveform peak positions, in units of the areas Aa to Db, and includes as code information the number of waves W (0 to 15) of the measurement-waveform peak right above the code symbol203on the measurement-waveform202.

Specifically, each of areas Ax, Bx, Cx, and Dx (x=a, b) represents a binary digit (0 or 1), and the areas Ax, Bx, C, and Dx collectively represent a 4-digit binary number. When the code symbol203exists in each of areas Xa (X=A, B, C, D), the relevant binary digit represents 0. When the code symbol203exists in each of areas Xb (X=A, B, C, D), the relevant binary digit represents 1. For example, referring to an area204illustrated inFIG. 2C, since the code symbol203exists in the areas Aa, Bb, Ca, and Db, the 4-digit binary number is 0101. The binary number 0101 is converted into a decimal number W of 5.

A gap area E is arranged between the areas Ax, Bx, Cx, and Dx (x=a, b) to prevent mis-detection due to noise. The code symbol203does not exist in the gap area E.

In step S402, the projector102projects onto the object101the projection pattern201(seeFIG. 2A) generated in step S401.

In step S403, the camera104captures an image of the object101, on which the projection pattern201is projected by the projector102, to generate a captured image301, and transmits the generated captured image301to the subsequent projection pattern extraction unit105.

In step S404, the projection pattern extraction unit105selectively extracts the measurement-waveform202in the projection pattern201existing in the captured image301transmitted from the camera104. Processing in step S304will be described in detail below with reference to the flowchart illustrated inFIG. 5.

In step S501, the projection pattern extraction unit105applies Sobel filtering performing a differential action in the vertical direction on the captured image301(acquired in step S403) to generate a vertical Sobel filter image. The Sobel filter is a kind of convolution filter. The present exemplary embodiment employs a Sobel filter having a 3×3 matrix size, as illustrated inFIG. 6.

In step S502, the projection pattern extraction unit105scans the vertical Sobel filter image (acquired in step S501) to detect positions where the sign of the value is inverted, as measurement-waveform peak positions. The following describes detailed procedures for obtaining a measurement-waveform peak position by using a vertical Sobel filter.

First, assume that a vertical Sobel filter image Ivgives a value Iv(x) at a scanning position x. The scanning direction is set as the direction approximately perpendicular to the measurement-waveform202, as illustrated inFIG. 7. The projection pattern extraction unit105scans the vertical Sobel filter image Ivin this scanning direction to detect a position xtat which the value Iv(xt) exceeds a threshold value T, as represented by formula (1).
Iv(xt)>Tformula (1)

FIG. 8illustrates an exemplary variation in value of the vertical Sobel filter image Ivin the scanning direction. A portion indicated by a section801is equivalent to a position where a mountain-shaped portion of the measurement-waveform202exists. This threshold value processing eliminates noise and an effect of a minute value in a section802due to horizontal line pattern distortion by the object101, enabling accurately detecting a rising portion803produced by the measurement-waveform202in the captured image301.

Then, starting from the position xt, the projection pattern extraction unit105scans a position x0where the following formula (2) is satisfied.
Iv(x0)=0  formula (2)

The position x0satisfying the formula (2) indicates the position of the center of gravity where the luminance of the measurement-waveform202provides a local peak, as illustrated inFIG. 8. The projection pattern extraction unit105acquires the position x0as a measurement-waveform peak point cm(m=1, 2, 3, . . . , mmax). The projection pattern extraction unit105repeats the above-described scanning for all of the positions x in the vertical Sobel filter image Ivto acquire all of the measurement-waveform peak points cmin the vertical Sobel filter image Ivas a measurement-waveform peak point group C.FIG. 9Aillustrates a measurement-waveform point group C901at which the relevant positions in the captured image103are plotted.

In step S503, the projection pattern extraction unit105groups proximity points of the measurement-waveform peak point group C (acquired in step S403), and labels each group as a single area Ln(n=1, 2, 3, . . . ). If the number of measurement-waveform peak points cmof a group is equal to or less than a threshold value Nc, the projection pattern extraction unit105recognizes the relevant group as noise, and hence does not label the group.

The above-described labeling can satisfactorily eliminate mis-detection due to noise and a luminance variation of the code symbol203, and a measurement-waveform peak curve Lncan be selectively extracted which is a curved area.FIG. 9Billustrates a measurement-waveform peak curve Ln902extracted through labeling.

In step S405, the measurement-waveform corresponding unit106extracts code information from the captured image301(acquired in step S403) and the measurement-waveform202(extracted in step S404). Based on this code information, the measurement-waveform corresponding unit106correlates the number of waves between the measurement-waveform peak curve Ln(extracted in step S404) and each measurement-waveform peak curve in the captured image301.

FIG. 10is a flowchart illustrating processing in step S405for extracting code information, and correlating the number of waves for the measurement-waveform peak curve Ln. Processing in step S405will be described below with reference to the step number of the flowchart inFIG. 10.

In step S1001, the measurement-waveform corresponding unit106divides the captured image301(acquired in step S403) in the vertical and horizontal directions into the following areas.

As first, the measurement-waveform corresponding unit106divides the captured image301in the vertical direction. As described above, the projection pattern201is divided into areas Aa, Ab, Ba, Bb, Ca, Cb, Da, and Db repetitively arranged in this order in the horizontal direction, and a gap area E inserted among areas Aa to Db. In the present exemplary embodiment, the optical axis of the camera104and the optical axis of the projector102are in parallel, and arranged almost perpendicularly to the object101. This arrangement makes the division areas Aa to Db of the projection pattern201constant even in the captured image301, regardless of the shape of the object101. The division areas Aa to Db in the captured image301are prestored before measurement, and the captured image301is divided in the vertical direction.

Secondly, the measurement-waveform corresponding unit106divides the captured image301in the horizontal direction. Specifically, the measurement-waveform corresponding unit106divides an area enclosed by an arbitrary measurement-waveform peak curve Ln(extracted in step S404) and the closest lower measurement-waveform peak curve Ln+1.

The measurement-waveform corresponding unit106acquires as a division area SXxLn(Xx=Aa, Ab, Db, E) the captured image301to which the above-described first and second divisions have been applied. The measurement-waveform corresponding unit106does not divide an area where either the measurement-waveform peak curve Lnor Ln+1is discontinuous in the middle of a division unit in the vertical direction, and excludes such an area from subsequent processing.

In step S1002, the measurement-waveform corresponding unit106scans all of the measurement-waveform peak curves Ln(acquired in step S503) in the captured image301.

In step S1003, the measurement-waveform corresponding unit106determines whether all of the division areas SAaLnto SDbLnexist in an area enclosed by the measurement-waveform peak curve Lnto be scanned and the measurement-waveform peak curve Ln+1. As described above, the areas Aa to Db repetitively appear. Therefore, if these areas appear at least once, all of the division areas SAaLnto SDbLnare determined to exist. When all of the division areas SAaLnto SDbLnexist (YES in step S1003), the processing proceeds to step S1004. Otherwise, when not all of the division areas SAaLnto SDbLnexist (NO in step S1003), the processing returns to step S1002.

In step S1004, the measurement-waveform corresponding unit106initializes to 0 the number of waves WLnon the measurement-waveform202of the measurement-waveform peak curve Lnwithin a projected image301. As described above, the number of waves WLnis equivalent to the code information.

In step S1005, the measurement-waveform corresponding unit106compares the average luminance of the division area SAaLnwith that of the division area SAbLnin the captured image301. The measurement-waveform corresponding unit106determines that the code symbol203exists in an area having a higher average luminance. When the average luminance of the area SAaLnis higher than that of the area SAbLn(YES in step S1005), the processing proceeds to step S1006. Otherwise (NO in step S1005), the processing proceeds to step S1007.

In step S1006, the measurement-waveform corresponding unit106adds 8 (equivalent to a binary number 1000) to the number of waves WLn.

In step S1007, the measurement-waveform corresponding unit106compares the average luminance of the division area SBaLnwith that of the division area SBbLnin the captured image301. The measurement-waveform corresponding unit106determines that the code symbol203exists in an area having a higher average luminance. When the average luminance of the area SBaLnis higher than that of the area SBbLn(YES in step S1007), the processing proceeds to step S1008. Otherwise (NO in step S1007), the processing proceeds to step S1009.

In step S1008, the measurement-waveform corresponding unit106adds 4 (equivalent to a binary number 0100) to the number of waves WLn.

In step S1009, the measurement-waveform corresponding unit106compares the average luminance of the division area SCaLnwith that of the division area SCbLnin the captured image301. The measurement-waveform corresponding unit106determines that the code symbol203exists in an area having a higher average luminance. When the average luminance of the area SCaLnis higher than that of the area SCbLn(YES in step S1009), the processing proceeds to step S1010. Otherwise (NO in step S1009), the processing proceeds to step S1011.

In step S1010, the measurement-waveform corresponding unit106adds 2 (equivalent to a binary number 0010) to the number of waves WLn.

In step S1011, the measurement-waveform corresponding unit106compares the average luminance of the division area SDaLnwith that of the division area SDbLnin the captured image301. The measurement-waveform corresponding unit106determines that the code symbol203exists in an area having a higher average luminance. When the average luminance of the area SDaLnis higher than that of the area SDbLn(YES in step S1011), the processing proceeds to step S1012. Otherwise (NO in step S1011), the processing proceeds to step S1013.

In step S1012, the measurement-waveform corresponding unit106adds 1 (equivalent to a binary number 0001) to the number of waves WLn.

In step S1013, the measurement-waveform corresponding unit106determines whether scanning is completed for all of the measurement-waveform peak curves Lnexisting within the captured image301. When scanning is completed (YES in step S1013), the processing in step S405ends. Otherwise, when scanning is not completed (NO in step S1013), the processing returns to step S1002to repeat scanning.

By performing the above procedures, for each measurement-waveform peak curve Ln, code information can be extracted from the code symbol203and the number of waves WLncorresponding to measurement-waveform peaks in the projection pattern201can be acquired.

In step S406, the phase calculation unit107scans all of the division areas SXxLn(Xx=Aa, Ab, . . . , Db, E) (acquired in step S1001) in the captured image301.

In step S407, the phase calculation unit107determines whether the code symbol203exists in the division area SXxLn(Xx=Aa, Ab, . . . , Db) to be scanned. To determine whether the code symbol203exists, the phase calculation unit107compares the average luminance of the division area SXaLnwith that of the division area SXbLn. When the average luminance of the division area SXaLnis higher than that of the division area SXbLn, the phase calculation unit107determines that the code symbol203exists in the division area SXaLn. Otherwise, when the average luminance of the division area SXbLnis higher than that of the division area SXaLn, the phase calculation unit107determines that the code symbol203exists in the division area SXbLn. As described above, the phase calculation unit107identifies the projecting positions of the code symbols203in units of division areas. When the code symbol203does not exist (NO in step S407), the processing proceeds to step S408. Otherwise, when the code symbol203exists (YES in step S407), the processing proceeds to step S409without performing phase calculation. Specifically, in an area where the code symbol203exists, the phase calculation unit107does not perform phase calculation and therefore the phase value is indefinite. However, by interpolating phase values based on phases obtained in peripheral areas, a phase value can be obtained even for an area where the code symbol203exists.

In step S408, the phase calculation unit107performs phase calculation in the division area SXaLnto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern201.

FIG. 11is a flowchart illustrating procedures for performing phase calculation in the division area SXaLnin step S408. Processing in step S408will be described below with reference to the step number of the flowchart.

In step S1101, the phase calculation unit107divides the division area SXaLninto pixel sequences in the vertical direction, and scans each pixel sequence as a scanning line Vp(p=1, 2, 3, . . . , pmax), where pmaxindicates the horizontal pixel width of the division area SXaLn.

In step S1102, the phase calculation unit107sets a luminance Buppof an upper end pixel to Qupp, sets a luminance Bdownpof a lower end pixel to Qdownp, and sets a luminance Bminpof a minimum luminance pixel to Qminpin the scanning line Vpto be scanned. The phase calculation unit107assigns these values to the following formulas (3) and (4) to convert a luminance Bpq(q=1, 2, 3, . . . , qmax) of the pixel Qpqconstituting the scanning line Vpinto a normalized luminance Bnpq, where qmaxindicates the number of pixels of the scanning line Vp.

When Qupp≦Qpq≦Qminp

When Qminp<Qpq≦Qdownp

In step S1103, the phase calculation unit107calculates relative phases of 0 to π [rad] in the scanning line Vpto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern201. On the premise that the surface color of the object is constant within the range of the scanning line Vp, the normalized luminance Bnpqobtained in step S1102directly indicates a relative phase based on the cosine value when the phase of the upper end scanning line Vpis set to 0 within the phase range from 0 to π. Therefore, a relative phase value θQpq[rad] on an arbitrary pixel Qpqin the scanning line Vpis represented by the following formula (5).
θQpq=cos−1QpqFormula (5)

When 0≦θQpq≦π, based on the above-described relative phase value θQpqand the number of waves WLnin the measurement-waveform peak area right above the division area SXaLnincluding the scanning line Vp, the vertical position YQpqof an arbitrary pixel Qpqon the projection pattern201is represented by the following formula (6).

Y0 indicates the vertical position of the measurement-waveform peak arranged at the top of the projection pattern201, Yphaseindicates the unit width between measurement-waveform peaks on the projection pattern201, and π indicates the circular constant. By performing the above-described calculation, the vertical position of an arbitrary pixel Qpqcan be obtained within the relative phase range from 0 to π on the scanning line Vp.

In step S1104, the phase calculation unit107calculates relative phases of π to 2π [rad] within the scanning line Vpto be scanned, to acquire the vertical position YQpqwhich is equivalent to the absolute phase in the projection pattern201. When the reflection factor on the surface of the object in the projection pattern201is constant within the range of the scanning line Vp, and the phase of the lower end scanning line Vpis 2 π, the relative phase value θQpq[rad] on an arbitrary pixel Qpqin the scanning line Vpis represented by the following formula (7).
θQpq=2π−cos−1QpqFormula (7)

When π<θQpq≦2π, based on the above-described relative phase value θQpqand the number of waves WLnfor the measurement-waveform peak area right above the division area SXaLnincluding the scanning line Vp, the vertical position YQpqof an arbitrary pixel Qpqon the projection pattern201is represented by the formula (6), similar to step S1103. By performing the above-described calculation, the vertical position of an arbitrary pixel Qpqcan be obtained within the relative phase range from π to 2π on the scanning line Vp.

In step S1105, the phase calculation unit107determines whether scanning is completed for all of the scanning lines Vpexisting in the division area SXaLn. When scanning is completed (YES in step S1105), the processing in step S408ends. Otherwise, when scanning is not completed (NO in step S1105), the processing returns to step S1101to repeat scanning.

By performing the above-described procedures, phase calculation for all of the pixels in the division area SXaLnto be scanned can be performed to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern201.

In step S409, the three-dimensional shape calculation unit108calculates a three-dimensional shape of the object101by using the vertical position YQpqof the pixel Qpqexisting in each division area SXaLn.FIG. 12schematically illustrates a case where the position of an arbitrary measuring point1204at the vertical position YQpqon the projection pattern201is measured by using a camera coordinate system1202in which a principal point position1201of the optical system of the camera104is an origin O (0, 0). A vertical position1203of the projection pattern201on the object101forms a line of intersection between a plane formed in three-dimensional space by the vertical position YQpqof the projection pattern201and the object101. An optical cutting plane1205formed by the vertical position YQpqof the projection pattern201is pre-calibrated by using the camera coordinate system1202based on the following formula (8).
αYpqx+βYpqy+γYpqz+εYpq=0  Formula (8)

The three-dimensional position of the measuring point1204existing at the vertical position1203of the projection pattern201on the object101exists on a straight line1208represented by the following formula (9). In this case, the pixel Qpq(Qx, Qy, −f) of a projection point1207on an image1206captured by the camera104illustrated inFIG. 1is used. The captured image1206illustrated inFIG. 12has the same size as a projected image on an image sensor when the camera104illustrated inFIG. 1is assumed to be a pinhole camera. Further, the captured image1206is in parallel with the xy plane of the camera coordinate system1202, and the center of the captured image1206is disposed at a distance of −f (focal distance) from the origin O in the Z-axis direction.

t indicates a parameter of an arbitrary real number. Since an intersection between the optical cutting plane1205represented by the formula (8) and the straight line1208represented by the formula (9) is the measuring point1204, a position D (Dx, DY, Dz) of the measuring point1204is represented by the following formula (10) in the camera coordinate system1202.

In step S410, the three-dimensional shape calculation unit108determines whether scanning is completed for all of the division areas SXaLnexisting in the captured image301. When scanning is completed (YES in step S410), the processing according to the present exemplary embodiment ends. Otherwise, when scanning is not completed (NO in step S410), the processing returns to step S406to repeat scanning.

By performing steps S402to S410in this way to apply the vertical position YQpqon the projection pattern201to the corresponded pixel Qpqwithin the captured image301, a three-dimensional shape of the entire object101can be obtained based on a set of the measuring points1204.

Thus, according to the present exemplary embodiment, it is possible to measure a three-dimensional shape of an object with high density and high accuracy based on a captured image of the object on which a monochromatic pattern is projected.

Although the projection pattern201according to the present exemplary embodiment represents code information as the number of waves by using four code symbols representing 0 and 1, it may be possible to use other methods for robustly representing code information in a minimum area. For example, by representing a pseudo-random numerical sequence with multivalued code symbols, and comparing the numerical sequence with the numerical sequence of the acquired code symbols, code information can be obtained which can be robustly corresponded even if the number of acquired code symbols changes.

Other desirable exemplary embodiments of the present invention will be described below.

The overall configuration of an image information processing apparatus according to a second exemplary embodiment is basically the same as the configuration according to the first exemplary embodiment illustrated inFIG. 1.

However, instead of the projection pattern201, a projection pattern1301illustrated inFIG. 13Ais used as a projection pattern to be projected onto the measurement target object101by the projector102. The camera104functions as an imaging unit for acquiring a captured image1401of a projection pattern illustrated inFIG. 14Bprojected onto the object101illustrated inFIG. 14A.

FIG. 15is a flowchart illustrating processing according to the present exemplary embodiment. Processing in each step of the flowchart will be described below.

In step S1501, the projection pattern generation unit103generates data of the projection pattern1301illustrated inFIG. 13Aaccording to the following rules.

The projection pattern1301is a monochromatic pattern including a measurement-waveform1302having a similar luminance variation to that in the measurement-waveform202, and a plurality of code symbols1303having a luminance variation in the same direction as the measurement-waveform1302, superimposed between the waves of the measurement-waveform1302. Information used for corresponding the number of waves in the projection pattern1301is coded in the code symbols1303.

As illustrated inFIG. 13B, the projection pattern1301includes areas Aa, Ab, Ba, Bb, Ca, Cb, Da, and Db . . . repetitively arranged in this order in the horizontal direction, and a gap area E inserted between areas A, B, C, and D. A code symbol1303is arranged in a similar format to that of the code symbols203of the projection pattern201according to the first exemplary embodiment, and includes as code information the number of waves W (0 to 15) for the measurement-waveform peak right above the code symbol1303on the measurement-waveform1302.

Further, as illustrated inFIG. 13C, the code symbol1303has a Gaussian functional continuous luminance variation which is steeper and thinner than the unit waveform of the measurement-waveform1302. This luminance variation is detected to enable phase calculation (described below).

The projector102projects onto the object101the projection pattern1301(seeFIG. 2A) generated in step S1501.

In step S1502, the camera104captures an image of the object101, on which the projection pattern1301is projected, to acquire the captured image1401illustrated inFIG. 14B.

In step S1503, the projection pattern extraction unit105selectively extracts the measurement-waveform1302from the captured image1401(acquired in step S1503). Processing in step S1504may be performed in a similar way to the processing in step S404according to the first exemplary embodiment.

In step S1504, the projection pattern extraction unit105extracts code information from the captured image1401(acquired in step S1503) and the measurement-waveform1302(extracted in step S1503). Based on this code information, the measurement-waveform corresponding unit106correlates the number of waves between the measurement-waveform peak curve Ln(extracted in step S1503) and each measurement-waveform peak curve in the captured image1401. Processing in step S1505may be performed in a similar way to the processing in step S405according to the first exemplary embodiment. The code symbol1303according to the present exemplary embodiment is equivalent to the code symbol203in step S405.

By performing similar processing to step S405, for each measurement-waveform peak curve Ln, code information can be extracted from the code symbol1303to acquire the number of waves WLncorresponding to measurement-waveform peaks in the projection pattern1301.

In step S1505, the phase calculation unit107scans all of the division areas SXxLn(Xx=Aa, Ab, . . . , Db, E) (acquired in step S1505) in the captured image301.

In step S1506, the phase calculation unit107determines whether the code symbol1303exists in the division area SXxLn(Xx=Aa, Ab, . . . , Db) to be scanned. To determine whether the code symbol1303exists, the phase calculation unit107compares the average luminance of the division area SXaLnwith that of the division area SXaLn. When the average luminance of the division area SXaLnis higher than that of the division area SXbLn, the phase calculation unit107determines that the code symbol1303exists in the division area SXaLn. Otherwise, when the average luminance of the division area SXbLnis higher than that of the division area SXaLn, the phase calculation unit107determines that the code symbol1303exists in the division area SXbLn. As described above, the phase calculation unit107identifies the projecting positions of the code symbols1303in units of division areas. When the code symbol1303does not exist (NO in step S1506), the processing proceeds to step S1507. Otherwise, when the code symbol1303exists (YES in step S1506), the processing proceeds to step S1508.

In step S1507, the phase calculation unit107performs phase calculation in the division area SXaLnwhere the code symbol1303does not exist to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1301. Processing in step S1508may be performed in a similar way to the processing in step S408according to the first exemplary embodiment.

By performing similar processing to step S408, phase calculation can be performed on all of the pixels in the division area SXaLnto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1301.

In step S1508, the phase calculation unit107performs phase calculation in the division area SXaLnwhere the code symbol1303exists to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1301.

Processing for performing phase calculation in the division area SXaLnin step S1509is similar to the processing of the flowchart illustrated inFIG. 11according to the first exemplary embodiment. Processing in step S1509will be described below with reference to the step number of the flowchart.

In step S1101, the phase calculation unit107divides the division area SXaLninto pixel sequences in the vertical direction, and scans each pixel sequence as a scanning line Vp(P=1, 2, 3, . . . , pmax) r where pmaxindicates the horizontal pixel width of the division area SXaLn.

In step S1102, the phase calculation unit107sets a luminance Buppof an upper end pixel Quppa luminance Bdownpof a lower end pixel Qdownp, and a pixel Qcodepfor the luminance peak of the code symbol1303in the scanning line Vpto be scanned. The phase calculation107sets a luminance Bupminpof a minimum luminance pixel Qupminpabove the pixel Qcodep, and a luminance Bdownminpof a minimum luminance pixel Qdownminpbelow the pixel Qcodep. The phase calculation unit107assigns these values to the following formulas (11) and (12) to convert the luminance Bpq(q=1, 2, 3, . . . , qmax) of the pixel Qpqconstituting the scanning line Vpinto a normalized luminance Bnpq, where qmaxindicates the number of pixels of the scanning line Vp.

When Qupp≦Qpq≦Qcodep

When Qcodep<Qpq≦Qdownp

In step S1103, the phase calculation unit107calculates relative phases 0 to π [rad] on the scanning line Vpto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1301. On the premise that the surface color of the object is constant within the range of the scanning line Vp, the normalized luminance Bnpqobtained in step S1102equals the luminance value of a luminance function Bt=F(θ) for the division area SXaLnincluding the code symbol1303of the projection pattern1301as illustrated inFIG. 16. Therefore, within the phase range from 0 to π, the relative phase value θQpq[rad] on the pixel Qpqis represented by the following formula (13).
θQpq=F−1(Bnpq)  Formula 13

When 0≦θQpq<π, by acquiring in advance the luminance function Bt=F(θ) from the projection pattern1301, prestoring the luminance function, and assigning the normalized luminance Bnpqto the formula (13), the relative phase value θQpqcan be obtained. However, as illustrated inFIG. 16A, when an arbitrary luminance Bsampleis assigned Bnpq, the relative phase value cannot be uniquely determined, i.e., there are two different solution candidates θ1and θ2. Therefore, by using the following formula (14), the phase calculation unit107determines the position of the relative phase value θQpqbased on the sign of the differential luminance value ΔBt=F′(θ), as illustrated inFIG. 16B.

θupminindicates a relative phase value which provides the minimum luminance in an area sandwiched between the code symbol1303and the upper measurement-waveform1302existing in the projection pattern1301.

The differential luminance value ΔBthas different signs even with the same luminance value, as with θ1and θ2illustrated inFIG. 16B. Therefore, by making the determination based on the formula (14), the relative phase value θQpqcan be uniquely calculated.

Based on the obtained relative phase value θQpqand the number of waves WLnin the measurement-waveform peak area right above the division area SXaLnincluding the scanning line Vp, the vertical position YQpqof an arbitrary pixel Qpqon the projection pattern1301is similarly represented by the formula (6) according to the first exemplary embodiment.

By performing the calculation based on the formula (6), the vertical position of an arbitrary pixel Qpqcan be obtained within the relative phase range from 0 to π on the scanning line Vp.

In step S1104, the phase calculation unit107calculates relative phases π to 2π [rad] on the scanning line Vpto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern201. Similar to step S1103, the normalized luminance Bnpq(obtained in step S1102) is represented by the formula (15) based on the luminance function Bθ=F(θ) for the division area SXaLn.
θQpq=F−1(Bnpq)  Formula 15

When π≦θQpq<2 π, also in step S1104, the relative phase value cannot be uniquely determined, i.e., there are two different solution candidates. Therefore, by using the following formula (16), the phase calculation unit107determines the position of the relative phase value θQpqbased on the sign of the differential luminance value ΔBt=F′(θ), as illustrated inFIG. 16B.

θdownminindicates a relative phase value which provides the minimum luminance in an area sandwiched between the code symbol1303and the upper measurement-waveform1302existing in the projection pattern1301.

By performing the determination based on the formula (16), the relative phase value θQpqcan be uniquely calculated.

Based on the obtained relative phase value θQpqand the number of waves WLnin the measurement-waveform peak area right above the division area SXaLnincluding the scanning line Vp, the vertical position YQpqof an arbitrary pixel Qpqon the projection pattern1301is similarly represented by the formula (6) according to the first exemplary embodiment. By performing calculation based on the formula (6), the vertical position of an arbitrary pixel Qpqcan be acquired within the relative phase range from π to 2π on the scanning line Vp.

In step S1105, the phase calculation unit107determines whether scanning is completed for all of the scanning lines Vpexisting in the division area SXaLn. When scanning is completed (YES in step S1105), the processing in step S1509ends. Otherwise, when scanning is not completed (NO in step S1105), the processing returns to step S1101to repeat scanning.

In step S1510, the three-dimensional shape calculation unit108calculates a three-dimensional shape of the object101by using the vertical position YQpqof the pixel Qpqexisting in each division area SXaLn. Processing in step S1510may be performed in a similar way to the processing in step S409according to the first exemplary embodiment.

In step S1510, the three-dimensional shape calculation unit108determines whether scanning is completed for all of the division areas SXaLnexisting in the captured image1401. When scanning is completed (YES in step S1510), the processing according to the present exemplary embodiment ends. Otherwise, when scanning is not completed (NO in step S1510), the processing returns to step S1505to repeat scanning.

By performing steps S1502to S1510in this way to apply the vertical position YQpqon the projection pattern1301to the corresponded pixel Qpqin the captured image1401, a three-dimensional shape of the entire object101can be obtained based on a set of the measuring points1204.

Thus, according to the present exemplary embodiment, it is possible to measure a three-dimensional shape of an object, including an area where a code symbol exists, with high density and high accuracy based on a captured image of the object on which a monochromatic pattern is projected.

Although, in the present exemplary embodiment, each code symbol has a Gaussian functional continuous luminance variation which is thinner than a measurement-waveform, the code symbol configuration is not limited thereto. It is also possible to use other waveforms easily recognizable as a code, which enables robust symbol detection not easily affected by the object shape, and high-accuracy phase calculation. For example, a saw-tooth waveform having a linear luminance variation still thinner than that according to the present exemplary embodiment, and having a luminance peak higher than the measurement-waveform may be used to maintain detectable contrast of the code symbols even if a defocused state is produced by change in distance from the object101.

Another desirable exemplary embodiment of the present invention will be described below.

The overall configuration of an image information processing apparatus according to a third exemplary embodiment is basically similar to the configuration according to the first exemplary embodiment illustrated inFIG. 1.

However, instead of the projection pattern201, the projector102projects a projection pattern1701illustrated inFIG. 17Aonto the object101under measurement.

The camera104serves as an imaging unit for acquiring a captured image1801of a projection pattern1801illustrated inFIG. 18Bprojected onto the object101illustrated inFIG. 18A.

FIG. 19is a flowchart illustrating processing according to the present exemplary embodiment. Processing in each step of the flowchart will be described below.

In step S1901, the projection pattern generation unit103generates data of the projection pattern1701illustrated inFIG. 17Aaccording to the following rules.

The projection pattern1701is a monochromatic pattern including a measurement-waveform1702having a similar luminance variation to that in the measurement-waveform202, and a plurality of short horizontal linear code symbols1703superimposed between the waves of the measurement-waveform1702. Information used for corresponding the number of waves in the projection pattern1701is coded in the code symbols1703.

The projection pattern1701includes areas A, B, C, and D . . . repetitively arranged in this order in the horizontal direction, and a gap area E inserted between the areas A, B, C, and D. A short horizontal linear code symbol1703is arranged in between measurement-waveform peak positions, in units of the areas A to D, and includes as code information the number of waves W (0 to 15) for the measurement-waveform peak right above the code symbol1703on the measurement-waveform1702.

Specifically, each of the areas A, B, C, and D represents a binary digit (0 or 1), and the areas A, B, C, and D collectively represent a 4-digit binary number. When one code symbol1703exists in each area X (X=A, B, C, D), the relevant binary digit represents 0. When two code symbols1703exist in each code area X, the relevant binary digit represents 1.FIG. 17Cillustrates a luminance variation in an area where one code symbol1703exists, assuming that the peak position of the measurement-waveform1702corresponds to phase 0 on the horizontal axis, and the adjacent lower peak position of the measurement-waveform1702corresponds to phase 2π thereon.FIG. 17Dillustrates a luminance variation in an area where two code symbols1703exist.

For example, referring to an area1704illustrated inFIG. 17B, since the code symbol1703exists in the areas A, B, C, and D, the 4-digit binary number is 0101. The binary number 0101 is converted into a decimal number W of 5.

A gap area E is arranged between the areas A, B, C, and D to prevent mis-detection due to noise. The code symbol1703does not exist in the gap area E.

In step S1902, the projector102projects the projection pattern1701illustrated inFIG. 17A(generated in step S1901) onto the object101.

In step S1903, the camera104captures an image of the object101, on which the projection pattern1701is projected, to acquire the captured image1801illustrated inFIG. 18.

In step S1904, the projection pattern extraction unit105selectively extracts the measurement-waveform1702from the captured image1801(acquired in step S1903). Processing in step S1904may be performed in a similar way to the processing in step S404according to the first exemplary embodiment.

In step S1905, the measurement-waveform corresponding unit106extracts code information from the captured image1801(acquired in step S1903) and the measurement-waveform peak curve Ln(extracted in step S1904). Based on this code information, the measurement-waveform corresponding unit106corresponds the number of waves between the measurement-waveform peak curve Ln(extracted in step S1904) and each measurement-waveform peak curve in the captured image1801.

FIG. 20is a flowchart illustrating processing for extracting code information and corresponding the number of waves for the measurement-waveform peak curve Lnon the captured image in step S1905. Processing in step S1905will be described below with reference to the step number of the flowchart.

In step S2001, the measurement-waveform corresponding unit106divides the captured image1801(acquired in step S1903) in the vertical and horizontal directions into the following areas.

As first division, the measurement-waveform corresponding unit106divides the captured image1801in the vertical direction. As illustrated inFIG. 17B, the projection pattern1701includes areas A, B, C, and D repetitively arranged in this order in the horizontal direction, and a gap area E inserted between the areas A, B, C, and D . . . . In the present exemplary embodiment, the optical axis of the camera104and the optical axis of the projector102are in parallel, and arranged almost perpendicularly to the object101. This arrangement makes the division areas A to D of the projection pattern1701constant, regardless of the shape of the object101, even in the captured image1801. The division areas A to D in the captured image1801are prestored, and the captured image1801is divided in the vertical direction.

As second division, the measurement-waveform corresponding unit106divides the captured image1801in the horizontal direction. Specifically, the measurement-waveform corresponding unit106divides an area enclosed by an arbitrary measurement-waveform peak curve Ln(extracted in step S1904) and the closest lower measurement-waveform peak curve Ln+1.

The measurement-waveform corresponding unit106acquires as division area SXLn, (X=A, B, D, E) the captured image1801to which the above-described first and second divisions have been applied. The measurement-waveform corresponding unit106does not divide an area where either the measurement-waveform peak curve Lnor Ln+1is discontinuous in the middle of a division unit in the vertical direction, and excludes such an area from subsequent processing.

In step S2002, the measurement-waveform corresponding unit106scans all of the measurement-waveform peak curves Lnin the captured image1801(acquired in step S1904).

In step S2003, the measurement-waveform corresponding unit106determines whether all of the division areas SALnto SDLnexist in an area enclosed by the measurement-waveform peak curve Lnto be scanned and the measurement-waveform peak curve Ln+1. As described above, the areas A to D repetitively appear. Therefore, if these areas appear at least once, all of the division areas SALnto SDLnare determined to exist. When all of the division areas SALnto SDLnexist (YES in step S2003), the processing proceeds to step S2004. Otherwise, when not all of the division areas SALnto SDLnexist (NO in step S2003), the processing returns to step S2002.

In step S2004, the measurement-waveform corresponding unit106initializes to 0 the number of waves WLnon the measurement-waveform in a projected image1901of the measurement-waveform peak curve Ln. As described above, the number of waves WLnis equivalent to the code information.

In step S2005, the measurement-waveform corresponding unit106counts the number of code symbols1703in the division area SALnin the captured image1801. To count the number of code symbols1703, the measurement-waveform corresponding unit106may differentiate the luminance value Bpof each pixel in the division area SALnin the vertical direction p, and count the number of zero points. As indicated by the luminance differential value ΔBpillustrated inFIG. 21A, the number of code symbols1703is 1 when the number of zero points is three (Z1to Z3). As indicated by the luminance differential value ΔBpillustrated inFIG. 21B, the number of code symbols1703is 2 when the number of zero points is five (Z4to Z8). When the number of code symbols1703is determined to be 2 (2 in step S2005), the processing proceeds to step S2006. Otherwise, when the number of code symbols1703is determined to be 1 (1 in step S2005), i.e., in the case of 3 zero points, the processing proceeds to step S2007.

In step S2006, the measurement-waveform corresponding unit106adds 8 (equivalent to a binary number 1000) to the number of waves WLn.

In step S2007, the measurement-waveform corresponding unit106counts the number of code symbols1703in the division area SBLnin the captured image1801. Processing in step S2007may be performed in a similar way to the processing for counting the number of code symbols1703in the division area SALnin step S2005. When the number of code symbols1703is determined to be 2 (2 in step S2007), the processing proceeds to step S2008. Otherwise, when the number of code symbols1703is determined to be 1 (1 in step S2007), i.e., in the case of 3 zero points, the processing proceeds to step S2009.

In step S2008, the measurement-waveform corresponding unit106adds 4 (equivalent to a binary number 0100) to the number of waves WLn.

In step S2009, the measurement-waveform corresponding unit106counts the number of code symbols1703in the division area SCLnin the captured image1801. Processing in step S2009may be performed in a similar way to the processing for counting the number of code symbols1703in the division area SALnin step S2005. When the number of code symbols1703is determined to be 2 (2 in step S2009), the processing proceeds to step S2010. Otherwise, when the number of code symbols1703is determined to be 1 (1 in step S2009), i.e., in the case of 3 zero points, the processing proceeds to step S2011.

In step S2010, the measurement-waveform corresponding unit106adds 2 (equivalent to a binary number 0010) to the number of waves WLn.

In step S2011, the measurement-waveform corresponding unit106counts the number of code symbols1703in the division area SDLnin the captured image1801. Processing in step S2011may be performed in a similar way to the processing for counting the number of code symbols1703in the division area SALnin step S2005. When the number of code symbols1703is determined to be 2 (2 in step S2011), the processing proceeds to step S2012. Otherwise, when the number of code symbols1703is determined to be 1 (1 in step S2011), i.e., in the case of 3 zero points, the processing proceeds to step S2013.

In step S2012, the measurement-waveform corresponding unit106adds 1 (equivalent to a binary number 0001) to the number of waves WLn.

In step S2013, the measurement-waveform corresponding unit106determines whether scanning is completed for all of the measurement-waveform peak curves Lnexisting in the captured image1801. When scanning is completed (YES in step S2013), the processing in step S1905ends. Otherwise, when scanning is not completed (NO in step S2013), the processing returns to step S2002to repeat scanning.

By performing the above procedures, for each measurement-waveform peak curve Ln, code information can be extracted from the code symbol1703to acquire the number of waves WLncorresponding to measurement-waveform peaks in the projection pattern1701.

In step S1906, the phase calculation unit107scans all of the division areas SXLnin the captured image1801(X=A, B, D, E) (acquired in step S1902).

In step S1907, the phase calculation unit107determines whether the division area SXLnto be scanned (X=A, B, D, E) is SELnto identify the projecting position of the code symbol1703in units of division areas. When the division area SXLnto be scanned is SELn(YES in step S1907), the code symbol1703is determined not to be included, and the processing proceeds to step S1908. Otherwise (NO in step S1907), the code symbol1703is determined to be included, and the processing proceeds to step S1910.

In step S1908, the phase calculation unit107performs phase calculation in the division area SELnto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1701. Processing in step S1908may be performed in a similar way to the phase calculation in the division area SXxLnin step S408according to the first exemplary embodiment. By performing similar processing to step S407, a phase calculation can be made for all of the pixels in the division area SELnto be scanned, to acquire the vertical position YQpqequivalent to the absolute phase in the projection pattern1701.

In step S1909, the three-dimensional shape calculation unit108calculates a three-dimensional shape of the object101by using the vertical position YQpqof the pixel Qpqexisting in each division area SELn. Processing in step S1909may be performed in a similar way to the calculation of the three-dimensional shape in the division area SXxLnin step S409according to the first exemplary embodiment.

In step S1910, the three-dimensional shape calculation unit108determines whether scanning is completed for all of the division areas SXLnexisting in the captured image1801. When scanning is completed (YES in step S1910), the processing according to the present exemplary embodiment ends. Otherwise, when scanning is not completed (NO in step S1910), the processing returns to step S1906to repeat scanning.

By performing steps S1902to S2009in this way to apply the vertical position YQpqon the projection pattern1701to the corresponded pixel Qpqin the captured image1801, a three-dimensional shape of the entire object101can be obtained based on a set of the measuring points1204.

Thus, according to the present exemplary embodiment, it is possible to measure a three-dimensional shape of an object with less overlooked measurement areas and with higher density and higher accuracy than in the first exemplary embodiment based on a captured image of the object on which a monochromatic pattern is projected.

In the present exemplary embodiment, code symbols1703are linear symbols, and information coding is performed based on the number of code symbols1703. However, the code symbol configuration is not limited thereto as long as a code sequence used includes code symbols1703each being composed of at least two, of binary or more multi-values. It is also possible to use other information coding methods in which code symbol recognition is easy and coding is hardly affected by the object shape. For example, when information coding is performed based on high and low luminance values, a constant magnitude of code symbols1703can be maintained. Therefore, the above-described code symbol configuration is effective even if the number of code symbols1703cannot be recognized because of a defocused state produced by change in distance from the object.

Still another desirable exemplary embodiment of the present invention will be described below.

A three-dimensional shape measuring apparatus according to a fourth exemplary embodiment is directed to measuring the surface shape of an object, and has a configuration illustrated inFIG. 22. However, the configuration illustrated inFIG. 22is only an example, and various modifications are possible. For example, some constituent elements may be integrated into one constituent element, or one constituent element may be divided into a plurality of constituent elements.

A projector402functions as a projection unit for projecting a projection pattern on an object401to be measured. This projection pattern may be prestored in a memory in the projector402, or generated and supplied by a projection pattern generation unit (described below), or supplied from an external device (not illustrated) to the projector402.

FIG. 23Aillustrates a configuration of a projection pattern2301to be used for the present exemplary embodiment. The projection pattern2301to be used for the present exemplary embodiment has the following configuration. A measurement-waveform2302(first pattern) having a sinusoidal-wave-shaped luminance variation is arranged in the vertical direction (first direction). A symbol2303(second pattern), that is a waveform having information for identifying the position of the measurement-waveform2302on an object (described below), is arranged between the waves of the measurement-waveform2302. The projection pattern2301is a monochromatic pattern. This means that the measurement-waveform2302and the symbol2303have the same color. The symbol2303according to the present exemplary embodiment is a monochromatic symbol having monochromatic gradation.

A projection pattern generation unit403functions as a projection pattern generation apparatus for generating data of the projection pattern2301to be projected on an object by the projector402according to certain rules (described below).FIG. 23Aillustrates the projection pattern2301. The camera404functions as an imaging unit for capturing an image of the object401, on which the projection pattern2301is projected, to generate a captured image, and transmitting the captured image to a subsequent projection pattern extraction unit405. The optical axis of the camera404and the optical axis of the projector402are in parallel, and arranged almost perpendicularly to the object401.

Upon acquisition of the captured image transmitted from the camera404, the projection pattern extraction unit405selectively extracts the measurement-waveform2302and the symbol2303in the projection pattern2301existing in the acquired captured image, and transmits them to a symbol detection unit406at a later stage.

Upon acquisition of the symbol2303transmitted from the projection pattern extraction unit405, the symbol detection unit406decodes the acquired symbol2303. Then, the symbol detection unit406detects a peak position of the symbol2303as information for calculating the absolute phase of the measurement-waveform2302, and transmits the peak position to a phase calculation unit407(described below).

Upon acquisition of the peak position of the symbol2303transmitted from the symbol detection unit406, the phase calculation unit407calculates an absolute phase of the measurement-waveform2302in areas between the peak positions of the measurement-waveform2302in the captured image based on the acquired peak position of the symbol2303and the captured image. An area where the symbol403exists exhibits a different luminance variation from a luminance variation in the measurement-waveform402. For such an area, the phase calculation unit407switches phase calculation depending on the presence or absence of the symbol403. Further, the phase calculation unit407acquires the position of the projection pattern2301in the captured image based on the calculated absolute phase, and transmits the position to the three-dimensional shape calculation unit408.

The three-dimensional shape calculation unit408acquires the position of the projection pattern2301in the captured image transmitted from the phase calculation unit407. Then, by using the acquired position of the projection pattern2301and a previously-acquired positional relation between the projector402and the camera404, the three-dimensional shape calculation unit408calculates the depth from the camera404to the object401, i.e., a three-dimensional shape, in the captured image.

Processing performed by the three-dimensional shape measuring apparatus according to the present exemplary embodiment will be described below with reference toFIG. 24.

In step S2401, the projection pattern generation unit403illustrated inFIG. 22generates data of the projection pattern2301illustrated inFIG. 23Aaccording to the following rules.

The projection pattern2301is a monochromatic pattern including a measurement-waveform2302having a continuous sinusoidal-wave-shaped luminance variation repetitively arranged at certain distances in the vertical direction, and a symbol2303(as a reference position) superimposed between the waves of the measurement-waveform2302. Although, in the present exemplary embodiment, a waveform periodically changing in the vertical direction is used as the measurement-waveform2302, for example, the measurement-waveform configuration is not limited thereto. However, the present exemplary embodiment utilizes the principle of triangulation, and therefore cannot be applied to a case where the measurement-waveform2302has a luminance variation in the direction perpendicular to a straight line formed by the principal point of the camera404and the principal point of the projector402.FIG. 23Billustrates a variation in the luminance B when the direction of the luminance variation is set to the horizontal axis, the peak position of the measurement-waveform2302is set to the relative phase −π, and the adjacent lower peak position is set to the relative phase π.FIG. 23Cillustrates a variation in the luminance B in an area where the symbol2303exists between the measurement-waveforms2302.

The symbol2303has a luminance variation different from the unit waveform of the measurement-waveform2302so that the symbol2303can be distinguished from the measurement-waveform2302. In the present exemplary embodiment, the symbol2303has a Gaussian functional waveform centering on the origin O having a steeper luminance variation. A difference in luminance variation caused by the presence or absence of the symbol2303is detected in a step (described below), and used as a reference position for absolute phase calculation. An area where the symbol2303exists also exhibits a continuous luminance variation having a mountain shape. For such an area, the phase calculation unit407switches phase calculation (described below) depending on the presence or absence of the symbol2303.

In step S2402, the projector402projects the projection pattern2301illustrated inFIG. 23A(generated in step S2401) onto the object401.

In step S2403, the camera404captures an image of the object401, on which the projection pattern2301is projected by the projector402, to generate a captured image, and transmits the captured image to the projection pattern extraction unit405at a later stage.

In step S2404, the projection pattern extraction unit405selectively extracts a measurement-waveform peak curve formed of the measurement-waveform2302of the projection pattern2301existing in the captured image transmitted from the camera404. Processing in step S2404will be described in detail below with reference to the flowchart illustrated inFIG. 25.

In step S2501, the projection pattern extraction unit405applies Sobel filtering for measurement-waveform performing a differential action in the vertical direction on the captured image (transmitted from the camera404) to generate a measurement-waveform Sobel filter image. The Sobel filter is a kind of convolution filter. The present exemplary embodiment employs a filter having a 3×5 matrix size, as illustrated inFIG. 26.

The vertical width of the filter is set to obtain a maximum contrast which a gap between the measurement-waveforms2302allows in the captured image, however it may be set to other optimum values depending on the gap between the measurement-waveforms2302. Of course, the processing for generating a differential image of a captured image is not limited thereto. It is also possible to generate a differential image of a captured image with other methods.

In step S2502, the projection pattern extraction unit405scans the measurement-waveform Sobel filter image (acquired in step S2501) to detect positions where the sign of the pixel value is inverted, as measurement-waveform peak positions. The following describes detailed procedures for obtaining measurement-waveform peak positions based on a measurement-waveform Sobel filter image.

In the following descriptions, when a measurement-waveform Sobel filter image is denoted as Ivand the value at a scanning position x is denoted as Iv(x), the scanning direction is set as the direction almost perpendicular to measurement-waveform2302, as illustrated inFIG. 7. When the pixel value of a pixel at the coordinate position currently being referred to on a target line is larger than a threshold value T, the coordinate position is xt. Specifically, the projection pattern extraction unit405scans the measurement-waveform Sobel filter image in the scanning direction to detect a measurement-waveform detection start position xtexceeding a measurement line threshold Tm, as illustrated in the formula (17).
Iv(xt)>TmFormula 17

FIG. 8illustrates a part of pixel groups constituting the target line. Each pixel in a section802has a noise and a pixel value effected by horizontal line pattern distortion caused by the object401. Each pixel in a section801is a pixel equivalent to the position where a mountain-shaped portion of the measurement-waveform2302exists. As described above, Xt indicates a coordinate position which satisfies the above-described formula (17).

This threshold value processing eliminates the noise and the effect of a minute value in the section802due to horizontal line pattern distortion caused by the object401, enabling accurately detecting the rising portion803produced by the measurement-waveform2302from the captured image.

Then, the projection pattern extraction unit405scans the position x0which satisfies the following formula (18), starting with the position xt.
Iv(x0)=0  Formula 18

The position x0satisfying the formula (18) indicates the position of the center of gravity where the luminance of the measurement-waveform402locally reaches a peak, as illustrated inFIG. 8. The projection pattern extraction unit405acquires the coordinates position x0as a measurement-waveform peak point cm(m=1, 2, 3, . . . , mmax).

The projection pattern extraction unit405repeats the above-described scanning processing for all of the positions x in the measurement-waveform Sobel filter image Ivto acquire all of the measurement-waveform peak points cmin the measurement-waveform Sobel filter image Ivas a measurement-waveform peak point group C. Then, the projection pattern extraction unit405transmits the measurement-waveform group C obtained through the above-described processing to the phase calculation unit407at the later stage.

In step S2503, the projection pattern extraction unit405groups proximal points in the measurement-waveform peak point group C (acquired in step S2502), and labels each group as a single area Ln(n=1, 2, 3, . . . ). If the number of measurement-waveform peak points cmof a group is equal to or less than the threshold value Nc, the projection pattern extraction unit405regards the relevant group as a noise, and hence does not label the group.

By the above-described labeling, mis-detection due to the noise and a luminance variation of the symbol2303can be satisfactorily eliminated, and a measurement-waveform peak curve Lnwhich is a curved area can be selectively extracted.

By performing the above-described processing of selectively extracting, in step S2404, measurement-waveform peak curves Lnformed by the measurement-waveform402based on the captured image (acquired in step S2503) can be selectively extracted.

Returning toFIG. 24, in step S2405, the symbol detection unit406detects positions of the symbols2303based on the captured image transmitted from the camera404.

Processing in step S2405will be described in detail below with reference to the flowchart illustrated inFIG. 27.

In step S2701, the symbol detection unit106applies to the captured image (which is transmitted from the camera104) Sobel filtering for the symbol2303performing a differential action in the vertical direction, to generate a symbol Sobel filter image. Step S2701performs similar processing to step S401by using a symbol Sobel filter having a 3×3 matrix size, as illustrated inFIG. 6. The vertical width of the filter is set to obtain a maximum contrast with a luminance variation of the symbol403in the captured image, however, it may be set to other optimum values depending on the shape of the symbol403. Of course, the processing for generating a differential image of a captured image is not limited thereto. It is also possible to generate a differential image of a captured image with other methods.

In step S2702, the symbol detection unit406scans the symbol Sobel filter image (acquired in step S2701) to detect as a symbol detection start position xba point which is maximized in each scanning line and exceeds a symbol threshold value Tb.

In step S2703, the symbol detection unit406scans the symbol Sobel filter image (acquired in step S2701) starting from the symbol detection start position xb (obtained in step S2702), to detect a first position where the sign of the value is inverted, as a symbol peak position.

In this case, the symbol detection unit406performs calculation by using the symbol Sobel filter image instead of the measurement-waveform Sobel filter image Iv, and the symbol detection start position xb instead of the measurement-waveform detection start position xt. As a result of this calculation, the symbol detection unit406acquires as a symbol peak point bn(n=1, 2, 3, . . . , nmax) up to one position x0existing in units of scanning lines. Then, the symbol detection unit406labels as a symbol peak point group B all of the symbol peak points bnin the symbol Sobel filter image.

By performing the above operation, the symbol peak point group B (the position of the symbol2303) can be acquired based on the captured image.

In step S2406, the phase calculation unit407scans the captured image (transmitted from the camera404) with a scanning line Sy(y=1, 2, 3, . . . , ymax). Processing in subsequent steps is performed in units of scanning pixels pxyconstituting each scanning line Syin the captured image.

In step S2407, the phase calculation unit407determines whether the position of the scanning pixel pxyexists in a single phase of the symbol2303illustrated inFIG. 23A. Depending on the result of the determination, processing proceeds to step S2408or S2409.

Specifically, the phase calculation unit407searches for two adjacent pixels on the same scanning line Sy, starting with the scanning pixel pxy, to detect a measurement-waveform peak point cmand a symbol peak bn.

When a measurement-waveform peak point cmis first detected or no point is detected as both of the two adjacent pixels (NO in step S2407), the phase calculation unit407determines that the scanning pixel pxyexists outside a single phase of the symbol2303, and the processing proceeds to step S2408. Otherwise, when a symbol peak point bnis first detected as either one of the two adjacent pixels (YES in step S2407), the phase calculation unit407determines that the scanning pixel pxyexists in a single phase of the symbol2303, and the processing proceeds to step S2409.

In step S2408, the phase calculation unit407performs phase calculation for the scanning pixel pxyexisting outside a single phase of the symbol2303to acquire the vertical positions Yxyequivalent to the absolute phase in the projection pattern2301.

Processing in step S2408will be described in detail below with reference to the flowchart illustrated inFIG. 28.

In step S2801, the phase calculation unit407sets an area enclosed by measurement-waveform peak points cmon the same y-coordinate value as a search area Apxy, starting with the scanning pixel pxy.

In step S2802, the phase calculation unit407searches for a pixel having a smallest luminance value as a minimum luminance pixel pminin the search area Apxy(set in step S2801) to acquire a minimum luminance Bmin.

In step S2803, the phase calculation unit407assigns a luminance Bupof the upper end pixel, a luminance Bdownof a lower end pixel, and the minimum luminance Bminin the search area Apxyto the following formulas (19) and (20) to convert the luminance value Bxyof the scanning pixel pxyinto a normalized luminance Bnxy.

When the y-coordinate value of pxyin the captured image is larger than the y-coordinate value of pmin(ypxy>ypmin)

When the y-coordinate value of pxyin the captured image is equal to or less than the y-coordinate value of pmin(ypxy≦ypmin)

In step S2804, the phase calculation unit407acquires the vertical position Ypxyequivalent to the absolute phase of the scanning pixel pxyin the projection pattern2301.

First, in the normalized luminance Bnxyobtained in step S2803, the surface color of the object is constant within the range of the search area Apxy.

Within the phase range from −π to 0 where ypxy>ypminis satisfied, the normalized luminance Bnxyindicates a relative phase based on the sinusoidal value multiplied by −1 when the phase of the upper end pixel in the search area Apxyis set to −π and the phase of the minimum luminance pixel pmintherein is set to 0. Therefore, the relative phase value θpxy[rad] of the scanning pixel pxyexisting at a position within the phase range from −π to 0 is represented by the following formula (21).

When the y-coordinate value of pxyin the captured image is larger than pmin(ypxy>ypmin, −π≦θpxy≦0)
θpxy=−sin−1BnxyFormula 21

Likewise, within the phase range from 0 to π where ypxy≦ypminis satisfied, the relative phase value θpxyindicates a phase value based on the sinusoidal value when the phase of the lower end pixel in the search area Apxyis set to π and the phase of the minimum luminance pixel pmintherein is set to 0. Therefore, the relative phase value θpxy[rad] of the scanning pixel pxyexisting at a position within the phase range from 0 to π [rad] is represented by the following formula (22).

When the y-coordinate value of pxyin the captured image is equal to or less than pmin(ypxy≦ypmin, 0<θpxy<π)
θpxy=sin−1BnxyFormula 22

Based on the above-described relative phase value θpxy, the vertical position Ypxyof the scanning pixel pxyon the projection pattern2301is represented by the following formula (23).

Y0 indicates the vertical position of the symbol2303on the projection pattern2301. Yphaseindicates the unit width equivalent to a single phase between measurement-waveform peaks on the projection pattern2301. π indicates the circular constant, and W indicates the number of measurement-waveform peak points cmexisting between the symbol peak point bnand the scanning pixel pxyexisting on the scanning line Sy.

By performing the above-described operations, phase calculation for the scanning pixel pxycan be performed to acquire the vertical position Yxy equivalent to the absolute phase in the projection pattern2301in step S2408.

In step S2409, the phase calculation unit407performs phase calculation for the scanning pixel pxyexisting in a single phase of the symbol2303to acquire the vertical positions Yxy equivalent to the absolute phase in the projection pattern2301.

The procedures in step S2409are almost similar to those in step S2408illustrated inFIG. 28. However, since detailed operations are different, the processing in step S2409will be described below with reference to the step number of the flowchart illustrated inFIG. 28.

In step S2801, the phase calculation unit407sets as a search area Apxyan area which is enclosed by the measurement-waveform peak points cmon the same y-coordinate value, and includes the symbol peak point bn, starting with the scanning pixel pxy.

In step S2802, the phase calculation unit407searches for a minimum luminance pixel pmin1at positions above the y-coordinate value (ybn) of the captured image at which the symbol peak point bnexists, in the search area Apxy(set in step S2801). Likewise, the phase calculation unit407searches for a minimum luminance pixel pmin1at positions below the y-coordinate value (ybn) of the captured image at which the symbol peak point bpexists, in the search area Apxy(set in step S2801). Then, the phase calculation unit407acquires respective minimum luminances Bmin1and Bmin2.

In step S2803, the phase calculation unit407convert the luminance value Bxyof the scanning pixel pxyinto a normalized luminance Bnxy, by using the luminance Bupof the upper end pixel, the luminance Bdownof the lower end pixel, a luminance Bbnof the symbol peak point bn, and the minimum luminances Bmin1and Bmin2in the search area Apxy.

When the y-coordinate value of pxyin the captured image is larger than the y-coordinate value of pmin1(ypxy>ypmin1)

When the y-coordinate value of pxyin the captured image is equal to or less than the y-coordinate value pmin, and larger than the y-coordinate value of bn(ybn<ypxy≦ypmin1)

When the y-coordinate value of pxyin the captured image is equal to or less than the y-coordinate value of bn, and larger than the y-coordinate value of pmin2(ymin2<ypxy≦ybn)

When the y-coordinate value of pxyin the captured image is equal to or less than the y-coordinate value of pmin2(ypxy≦ypmin2)

In step S2804, the phase calculation unit407calculates a relative phase of the scanning pixel pxyto acquire the vertical position Ypxyequivalent to the absolute phase in the projection pattern2301, with reference to the position of the symbol peak point bn. On the premise that the surface color of the object is constant within the range of the search area Apxy, the normalized luminance Bnxyobtained in step S2803equals the luminance value of a luminance function Bt=F(θ) in the search area Apxyincluding the symbol2303of the projection pattern2301, as illustrated inFIG. 23C. Therefore, the relative phase value θQpq[rad] is represented by the following formula (28).
θpxy=F−1(Bpxy)  Formula 28

When π≦θpxy<π, by preacquiring the luminance function Bt=F(θ) from the projection pattern2301, prestoring the luminance function, and assigning the normalized luminance Bnxyto the formula (28), the relative phase value θQxycan be obtained. However, as illustrated inFIG. 29A, when an arbitrary luminance Bsampleis assigned Bnxy, the relative phase value cannot be uniquely determined, i.e., there are four different solution candidates θ1, θ2, θ3, and θ4. Therefore, the luminance differential function ΔBt=F′(θ), as illustrated inFIG. 29B, is used to determine the position of θpxy. Specifically, the phase calculation unit107determines the position of θpxybased on the sign of the differential value of the scanning pixel pxy, and a positional relation of the y position ysimbolto the symbol peak bnon the captured image by using the following formula (29).

θmin1indicates a relative phase value which provides the minimum luminance Bmin1in an area sandwiched between the symbol2303and the upper measurement-waveform402on the y coordinates existing in the projection pattern2301. Likewise, θmin2indicates a relative phase value which provides the minimum luminance Bmin2in an area sandwiched between the symbol2303and the lower measurement-waveform2302on the y coordinates existing in the projection pattern2301.

As described above, even pixels having the same luminance value have different signs of the ΔBtand different positional relations with the symbol2303. Therefore, by performing the determination based on the formula (29), the relative phase value θpxycan be uniquely calculated.

The vertical position Ypxyof the scanning pixel Pxy on the projection pattern2301is similarly represented by the formula (7) based on the relative phase value θpxy, and the number W of measurement-waveform peak points cmexisting between the symbol peak point bnon the scanning line Syand the scanning pixel pxy.

By performing calculation based on the formula (7), the vertical position Ypxyof the scanning pixel pxyon the projection pattern2301can be obtained.

By performing the above-described operations, a phase calculation for the scanning pixel pxycan be made to acquire the vertical position Yxy equivalent to the absolute phase in the projection pattern2301in step S2409.

In step S2410, the three-dimensional shape calculation unit408calculates a three-dimensional shape of the object401by using the vertical position Ypxyof the scanning pixel pxyon the projection pattern2301obtained in steps S2408and S2409. Processing in step S2410is similar to the processing in step S409, and redundant descriptions will be omitted.

In step S2411, the three-dimensional shape calculation unit408determines whether scanning is completed for all of the scanning pixels pxyexisting in the captured image401. When scanning is completed (YES in step S2411), the processing according to the present exemplary embodiment ends. Otherwise, when scanning is not completed (NO in step S2411), the processing returns to step S2406to repeat scanning.

By performing steps S2401to S2411in this way to apply the vertical position Ypxyon the projection pattern2301to the corresponded scanning pixel pxyin the captured image401, a three-dimensional shape of the entire object can be obtained.

Thus, according to the present exemplary embodiment, it is possible to measure a three-dimensional shape of an object with high density and high accuracy based on a captured image of the object on which a monochromatic pattern is projected.

Still another desirable exemplary embodiment of the present invention will be described below.

The overall configuration of an image information processing apparatus according to a fifth exemplary embodiment is basically similar to the configuration according to the fourth exemplary embodiment illustrated inFIG. 22.

However, instead of the projection pattern401, the projector402projects a projection pattern3001illustrated inFIG. 30Aonto the object401to be measured.

The camera404functions as an imaging unit for capturing an image of the object401illustrated inFIG. 22, on which the projection pattern3001illustrated inFIG. 30Ais projected, to generate a captured image, and transmitting the captured image to a subsequent projection pattern extraction unit405.

The flowchart illustrating the processing according to the present exemplary embodiment is basically the same as the flowchart according to the third exemplary embodiment illustrated inFIG. 24.

However, the processing according to the present exemplary embodiment differs from the processing according to the third exemplary embodiment only in two different steps: step S2401in which the projection pattern3001is generated, and step S2405in which the symbol3003is detected. Processing according to the present exemplary embodiment will be described below with reference to the step number of each flowchart.

In step S2401, the projection pattern generation unit403generates data of the projection pattern3001illustrated inFIG. 30Aaccording to the following rules.

The projection pattern3001is a monochromatic pattern including a measurement-waveform3002having a continuous and periodical sinusoidal-wave-shaped luminance variation which is repetitively arranged in the vertical direction, and a symbol3003(as a reference position) superimposed between the waves of the measurement-waveform3002.FIG. 30Billustrates a variation in the luminance B when the direction of the luminance variation is set to the horizontal axis, the peak position of the measurement-waveform3002is set to the relative phase −π, and the adjacent lower peak position is set to the relative phase π.FIG. 30Cillustrates a variation in the luminance B in an area where the symbol3003exists between the measurement-waveforms3002.

The symbol3003has a steepest luminance variation in the projection pattern3001. The symbol detection unit406detects the symbol3003in step S2405, and uses the symbol3003as a reference position for calculating the absolute phase of the measurement-waveform3002. An area where the symbol3003exists also exhibits a continuous luminance variation. In such an area, the phase calculation unit407switches phase calculation (described below) depending on the presence or absence of the symbol3003to make a depth calculation.

In step S2405, the symbol detection unit406detects the position of the symbol3003based on the captured image transmitted from the camera404.

Processing in step S2405will be described in detail below with reference to the flowchart illustrated inFIG. 31.

In step S3101, the symbol detection unit406applies to the captured image (acquired in step S2403) Sobel filtering for the symbol3003performing a differential action in the vertical direction, to generate a symbol Sobel filter image.

In step S3102, the symbol detection unit406scans the symbol Sobel filter image (acquired in step S3101) to detect as a symbol peak point bna point which is maximized in each scanning line and exceeds a symbol threshold value Tbp. In the present exemplary embodiment, as illustrated inFIG. 30C, the luminance value exhibits a steep luminance variation at the position of the symbol peak point bn, and the position providing a maximum value of luminance variation serves as the symbol peak point bnas it is. As a result, the symbol detection unit406acquires as a symbol peak point bn(n=1, 2, 3, . . . , nmax) up to one position x0existing in units of scanning lines. Then, the symbol detection unit406labels as a symbol peak point group B all of the symbol peak points bnin the symbol Sobel filter image.

With the above-described procedures, the symbol detection unit406detects the position of the symbol3003based on the captured image.

Thus, according to the present exemplary embodiment, it is possible to measure a three-dimensional shape of an object with high density and high accuracy based on a captured image of the object on which a monochromatic pattern is projected.

Other Embodiments

This application claims priority from Japanese Patent Application No. 2012-195085 filed Sep. 5, 2012, and No. 2012-195086 filed Sep. 5, 2012 which are hereby incorporated by reference herein in their entirety.