Patent ID: 12196542

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. According to the description, claims and the drawings disclosed in the specification, one skilled in the art may easily understand the concepts and features of the present invention. The following embodiments further illustrate various aspects of the present invention, but are not meant to limit the scope of the present invention.

FIG.1is a schematic block diagram of a three-dimensional measurement system of an embodiment of the disclosure. This system is adapted to measure a height of a target object40, wherein the target object40is disposed at a reference surface60. As shown inFIG.1, a three-dimensional (3D) measurement system100based on phase shift includes a projection device10, a camera20and a processor30. The projection device10is configured to project structural light SL to the target object40. The camera20is configured to photograph the target object40to generate a target image. A field of view (FOV) F of the camera20has to include the reference surface60and the target object40, and an angle of photographing of the camera20may be adaptively adjusted according to a location and a size of the target object40. The reference surface60of the embodiment is a flat surface, but in other embodiments, the reference surface may also be a surface or curved surface of non-uniform height, the reference surface is not limited by the surface of the embodiment.

Generally, the principle of 3D measurement based on phase shift is through projecting the structural light SL (for example, texture image, phase fringe or grating image) to the surface of the target object40, collecting fringe data of reflection of the surface of the target object40, then calculating height information of the surface of the target object40with trigonometry. Through a phase-height conversion parameter k, the height information may be calculated with phase information in the target image, as shown by equation (1) below.

h⁡(x,y)=L1+2⁢π⁢dΔ∅⁡(x,y)⁢P≈k·Δ∅⁡(x,y)Equation⁢(1)

Assuming the reference surface60is located on a x-y plane, h(x, y) represents a height at the coordinate (x, y), P represents a periodic length of the structural light SL, d represents a distance between the projection device10and the camera20, L represents a distance between the camera20and the reference surface60, k represents the phase-height conversion parameter, and ΔØ(x, y) represents a phase-shift of the target object40and the reference surface60at the coordinate (x, y). The disclosure provides a method of generating the phase-height conversion parameter k in a faster way, and a method and system of calculating object height using the phase-height conversion parameter k.

As shown inFIG.1, the processor30is electrically connected to the projection device10and the camera20to control the projection device10emitting the structural light SL and control the camera20photographing the target image. The processor30is configured to execute commands, for example, to execute a plurality of first commands according to the target image to calculate the height of the target object40, wherein the corresponding process of these first commands are shown inFIG.2.

FIG.2is a flowchart of a three-dimensional measurement system based on phase shift of an embodiment of the disclosure. Step S1is “obtaining a plurality of pieces of target phase data according to the target image”, step S2is “compensating the pieces of target phase data according to an error phase model”, step S3is “obtaining basic phase data”, step S4is “calculating a phase difference according to the basic phase data and the pieces of target phase data”, and step S5is “calculating the height of the target object according to the phase difference and the phase-height conversion parameter”.

FIG.3Ais an example of the target image I1. In step S1, the processor30may generate a plurality of scan lines, wherein one of the scan lines is the scan line L1shown inFIG.3Apassing through the reference surface60and the target object40. The processor30obtains a plurality of pieces of target phase data described in step S1from the scan line L1. An absolute phase diagram shown inFIG.3Bmay be illustrated according to these pieces of target phase data, wherein the horizontal axis represents the coordinate of X direction, and vertical axis represents the phase with a unit of radian.

After step S1, step S2is performed. That is, step S2is performed after the processor30obtaining the pieces of target phase data according to the target image I1. In step S2, the processor30compensates these pieces of target phase data based on a phase error model. The phase error model reflects an error between measured target phase and ideal phase. This phase error model has to be built before step S2, wherein the method of building the phase error model is described below. In addition, in another embodiment, step S2may also be omitted for step S1to be directly followed by step S3.

After step S2, that is, the processor30performs step S3after compensating the pieces of target phase data according to the phase error model. In an embodiment of step S3, the processor30obtains pre-stored basic phase data. In detail, before the target object40is disposed at the reference surface60, the camera20photographs the reference surface60to generate the reference surface image in advance, and the processor30performs decoding computation according to this reference surface image to obtain a plurality of pieces of phase data of the reference surface image, the processor30then compensating pieces of phase data of the reference surface image to generate the basic phase data with an error calibration model. Therefore, in step S3, the processor30may obtain the basic phase data generated in advance. In another embodiment of step S3, the reference surface60may also be photographed according to the above process to generate the latest basic phase data in real time, the disclosure is not limited thereto.

Step S4is “calculating the phase difference according to the basic phase data and the pieces of target phase data”. In detail, as illustrated as the schematic diagram shown inFIG.3C, a plurality of pieces of phase data in the basic phase data and a plurality of pieces of data in the pieces of target phase data are subtracted to obtain a plurality of phase difference data according to the coordinate of the same pixel. According to a size of the phase difference,FIG.3Cmay be approximately divided into two parts: one being the coordinates 0˜900 and the coordinates 1600˜2500 corresponding to area in the reference image I1not having the target object40, and one being the coordinates 900˜1600 corresponding to area in the reference image I1having the target object40.

In step S5, as shown by equation (1) andFIG.3C, the processor30calculates a product of the phase difference ΔØ multiplied with the phase-height conversion parameter k as the height h of the target object40. According to the process of steps S1˜S5, the processor30may calculate the height of each point in the target image I1, thereby building a point cloud model of the target object40according to the pieces of height information.

Regarding the method of building the phase error model described in step S2, in detail, the processor30is configured to execute commands, such as a plurality of second commands to generate the phase error model, wherein the process that these second commands correspond to are shown inFIG.4.

FIG.4is a flowchart of building a phase error model of an embodiment of the disclosure. Step P1is “projecting the structural light to an error calibration surface”, step P2is “photographing the error calibration surface to generate an error calibration surface image”, step P3is “performing the decoding computation to obtain a plurality of pieces of phase data of the error calibration surface”, step P4is “obtaining a plurality of pieces of modeling phase data among a plurality of pieces of phase data of the error calibration surface image locating at an error calibration straight line”, step P5is “performing linear fitting on the pieces of modeling data to generate a plurality of pieces of ideal phase data”, step P6is “calculating a plurality of pieces of phase error data according to the pieces of modeling data and the pieces of ideal phase data”, and step P7is “building the phase error model according to the pieces of phase error data”.

In step P1and step P2, the processor30executing second commands controls the projection device10and the camera20perform the corresponding operations, respectively. In step P1, the structural light is periodic structural light, and a phase of the structural light increases along an extension direction.

FIG.5Ais an example of the error calibration surface image I2,FIG.5Bis an example of the pieces of modeling data. In an embodiment of step P3, the processor30performs the decoding computation according to the error calibration surface image I2to obtain a plurality of pieces of phase data corresponding to the error calibration surface42. In step P4, the processor30obtains a plurality of pieces of modeling phase data locating at the error calibration straight line from a plurality of pieces of phase data in the error calibration surface image I2according to an error calibration straight line in the error calibration surface image I2that is parallel to the extension direction. The processor30generates a scan line L2(the error calibration straight line) in the error calibration surface image I2shown inFIG.5A, and this scan line L2(the error calibration straight line) has to pass through the error calibration surface42and has a slope characteristic as shown inFIG.5B: on this scan line L2, distribution directions of phases a plurality of pieces of modeling data are increasing (equivalent to being parallel to the extension direction of the phase of the structural light). In other words, when a coordinate of a pixel of a piece of modeling data increases, its phase value also increases. Overall, step P3is: in the error calibration surface image I2, the processor30obtains a plurality of pieces of modeling data according to a straight line (the scan line L2, that is, the error calibration straight line) with increasing in the direction of phase distribution.

Since the error calibration surface42is a flat surface with no changes in height, ideally, the phase data corresponding to this error calibration surface42should show a linear increase. However, the hardware elements in the projection device10or the camera20may lead to error during measurement, causing the phase data corresponding to the error calibration surface42not in a perfect linear increasing. Therefore, in step P5, the processor30performs the linear fitting. In an embodiment, the processor30, for example, uses a least squares method according to the pieces of modeling data to generate the pieces of ideal phase data.FIG.5Cis an example of generating the pieces of ideal phase data using the least squares method, wherein the curve C1with slight oscillation is the pieces of modeling data, straight line C2is the pieces of ideal phase data. It should be noted that, the error calibration surface42of the embodiment is the same surface as the reference surface60in step S1˜S7, but in other embodiments, the error calibration surface42and the reference surface60in step S1˜S7may also be different surfaces.

In an embodiment of step P6, for each pixel coordinate, the processor30calculates a difference value between an ideal phase and an actual phase as a piece of phase error data, whereinFIG.5Dis an example of generating the pieces of phase error data with the above method.

In an embodiment of step P7, the processor30performs Fourier analysis and low-pass filtering on the pieces of phase error data to build the phase error model.FIG.5Eis an example of building the phase error model with the above method, wherein this phase error model may be used to compensate the pieces of target phase data in step S2.

In an embodiment, in the process of steps P1˜P7described above further includes a step of horizontal calibration. In detail, the processor30computing the first phase corresponding to the first calibration surface and the second phase corresponding to the second calibration surface according to the phase data further includes: according to a reference straight line in the at least one reference object image passing through the first calibration surface and the second calibration surface, the processor30obtains a plurality of pieces of reference phase data among the pieces of phase data locating at the reference straight line; the processor30obtains at least one group of phase data from the pieces of reference phase data, wherein the at least one group of phase data comprises at least two pieces of the pieces of reference phase data, and the at least one group of phase data corresponds to the first calibration surface or the second calibration surface; the processor30performs horizontal calibration on the pieces of phase data according to the at least one group of phase data; and the processor30computes the first phase corresponding to the first calibration surface and the second phase corresponding to the second calibration surface according to the pieces of reference phase data after performed with the horizontal calibration.

Regarding the method of generating the phase-height conversion parameter k described in step S5, in detail, the processor30is configured to execute commands, for example, to execute a plurality of third commands to generate the phase-height conversion parameter k, wherein the process corresponding to these third commands are shown inFIG.6.FIG.6is a flowchart of calibration method of three-dimensional measurement system of an embodiment of the disclosure, wherein this method is for generating the phase-height conversion parameter k. Step U1is “projecting the structural light to a reference object”, step U2is “photographing the reference object to generate a reference object image”, step U3is “obtaining a plurality of pieces of phase raw data according to the reference object image”, step U4is “compensating the pieces of phase raw data based on the phase error model”, step U5is “performing the horizontal calibration on the pieces of phase raw data”, and step U6is “calculating the phase difference and the phase-height conversion parameter according to these pieces of phase raw data”.

In step U1and step U2, the processor30executing third commands controls the projection device10to the camera20to perform corresponding operations, respectively.FIG.7is an example of the reference object50. In step U1, the reference object50includes the first calibration surface and the second calibration surface. In an embodiment, the reference object50includes a step height block52and a substrate54, and the step height block52is disposed on the substrate54. The first calibration surface is an upper surface (top surface) of the step height block52, and the second calibration surface is an upper surface (top surface) of the substrate54. In step U2, the camera20photographs the reference object50to obtain at least one reference object image, the disclosure does not limit the number of reference object images. In an embodiment, the same structural light pattern may be presented in different phases. Therefore, in steps U1to U2, the projection device10may project structural light with different phases, and the camera20obtains image one by one.

In an embodiment, a plurality of attributes of the step height block52and the substrate54may be adjusted according to requirements, wherein these attributes includes: a size of a top surface of the step height block52(may be represented by X52, Y52), a height difference Z52of the step height block52relative to the substrate54, a number of the step height block52, a location of the step height block52disposed on the substrate54, and a size of a top surface of the substrate54(may be represented by X54, Y54).

Regarding the sizes of the top surfaces X52. Y52of the step height block52, in an embodiment of the reference object image in step U2, the length X52and width Y52of the top surface of the step height block52are both at least 20 pixels, but the disclosure is not limited to the above numbers. In practice, a distance between the lens of the camera20and the step height block52as well as photographing angle may be adaptively adjusted according to a plurality of parameters (for example, focal length of the camera, the resolution of the reference object image), for the sizes of the top surfaces X52, Y52of the step height block52in the reference object image exceed default values.

Regarding the height difference Z52of the step height block52relative to the substrate54, its value depends on the Z-axis measurement precision required by the 3D measurement system100. In an embodiment, the height difference Z52is more than 10 times of the measurement precision. For example, if the precision is 1 micrometer (μm), the height difference Z52must be greater than or equal to 10 micrometers.

FIG.8is an example of the reference object50′ having two step height blocks521,522. As shown inFIG.8, the step height block522is disposed on the substrate54′, and the step height block521is disposed on the step height block522. The size of the top surface of the step height block521is smaller than the size of the top surface of the step height block522. In a different perspective, the two step height blocks521,522compose a two-level step height block52′ as a whole. Therefore, the step height block52′ has two height differences Z521, Z522relative to the substrate54. The disclosure does not limit the number of the step height block (height levels). Under the premise of not significantly affecting calibration speed, higher number (number of steps) of step height blocks may increase the precision of calibration.

Regarding the location of the step height block52disposed on the substrate54, the disclosure is not limited thereto.

Regarding the sizes of the top surfaces X54, Y54of the substrate54, please refer toFIG.7. In detail, the processor30may find an image region in the reference object image corresponding to the substrate54. In an embodiment, a size of this image region is at least ⅔ of the size of the reference object image. In other words, the length and width of the substrate54is at least ⅔ of the field of view F of the 3D measurement system100. The disclosure is not limited to the above values, and does not limit the shape of the substrate54being rectangle.

In step U3, the processor30performs the decoding computation to obtain a plurality of pieces of phase raw data of the at least one reference object image according to the at least one reference object image, wherein the method of performing the decoding computation to obtain a plurality of pieces of phase raw data is basically the same as the processor30obtaining a plurality of pieces of modeling data according to the error calibration surface image I2in step P3, the same process is not repeated herein.

In step U4, the processor30uses the phase error model built in step P7to compensate a plurality of pieces of phase raw data of the at least one reference object image. In other words, step P7has to be executed completely before step U4. In an embodiment, step U4may be omitted for step U5to be directly followed by step U3.

In step U5, the method of the processor30performing horizontal calibration on the pieces of phase raw data is basically the same as the processor30performing horizontal calibration on the pieces of target phase data in step S3, the same process is not repeated herein.

In step U6, the processor30computes the first phase corresponding to the first calibration surface and the second phase corresponding to the second calibration surface according to (calibrated and after horizontal calibration) the pieces of phase raw data. In an embodiment, the processor30obtains a plurality of the pieces of reference phase data locating at the reference straight line from these pieces of phase raw data according to a reference straight line in the at least one reference object image passing through the first calibration surface and the second calibration surface. The processor30computes the first phase corresponding to the first calibration surface and the second phase corresponding to the second calibration surface according to these pieces of reference phase data. In practice, the concept of the above-mentioned reference straight line is similar to the scan line L1passing through the error calibration surface42and the target object40shown inFIG.3A. In an embodiment, the first phase is an average of a plurality of the pieces of reference phase data of the first calibration surface, and the second phase is an average of a plurality of the pieces of reference phase data of the second calibration surface.

The processor30calculates a surface phase difference between the first phase and the second phase, then calculates a height of the second calibration surface relative to the first calibration surface according to the surface phase difference to obtain the phase-height conversion parameter k, wherein the method of calculating the phase difference is similar to step S4. Please refer toFIG.9, whereinFIG.9is a schematic diagram of the phase difference ΔØ0of the step height block52. The phase difference ΔØ0is associated with the first phase B1and the second phase B2, the first phase B1is associated with phase data corresponding to the substrate54among a plurality of pieces of phase data, the second phase B2is associated with a plurality of pieces of phase data corresponding to the step height block52among a plurality of pieces of phase data, and the phase-height conversion parameter k is associated with the height Z52of the step height block52and the phase difference ΔØ0.

In an embodiment, the phase difference is a difference value of ΔØ0the first phase B1and the second phase B2. That is, ΔØ0=B2−B1.

In an embodiment, the first phase B1corresponds to the average of a plurality of pieces of phase data of the substrate54and corresponds to a plurality of pieces of phase data of the substrate54, for example, phase data of the coordinates 0˜900 and phase data of the coordinates 1600˜2500. The second phase B2is the average of a plurality of pieces of phase data corresponding to the step height block52and corresponds to a plurality of pieces of phase data of the step height block52, for example, phase data of the coordinates 900˜1600.

In an embodiment, the phase-height conversion parameter k is a quotient of the height of the second calibration surface relative to the first calibration surface divided by the surface phase difference. In an embodiment, the phase-height conversion parameter k is a quotient of the height Z52of the step height block52divided by the phase difference ΔØ0, that is,

k=Z5⁢2Δ∅0.
After calculating the phase-height conversion parameter k, the phase-height conversion parameter k may be used in step S5inFIG.2to calculate the height of the target object40.

In the process shown inFIG.6, to elaborate the advantageous effects of the phase error model, please refer toFIG.10.FIG.10is a comparison of the height error heatmaps of the step height blocks at different locations before and after compensation. The unit of the height error inFIG.10is micrometer (μm), andFIGS.11A to11Care examples of the pieces of phase error data before and after compensation.

FIG.10shows the substrate54before and after compensation in a top view, and the grids in the substrate54represent a plurality of allowed locations for disposing the step height block52. When the step height block52is disposed on the locations52a,52b,52cof the substrate54, the phase diagram examples of before and after compensation are respectively shown inFIG.11A,FIG.11BandFIG.11C. It may be known according to two heatmaps of height errors shown inFIG.10that: before performing compensation using the phase error model in step S2or step U4, if the step height block52is disposed at the corner of the substrate54, the calculated height of the target object40has a relative large measurement error; after performing compensation using the phase error model in step S2or step U4, not matter which location of the substrate54the step height block52is disposed at, the calculated height of the target object40has the same error. This means the phase-height conversion parameter k generated with an embodiment of the disclosure is adapted to the entire field of view F that may be photographed by the camera20.

Please refer toFIG.1andFIG.7. In an embodiment, the disclosure provides a 3D measurement system based on phase shift, said system includes: the reference object50, the projection device10, the camera20and the processor30. The reference object50includes the first calibration surface (for example, the top surface of the step height block52) and the second calibration surface (for example, the top surface of the substrate54). The projection device10is configured to project the structural light SL to the reference object50. The camera20is configured to photograph the reference object50to obtain the at least one reference object image. The processor30is electrically connected to the camera20and the projection device10, wherein the processor30is configured to: performing the decoding computation according to the at least one reference object image to obtain a plurality of pieces of phase data of the at least one reference object image; calculating the first phase corresponding to the first calibration surface and the second phase corresponding to the second calibration surface according to these pieces of phase data; and calculating the surface phase difference between the first phase and the second phase, and calculating the height of the second calibration surface relative to the first calibration surface according to the surface phase difference to obtain the phase-height conversion parameter k.

In view of the above description, the disclosure provides a three-dimensional measurement system based on phase shift, height measuring method and method of generating phase-height conversion parameter. The disclosure may only need to photograph the reference object image of a single step height block and use the phase data in the reference object image to generate the phase-height conversion parameter needed for the following calibration. The generation of the phase-height conversion parameter is fast and is suitable for the entire picture. No matter which location the step height block is disposed at, the phase-height conversion parameter generated through the application of the disclosure has stability. In addition, the phase error model provided by the disclosure may perform adaptive phase compensation on the 3D measurement system, reduce phase error and improve accuracy when building 3D point cloud model.