Patent ID: 12198414

DETAILED DESCRIPTION

The technical solutions described herein generally relate to techniques for upscaling triangulation scanner images to reduce noise. A three-dimensional (3D) scanning device or “scanner” as depicted inFIG.1can be used to generate 3D points (referred to as a “point cloud”).

In particular,FIG.1depicts a system100for scanning an environment according to one or more embodiments described herein. The system100includes a computing device110coupled with a scanner120, which can be a 3D scanner or another suitable scanner. The coupling facilitates wired and/or wireless communication between the computing device110and the scanner120. The scanner120includes a set of sensors122. The set of sensors122can include different types of sensors, such as LIDAR sensor122A (light detection and ranging), RGB-D camera122B (red-green-blue-depth), and wide-angle/fisheye camera122C, and other types of sensors. The scanner120can also include an inertial measurement unit (IMU)126to keep track of a 3D movement and orientation of the scanner120. The scanner120can further include a processor124that, in turn, includes one or more processing units. The processor124controls the measurements performed using the set of sensors122. In one or more examples, the measurements are performed based on one or more instructions received from the computing device110. In an embodiment, the LIDAR sensor122A is a two-dimensional (2D) scanner that sweeps a line of light in a plane (e.g. a plane horizontal to the floor).

According to one or more embodiments described herein, the scanner120is a dynamic machine vision sensor (DMVS) scanner manufactured by FARO® Technologies, Inc. of Lake Mary, Florida, USA. DMVS scanners are discussed further with reference toFIGS.11A-18. In an embodiment, the scanner120may be that described in commonly owned United States Patent Publication 2018/0321383 entitled Triangulation Scanner having Flat Geometry and Projecting Uncoded Spots, the contents of which are incorporated by reference herein. It should be appreciated that the techniques described herein are not limited to use with DMVS scanners and that other types of 3D scanners can be used.

The computing device110can be a desktop computer, a laptop computer, a tablet computer, a phone, or any other type of computing device that can communicate with the scanner120.

In one or more embodiments, the computing device110generates a point cloud130(e.g., a 3D point cloud) of the environment being scanned by the scanner120using the set of sensors122. The point cloud130is a set of data points (i.e., a collection of three-dimensional coordinates) that correspond to surfaces of objects in the environment being scanned and/or of the environment itself. According to one or more embodiments described herein, a display (not shown) displays a live view of the point cloud130.

Turning now to an overview of technologies that are more specifically relevant to one or more embodiments described herein, triangulation scanners (see, e.g., the scanner120ofFIG.1and/or the triangulation scanner1101ofFIGS.11A-11E) generally include at least one projector and at least one camera. The projector and camera are separated by a baseline distance. Images of the laser projection pattern are used to generate 3D points. An example image200of a laser projection pattern is depicted inFIG.2. Due to the nature of triangulation scanners, 3D data is typically very noise on longer distances.

For scanners such as the triangulation scanner1101, a typical 2-sigma noise might be 500 m at a 500 mm measurement distance. In some applications, sensitivity for finding defects may be less than the 2-sigma noise (e.g., less than 500 m such as about 300 m). This prevent can prevent the use of such scanners for certain applications. The reason for this is a combination of camera and laser noise. For camera noise, pixel size plays an important role. For example,FIG.3depicts a portion300of the image200ofFIG.3showing a laser dot302that has been enlarged. In this example, the portion300is enlarged by about 1300%, although this enlargement amount is not intended to be limited and is only exemplary. As can be seen, the laser dot302is pixelated, and the individual pixels are easy to identify. To minimize this effect, pixel size must be decreased. To do this, the image200is upscaled as shown inFIG.4. This can be accomplished in various ways, including by replacing the camera of the scanner with a model that captures images in a higher resolution or by upscaling the image in software.

Substituting or replacing the camera in a scanner is a financially expensive approach, and upscaling the image in software is a computationally expensive approach. Upscaling using conventional techniques is computational expensive driven largely by interpolating between the pixels. For example, inFIG.4, an off-the-shelf photo editing application was used to perform the upscaling on the image200to generate the portion400. However, when using triangulation scanners such as a DMVS (which captures images, for example, at about 70 Hz), high performance is desired, and generating 3D data occupies a large amount of processing time. Upscaling and interpolating the images using conventional algorithm-based techniques would have a large negative impact on the processing time and thus impact the image capture speed, which would be reduced undesirably. The embodiments described herein provide for software-based upscaling without reducing image capture speed of the scanner by using artificial intelligence to upscale and interpolate images.

FIG.5depicts a flow diagram of a method500for upscaling triangulation scanner images to reduce noise according to one or more embodiments described herein. The method500can be performed by any suitable processing system, processing device, scanner, etc. such as the processing systems, processing devices, and scanners described herein. For example, the processor124is disposed in a three-dimensional scanner (e.g., the scanner120) such that performing the image recognition and performing the upscaling are performed by the three-dimensional scanner.

According to one or more embodiments described herein, the techniques for upscaling triangulation scanner images provided herein are a fully automated process that uses machine learning to perform pattern recognition and determine how edges and shapes within an image should look while increasing the overall size of an image. This process has been trained on large datasets, allowing it to accurately clear up images. In particular, the image data is not manipulated; rather the patterns, colors, and shapes in the image are recognized. This is referred to as a “raw data pattern.” After the raw data pattern is recognized in the image, a neural network is applied to deconvolute the pixel intensity. If this were performed conventionally, in a larger image with blurry edges and colors would result (see, e.g.,FIG.3). However, by training a neural network to perform the deconvolution only on the rasterization of the image, the real image is maintained and enhanced by presenting it better to the user and/or for further processing. This results in more pixels to work with and unmanipulated data, which in turns improves precision, such as for a laser raster to map point-to-pixel approach. Accordingly, faster and better results are achieved. The method500is now described in more detail.

At block502, a neural network is trained to perform image upscaling on images captured by a 3D triangulation scanner (e.g., the scanner120ofFIG.1, the triangulation scanner1101ofFIGS.7,8,9,10, and11, etc.). As described herein, a neural network can be trained to perform image upscaling, which is useful for reducing noise in scanned images, for example. More specifically, the present techniques can incorporate and utilize rule-based decision making and artificial intelligence (AI) reasoning to accomplish the various operations described herein, namely upscaling images, such as scanned images from triangulation scanners. The phrase “machine learning” broadly describes a function of electronic systems that learn from data. A machine learning system, engine, or module can include a trainable machine learning algorithm that can be trained, such as in an external cloud environment, to learn functional relationships between inputs and outputs that are currently unknown, and the resulting model can be used for automatically upscaling images. In one or more embodiments, machine learning functionality can be implemented using an artificial neural network (ANN) having the capability to be trained to perform a currently unknown function. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, and in particular the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNN) are a class of deep, feed-forward ANN that are particularly useful at analyzing visual imagery.

ANNs can be embodied as so-called “neuromorphic” systems of interconnected processor elements that act as simulated “neurons” and exchange “messages” between each other in the form of electronic signals. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in ANNs that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making ANNs adaptive to inputs and capable of learning. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by the pixels of an input image. After being weighted and transformed by a function determined by the network's designer, the activation of these input neurons are then passed to other downstream neurons, which are often referred to as “hidden” neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was read. It should be appreciated that these same techniques can be applied in the case of upscaling images.

To train the neural network, set of images (referred to as a training set of images) are created using, for example, a photo editing application. The training set of images includes pairs of images: an original image of a laser projection pattern and a downscaled image of the laser projection pattern. The downscaled image is a manually worsened version of the original image. For example, if the original image is a 1024×1024 image, the downscaled image is manually worsened to 256×256 and is then compared against the original image.

A neural network can be designed with a given depth and architecture particular to a specific scanner, such as a DMVS or other suitable scanner. According to one or more embodiments described herein, an autoencoder and autodecoder technique is applied with an intermediate long short-term memory (LSTM) layer chain between the encoding and decoding blocks. For example,FIGS.6A and6Bdepict an autoencoder602that implements machine learning according to one or more embodiments described herein. As shown inFIG.6A, the autoencoder602receives a scanned image601as an input and produces an upscaled image603as an output. An autoencoder, such as the autoencoder602, uses a neural network that learns in an unsupervised way. Autoencoders can be used in a variety of applications, such as dimensionality reduction, anomaly detection, denoising, etc. According to one or more embodiments described herein, the autoencoder602can be trained to recognize certain information in input data (e.g., the scanned image601). As one example, an autoencoder can be trained to recognize real information, such as handwriting, in a noisy image and to produce the recognized information without surrounding noise as an upscaled image (e.g., the upscaled image603). In examples, the output is a binarized image or an image that is capable of being binarized. An autoencoder can be trained to find real information in images with different segments with different gray value levels and process this segment information.

FIG.6Bdepicts the autoencoder602in more detail. In this example, the autoencoder602includes an encoder610that receives the scanned image601and a decoder620that produces the upscaled image602. The encoder610includes an input layer611(labeled as “X”), and the decoder620includes an output layer621(labeled as “X′”). The input layers611and the output layer621use an activation function, which may be non-linear. An example of an activation function is a rectified linear unit (ReLU). Each of the encoder610and the decoder620utilizes code630(labeled as “h”) in a latent space between the input layer611and the output layer621to perform denoising. In some examples, the code630can include the intermediate LSTM layer chain between the encoding and decoding blocks.

In an example, the neural network is trained all around purposes. That is, a model can be trained on images/data from multiple sources (e.g., customers) to produce a general model applicable across multiple data sets. In another example, the neural network is trained for a particular scanning implementation, which may be a perfect fit for a particular customer based on that customer's images/data, since the trained model is a perfect fit for the customer's particular environment/use.

After the neural network is trained, it can be used as an evaluation script to evaluate scanned images from the scanner. The scanned images, which include a laser pattern, are upscaled using the trained neural network. The benefit of this approach is high precision, taken into consideration that the overhead created in the chain by the upscaling step can be done in real-time or near-real-time. For example, at block504of the method500, the 3D triangulation scanner captures an image as described herein. Once the image is captured, the trained neural network is applied to upscale the image at block506and506.

Particularly, at block506, the image is input into the neural network, and the neural network performs pattern recognition on the image. This can include recognizing a pattern, a color, a shape, etc. in the image. For example, inFIG.2, the laser dot302is recognized as having a circular shape. At block508, the neural network is used to perform upscaling of the image to increase the resolution of the image while maintaining the pattern, color, shape, etc. of the original image to generate an upscaled image (see, e.g.,FIG.5). As shown inFIG.5, the upscaled image is a higher resolution compared to a non-upscaled image (see, e.g.,FIG.4).

Additional processes also may be included, and it should be understood that the process depicted inFIG.5represents an illustration, and that other processes may be added or existing processes may be removed, modified, or rearranged without departing from the scope of the present disclosure.

Turning now toFIG.7, it may be desired to capture three-dimensional (3D) measurements of objects. For example, the point cloud130ofFIG.1may be captured by the scanner120. One such example of the scanner120is now described. Such example scanner is referred to as a DVMS scanner by FARO®.

In an embodiment illustrated inFIGS.7,8,9,10, and11, a triangulation scanner1101includes a body1105, a projector1120, a first camera1130, and a second camera1140. In an embodiment, the projector optical axis1122of the projector1120, the first-camera optical axis1132of the first camera1130, and the second-camera optical axis1142of the second camera1140all lie on a common plane1150, as shown inFIGS.9,10. In some embodiments, an optical axis passes through a center of symmetry of an optical system, which might be a projector or a camera, for example. For example, an optical axis may pass through a center of curvature of lens surfaces or mirror surfaces in an optical system. The common plane1150, also referred to as a first plane1150, extends perpendicular into and out of the paper inFIG.10.

In an embodiment, the body1105includes a bottom support structure1106, a top support structure1107, spacers1108, camera mounting plates1109, bottom mounts1110, dress cover1111, windows1112for the projector and cameras, Ethernet connectors1113, and GPIO connector1114. In addition, the body includes a front side1115and a back side1116. In an embodiment, the bottom support structure1106and the top support structure1107are flat plates made of carbon-fiber composite material. In an embodiment, the carbon-fiber composite material has a low coefficient of thermal expansion (CTE). In an embodiment, the spacers1108are made of aluminum and are sized to provide a common separation between the bottom support structure1106and the top support structure1107.

In an embodiment, the projector1120includes a projector body1124and a projector front surface1126. In an embodiment, the projector1120includes a light source1125that attaches to the projector body1124that includes a turning mirror and a diffractive optical element (DOE), as explained herein below with respect toFIGS.15A,15B,15C. The light source1125may be a laser, a superluminescent diode, or a partially coherent LED, for example. In an embodiment, the DOE produces an array of spots arranged in a regular pattern. In an embodiment, the projector1120emits light at a near infrared wavelength.

In an embodiment, the first camera1130includes a first-camera body1134and a first-camera front surface36. In an embodiment, the first camera includes a lens, a photosensitive array, and camera electronics. The first camera1130forms on the photosensitive array a first image of the uncoded spots projected onto an object by the projector1120. In an embodiment, the first camera responds to near infrared light.

In an embodiment, the second camera1140includes a second-camera body1144and a second-camera front surface1146. In an embodiment, the second camera includes a lens, a photosensitive array, and camera electronics. The second camera1140forms a second image of the uncoded spots projected onto an object by the projector1120. In an embodiment, the second camera responds to light in the near infrared spectrum. In an embodiment, a processor1102is used to determine 3D coordinates of points on an object according to methods described herein below. The processor1102may be included inside the body1105or may be external to the body. In further embodiments, more than one processor is used. In still further embodiments, the processor1102may be remotely located from the triangulation scanner.

FIG.11is a top view of the triangulation scanner1101. A projector ray1128extends along the projector optical axis from the body of the projector1124through the projector front surface1126. In doing so, the projector ray1128passes through the front side1115. A first-camera ray1138extends along the first-camera optical axis1132from the body of the first camera1134through the first-camera front surface1136. In doing so, the front-camera ray1138passes through the front side1115. A second-camera ray1148extends along the second-camera optical axis1142from the body of the second camera1144through the second-camera front surface1146. In doing so, the second-camera ray1148passes through the front side1115.

FIG.12Ashows elements of a triangulation scanner1200athat might, for example, be the triangulation scanner1101shown inFIGS.7-11. In an embodiment, the triangulation scanner1200aincludes a projector1250, a first camera1210, and a second camera1230. In an embodiment, the projector1250creates a pattern of light on a pattern generator plane1252. An exemplary corrected point1253on the pattern projects a ray of light1251through the perspective center1258(point D) of the lens1254onto an object surface1270at a point1272(point F). The point1272is imaged by the first camera1210by receiving a ray of light from the point1272through the perspective center1218(point E) of the lens1214onto the surface of a photosensitive array1212of the camera as a corrected point1220. The point1220is corrected in the read-out data by applying a correction value to remove the effects of lens aberrations. The point1272is likewise imaged by the second camera1230by receiving a ray of light from the point1272through the perspective center1238(point C) of the lens1234onto the surface of the photosensitive array1232of the second camera as a corrected point1235. It should be understood that as used herein any reference to a lens includes any type of lens system whether a single lens or multiple lens elements, including an aperture within the lens system. It should be understood that any reference to a projector in this document refers not only to a system projecting with a lens or lens system an image plane to an object plane. The projector does not necessarily have a physical pattern-generating plane1252but may have any other set of elements that generate a pattern. For example, in a projector having a DOE, the diverging spots of light may be traced backward to obtain a perspective center for the projector and also to obtain a reference projector plane that appears to generate the pattern. In most cases, the projectors described herein propagate uncoded spots of light in an uncoded pattern. However, a projector may further be operable to project coded spots of light, to project in a coded pattern, or to project coded spots of light in a coded pattern. In other words, in some aspects of the disclosed embodiments, the projector is at least operable to project uncoded spots in an uncoded pattern but may in addition project in other coded elements and coded patterns.

In an embodiment where the triangulation scanner1200aofFIG.12Ais a single-shot scanner that determines 3D coordinates based on a single projection of a projection pattern and a single image captured by each of the two cameras, then a correspondence between the projector point1253, the image point1220, and the image point1235may be obtained by matching a coded pattern projected by the projector1250and received by the two cameras1210,1230. Alternatively, the coded pattern may be matched for two of the three elements—for example, the two cameras1210,1230or for the projector1250and one of the two cameras1210or1230. This is possible in a single-shot triangulation scanner because of coding in the projected elements or in the projected pattern or both.

After a correspondence is determined among projected and imaged elements, a triangulation calculation is performed to determine 3D coordinates of the projected element on an object. ForFIG.12A, the elements are uncoded spots projected in a uncoded pattern. In an embodiment, a triangulation calculation is performed based on selection of a spot for which correspondence has been obtained on each of two cameras. In this embodiment, the relative position and orientation of the two cameras is used. For example, the baseline distance B3between the perspective centers1218and1238is used to perform a triangulation calculation based on the first image of the first camera1210and on the second image of the second camera1230. Likewise, the baseline B1is used to perform a triangulation calculation based on the projected pattern of the projector1250and on the second image of the second camera1230. Similarly, the baseline B2is used to perform a triangulation calculation based on the projected pattern of the projector1250and on the first image of the first camera1210. In an embodiment, the correspondence is determined based at least on an uncoded pattern of uncoded elements projected by the projector, a first image of the uncoded pattern captured by the first camera, and a second image of the uncoded pattern captured by the second camera. In an embodiment, the correspondence is further based at least in part on a position of the projector, the first camera, and the second camera. In a further embodiment, the correspondence is further based at least in part on an orientation of the projector, the first camera, and the second camera.

The term “uncoded element” or “uncoded spot” as used herein refers to a projected or imaged element that includes no internal structure that enables it to be distinguished from other uncoded elements that are projected or imaged. The term “uncoded pattern” as used herein refers to a pattern in which information is not encoded in the relative positions of projected or imaged elements. For example, one method for encoding information into a projected pattern is to project a quasi-random pattern of “dots” in which the relative position of the dots is known ahead of time and can be used to determine correspondence of elements in two images or in a projection and an image. Such a quasi-random pattern contains information that may be used to establish correspondence among points and hence is not an example of a uncoded pattern. An example of an uncoded pattern is a rectilinear pattern of projected pattern elements.

In an embodiment, uncoded spots are projected in an uncoded pattern as illustrated in the scanner system12100ofFIG.12B. In an embodiment, the scanner system12100includes a projector12110, a first camera12130, a second camera12140, and a processor12150. The projector projects an uncoded pattern of uncoded spots off a projector reference plane12114. In an embodiment illustrated inFIGS.12B and12C, the uncoded pattern of uncoded spots is a rectilinear array12111of circular spots that form illuminated object spots12121on the object12120. In an embodiment, the rectilinear array of spots12111arriving at the object12120is modified or distorted into the pattern of illuminated object spots12121according to the characteristics of the object12120. An exemplary uncoded spot12112from within the projected rectilinear array12111is projected onto the object12120as a spot12122. The direction from the projector spot12112to the illuminated object spot12122may be found by drawing a straight line12124from the projector spot12112on the reference plane12114through the projector perspective center12116. The location of the projector perspective center12116is determined by the characteristics of the projector optical system.

In an embodiment, the illuminated object spot12122produces a first image spot12134on the first image plane12136of the first camera12130. The direction from the first image spot to the illuminated object spot12122may be found by drawing a straight line12126from the first image spot12134through the first camera perspective center12132. The location of the first camera perspective center12132is determined by the characteristics of the first camera optical system.

In an embodiment, the illuminated object spot12122produces a second image spot12144on the second image plane12146of the second camera12140. The direction from the second image spot12144to the illuminated object spot12122may be found by drawing a straight line12126from the second image spot12144through the second camera perspective center12142. The location of the second camera perspective center12142is determined by the characteristics of the second camera optical system.

In an embodiment, a processor12150is in communication with the projector12110, the first camera12130, and the second camera12140. Either wired or wireless channels12151may be used to establish connection among the processor12150, the projector12110, the first camera12130, and the second camera12140. The processor may include a single processing unit or multiple processing units and may include components such as microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and other electrical components. The processor may be local to a scanner system that includes the projector, first camera, and second camera, or it may be distributed and may include networked processors. The term processor encompasses any type of computational electronics and may include memory storage elements.

FIG.12Eshows elements of a method12180for determining 3D coordinates of points on an object. An element12182includes projecting, with a projector, a first uncoded pattern of uncoded spots to form illuminated object spots on an object.FIGS.12B,12Cillustrate this element12182using an embodiment12100in which a projector12110projects a first uncoded pattern of uncoded spots12111to form illuminated object spots12121on an object12120.

A method element12184includes capturing with a first camera the illuminated object spots as first-image spots in a first image. This element is illustrated inFIG.12Busing an embodiment in which a first camera12130captures illuminated object spots12121, including the first-image spot12134, which is an image of the illuminated object spot12122. A method element12186includes capturing with a second camera the illuminated object spots as second-image spots in a second image. This element is illustrated inFIG.12Busing an embodiment in which a second camera140captures illuminated object spots12121, including the second-image spot12144, which is an image of the illuminated object spot12122.

A first aspect of method element12188includes determining with a processor 3D coordinates of a first collection of points on the object based at least in part on the first uncoded pattern of uncoded spots, the first image, the second image, the relative positions of the projector, the first camera, and the second camera, and a selected plurality of intersection sets. This aspect of the element12188is illustrated inFIGS.12B,12Cusing an embodiment in which the processor12150determines the 3D coordinates of a first collection of points corresponding to object spots12121on the object12120based at least in the first uncoded pattern of uncoded spots12111, the first image12136, the second image12146, the relative positions of the projector12110, the first camera12130, and the second camera12140, and a selected plurality of intersection sets. An example fromFIG.12Bof an intersection set is the set that includes the points12112,12134, and12144. Any two of these three points may be used to perform a triangulation calculation to obtain 3D coordinates of the illuminated object spot12122as discussed herein above in reference toFIGS.12A,12B.

A second aspect of the method element12188includes selecting with the processor a plurality of intersection sets, each intersection set including a first spot, a second spot, and a third spot, the first spot being one of the uncoded spots in the projector reference plane, the second spot being one of the first-image spots, the third spot being one of the second-image spots, the selecting of each intersection set based at least in part on the nearness of intersection of a first line, a second line, and a third line, the first line being a line drawn from the first spot through the projector perspective center, the second line being a line drawn from the second spot through the first-camera perspective center, the third line being a line drawn from the third spot through the second-camera perspective center. This aspect of the element12188is illustrated inFIG.12Busing an embodiment in which one intersection set includes the first spot12112, the second spot12134, and the third spot12144. In this embodiment, the first line is the line12124, the second line is the line12126, and the third line is the line12128. The first line12124is drawn from the uncoded spot12112in the projector reference plane12114through the projector perspective center12116. The second line12126is drawn from the first-image spot12134through the first-camera perspective center12132. The third line12128is drawn from the second-image spot12144through the second-camera perspective center12142. The processor12150selects intersection sets based at least in part on the nearness of intersection of the first line12124, the second line12126, and the third line12128.

The processor12150may determine the nearness of intersection of the first line, the second line, and the third line based on any of a variety of criteria. For example, in an embodiment, the criterion for the nearness of intersection is based on a distance between a first 3D point and a second 3D point. In an embodiment, the first 3D point is found by performing a triangulation calculation using the first image point12134and the second image point12144, with the baseline distance used in the triangulation calculation being the distance between the perspective centers12132and12142. In the embodiment, the second 3D point is found by performing a triangulation calculation using the first image point12134and the projector point12112, with the baseline distance used in the triangulation calculation being the distance between the perspective centers12134and12116. If the three lines12124,12126, and12128nearly intersect at the object point12122, then the calculation of the distance between the first 3D point and the second 3D point will result in a relatively small distance. On the other hand, a relatively large distance between the first 3D point and the second 3D would indicate that the points12112,12134, and12144did not all correspond to the object point12122.

As another example, in an embodiment, the criterion for the nearness of the intersection is based on a maximum of closest-approach distances between each of the three pairs of lines. This situation is illustrated inFIG.12D. A line of closest approach12125is drawn between the lines12124and12126. The line12125is perpendicular to each of the lines12124,12126and has a nearness-of-intersection length a. A line of closest approach12127is drawn between the lines12126and12128. The line12127is perpendicular to each of the lines12126,12128and has length b. A line of closest approach12129is drawn between the lines12124and12128. The line12129is perpendicular to each of the lines12124,12128and has length c. According to the criterion described in the embodiment above, the value to be considered is the maximum of a, b, and c. A relatively small maximum value would indicate that points12112,12134, and12144have been correctly selected as corresponding to the illuminated object point12122. A relatively large maximum value would indicate that points12112,12134, and12144were incorrectly selected as corresponding to the illuminated object point12122.

The processor12150may use many other criteria to establish the nearness of intersection. For example, for the case in which the three lines were coplanar, a circle inscribed in a triangle formed from the intersecting lines would be expected to have a relatively small radius if the three points12112,12134,12144corresponded to the object point12122. For the case in which the three lines were not coplanar, a sphere having tangent points contacting the three lines would be expected to have a relatively small radius.

It should be noted that the selecting of intersection sets based at least in part on a nearness of intersection of the first line, the second line, and the third line is not used in most other projector-camera methods based on triangulation. For example, for the case in which the projected points are coded points, which is to say, recognizable as corresponding when compared on projection and image planes, there is no need to determine a nearness of intersection of the projected and imaged elements. Likewise, when a sequential method is used, such as the sequential projection of phase-shifted sinusoidal patterns, there is no need to determine the nearness of intersection as the correspondence among projected and imaged points is determined based on a pixel-by-pixel comparison of phase determined based on sequential readings of optical power projected by the projector and received by the camera(s). The method element12190includes storing 3D coordinates of the first collection of points.

An alternative method that uses the intersection of epipolar lines on epipolar planes to establish correspondence among uncoded points projected in an uncoded pattern is described in U.S. Pat. No. 9,599,455 (′455) to Heidemann, et al., the contents of which are incorporated by reference herein. In an embodiment of the method described in patent '455, a triangulation scanner places a projector and two cameras in a triangular pattern. An example of a triangulation scanner1300having such a triangular pattern is shown inFIG.13. The triangulation scanner1300includes a projector1350, a first camera1310, and a second camera1330arranged in a triangle having sides A1-A2-A3. In an embodiment, the triangulation scanner1300may further include an additional camera1390not used for triangulation but to assist in registration and colorization.

Referring now toFIG.14the epipolar relationships for a 3D imager (triangulation scanner)1490correspond with 3D imager1300ofFIG.13in which two cameras and one projector are arranged in the shape of a triangle having sides1402,1404,1406. In general, the device1, device2, and device3may be any combination of cameras and projectors as long as at least one of the devices is a camera. Each of the three devices1491,1492,1493has a perspective center O1, O2, O3, respectively, and a reference plane1460,1470, and1480, respectively. InFIG.14, the reference planes1460,1470,1480are epipolar planes corresponding to physical planes such as an image plane of a photosensitive array or a projector plane of a projector pattern generator surface but with the planes projected to mathematically equivalent positions opposite the perspective centers O1, O2, O3. Each pair of devices has a pair of epipoles, which are points at which lines drawn between perspective centers intersect the epipolar planes. Device1and device2have epipoles E12, E21on the planes1460,1470, respectively. Device1and device3have epipoles E13, E31, respectively on the planes1460,1480, respectively. Device2and device3have epipoles E23, E32on the planes1470,1480, respectively. In other words, each reference plane includes two epipoles. The reference plane for device1includes epipoles E12and E13. The reference plane for device2includes epipoles E21and E23. The reference plane for device3includes epipoles E31and E32.

In an embodiment, the device3is a projector1493, the device1is a first camera1491, and the device2is a second camera1492. Suppose that a projection point P3, a first image point P1, and a second image point P2are obtained in a measurement. These results can be checked for consistency in the following way.

To check the consistency of the image point P1, intersect the plane P3-E31-E13with the reference plane1460to obtain the epipolar line1464. Intersect the plane P2-E21-E12to obtain the epipolar line1462. If the image point P1has been determined consistently, the observed image point P1will lie on the intersection of the determined epipolar lines1462and1464.

To check the consistency of the image point P2, intersect the plane P3-E32-E23with the reference plane1470to obtain the epipolar line1474. Intersect the plane P1-E12-E21to obtain the epipolar line1472. If the image point P2has been determined consistently, the observed image point P2will lie on the intersection of the determined epipolar lines1472and1474.

To check the consistency of the projection point P3, intersect the plane P2-E23-E32with the reference plane1480to obtain the epipolar line1484. Intersect the plane P1-E13-E31to obtain the epipolar line1482. If the projection point P3has been determined consistently, the projection point P3will lie on the intersection of the determined epipolar lines1482and1484.

It should be appreciated that since the geometric configuration of device1, device2and device3are known, when the projector1493emits a point of light onto a point on an object that is imaged by cameras1491,1492, the 3D coordinates of the point in the frame of reference of the 3D imager1490may be determined using triangulation methods.

Note that the approach described herein above with respect toFIG.14may not be used to determine 3D coordinates of a point lying on a plane that includes the optical axes of device1, device2, and device3since the epipolar lines are degenerate (fall on top of one another) in this case. In other words, in this case, intersection of epipolar lines is no longer obtained. Instead, in an embodiment, determining self-consistency of the positions of an uncoded spot on the projection plane of the projector and the image planes of the first and second cameras is used to determine correspondence among uncoded spots, as described herein above in reference toFIGS.12B,12C,12D,12E.

FIGS.15A,15B,15C,15D,15Eare schematic illustrations of alternative embodiments of the projector1120. InFIG.15A, a projector1500includes a light source, mirror1504, and diffractive optical element (DOE)1506. The light source1502may be a laser, a superluminescent diode, or a partially coherent LED, for example. The light source1502emits a beam of light1510that reflects off mirror1504and passes through the DOE. In an embodiment, the DOE11506produces an array of diverging and uniformly distributed light spots512. InFIG.15B, a projector1520includes the light source1502, mirror1504, and DOE1506as inFIG.15A. However, in the projector1520ofFIG.15B, the mirror1504is attached to an actuator1522that causes rotation1524or some other motion (such as translation) in the mirror. In response to the rotation1524, the reflected beam off the mirror1504is redirected or steered to a new position before reaching the DOE1506and producing the collection of light spots1512. In system1530ofFIG.15C, the actuator is applied to a mirror1532that redirects the beam1512into a beam1536. Other types of steering mechanisms such as those that employ mechanical, optical, or electro-optical mechanisms may alternatively be employed in the systems ofFIG.15A,15B,15C. In other embodiments, the light passes first through the pattern generating element1506and then through the mirror1504or is directed towards the object space without a mirror1504.

In the system1540ofFIG.5D, an electrical signal is provided by the electronics1544to drive a projector pattern generator1542, which may be a pixel display such as a Liquid Crystal on Silicon (LCoS) display to serve as a pattern generator unit, for example. The light1545from the LCoS display1542is directed through the perspective center1547from which it emerges as a diverging collection of uncoded spots1548. In system1550ofFIG.15E, a source is light1552may emit light that may be sent through or reflected off of a pattern generating unit1554. In an embodiment, the source of light1552sends light to a digital micromirror device (DMD), which reflects the light1555through a lens1556. In an embodiment, the light is directed through a perspective center1557from which it emerges as a diverging collection of uncoded spots1558in an uncoded pattern. In another embodiment, the source of light1562passes through a slide1554having an uncoded pattern of dots before passing through a lens1556and proceeding as an uncoded pattern of light1558. In another embodiment, the light from the light source1552passes through a lenslet array1554before being redirected into the pattern1558. In this case, inclusion of the lens1556is optional.

The actuators1522,1534, also referred to as beam steering mechanisms, may be any of several types such as a piezo actuator, a microelectromechanical system (MEMS) device, a magnetic coil, or a solid-state deflector.

FIG.16Ais an isometric view of a triangulation scanner1600that includes a single camera1602and two projectors1604,1606, these having windows1603,1605,1607, respectively. In the triangulation scanner1600, the projected uncoded spots by the projectors1604,1606are distinguished by the camera1602. This may be the result of a difference in a characteristic in the uncoded projected spots. For example, the spots projected by the projector1604may be a different color than the spots projected by the projector1606if the camera1602is a color camera. In another embodiment, the triangulation scanner1600and the object under test are stationary during a measurement, which enables images projected by the projectors1604,1606to be collected sequentially by the camera1602. The methods of determining correspondence among uncoded spots and afterwards in determining 3D coordinates are the same as those described earlier inFIG.12for the case of two cameras and one projector. In an embodiment, the triangulation scanner1600includes a processor1102that carries out computational tasks such as determining correspondence among uncoded spots in projected and image planes and in determining 3D coordinates of the projected spots.

FIG.16Bis an isometric view of a triangulation scanner1620that includes a projector1622and in addition includes three cameras: a first camera1624, a second camera1626, and a third camera1628. These aforementioned projector and cameras are covered by windows1623,1625,1627,1629, respectively. In the case of a triangulation scanner having three cameras and one projector, it is possible to determine the 3D coordinates of projected spots of uncoded light without knowing in advance the pattern of dots emitted from the projector. In this case, lines can be drawn from an uncoded spot on an object through the perspective center of each of the three cameras. The drawn lines may each intersect with an uncoded spot on each of the three cameras. Triangulation calculations can then be performed to determine the 3D coordinates of points on the object surface. In an embodiment, the triangulation scanner1620includes the processor1102that carries out operational methods such as verifying correspondence among uncoded spots in three image planes and in determining 3D coordinates of projected spots on the object.

FIG.16Cis an isometric view of a triangulation scanner1640like that ofFIG.1Aexcept that it further includes a camera1642, which is coupled to the triangulation scanner1640. In an embodiment the camera1642is a color camera that provides colorization to the captured 3D image. In a further embodiment, the camera1642assists in registration when the camera1642is moved—for example, when moved by an operator or by a robot.

FIGS.17A,17Billustrate two different embodiments for using the triangulation scanner1in an automated environment.FIG.17Aillustrates an embodiment in which a scanner1is fixed in position and an object under test1702is moved, such as on a conveyor belt1700or other transport device. The scanner1obtains 3D coordinates for the object1702. In an embodiment, a processor, either internal or external to the scanner1, further determines whether the object1702meets its dimensional specifications. In some embodiments, the scanner1is fixed in place, such as in a factory or factory cell for example, and used to monitor activities. In one embodiment, the processor1102monitors whether there is risk of contact with humans from moving equipment in a factory environment and, in response, issue warnings, alarms, or cause equipment to stop moving.

FIG.17Billustrates an embodiment in which a triangulation scanner1is attached to a robot end effector1710, which may include a mounting plate1712and robot arm1714. The robot may be moved to measure dimensional characteristics of one or more objects under test. In further embodiments, the robot end effector is replaced by another type of moving structure. For example, the triangulation scanner1101may be mounted on a moving portion of a machine tool.

FIG.18is a schematic isometric drawing of a measurement application1800that may be suited to the triangulation scanners described herein above. In an embodiment, a triangulation scanner1101sends uncoded spots of light onto a sheet of translucent or nearly transparent material1810such as glass. The uncoded spots of light1802on the glass front surface1812arrive at an angle to a normal vector of the glass front surface1812. Part of the optical power in the uncoded spots of light1802pass through the front surface1812, are reflected off the back surface1814of the glass, and arrive a second time at the front surface1812to produce reflected spots of light1804, represented inFIG.18as dashed circles. Because the uncoded spots of light1802arrive at an angle with respect to a normal of the front surface1812, the spots of light1804are shifted laterally with respect to the spots of light1802. If the reflectance of the glass surfaces is relatively high, multiple reflections between the front and back glass surfaces may be picked up by the triangulation scanner1.

The uncoded spots of lights1802at the front surface1812satisfy the criterion described with respect toFIG.12in being intersected by lines drawn through perspective centers of the projector and two cameras of the scanner. For example, consider the case in which inFIG.12the element1250is a projector, the elements1210,1230are cameras, and the object surface1270represents the glass front surface1270. InFIG.12, the projector1250sends light from a point1253through the perspective center1258onto the object1270at the position1272. Let the point1253represent the center of a spot of light1802inFIG.18. The object point1272passes through the perspective center1218of the first camera onto the first image point1220. It also passes through the perspective center1238of the second camera1230onto the second image point1235. The image points1200,1235represent points at the center of the uncoded spots1802. By this method, the correspondence in the projector and two cameras is confirmed for an uncoded spot1802on the glass front surface1812. However, for the spots of light1804on the front surface that first reflect off the back surface, there is no projector spot that corresponds to the imaged spots. In other words, in the representation ofFIG.12, there is no condition in which the lines1211,1231,1251intersect in a single point1272for the reflected spot1204. Hence, using this method, the spots at the front surface may be distinguished from the spots at the back surface, which is to say that the 3D coordinates of the front surface are determined without contamination by reflections from the back surface. This is possible as long as the thickness of the glass is large enough and the glass is tilted enough relative to normal incidence. Separation of points reflected off front and back glass surfaces is further enhanced by a relatively wide spacing of uncoded spots in the projected uncoded pattern as illustrated inFIG.18. Although the method ofFIG.18was described with respect to the scanner1, the method would work equally well for other scanner embodiments such as the scanners1600,1620,1640ofFIGS.16A,16B,16C, respectively.

Terms such as processor, controller, computer, DSP, FPGA are understood in this document to mean a computing device that may be located within an instrument, distributed in multiple elements throughout an instrument, or placed external to an instrument.

While embodiments of the invention have been described in detail in connection with only a limited number of embodiments, it should be readily understood that the invention is not limited to such disclosed embodiments. Rather, the embodiments of the invention can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the invention. Additionally, while various embodiments of the invention have been described, it is to be understood that aspects of the invention may include only some of the described embodiments. Accordingly, the embodiments of the invention are not to be seen as limited by the foregoing description but is only limited by the scope of the appended claims.