THREE-DIMENSIONAL MEASUREMENT DEVICE WITH COLOR CAMERA

A three-dimensional (3D) measurement system, method and computer program product are provided. The system includes a noncontact measurement device that measures a distance from the noncontact measurement device to a surface. The device includes a projector that emits a light pattern. A measurement camera is coupled to the noncontact measurement device. A first color camera is provided having a first quality parameter. A second color camera is coupled to the device, the second color camera having a second quality parameter, the second quality parameter being larger than the first quality parameter. One or more processors are operably coupled to the noncontact measurement device that determine 3D coordinates of at least one point in a field of view and selectively acquiring an image with the second color camera.

BACKGROUND

The subject matter disclosed herein relates to a three-dimensional (3D) measurement device, and in particular to a 3D measurement device operable to selectively acquire high quality color images.

A 3D imager is a portable device includes a projector that projects light patterns on the surface of an object to be scanned. Typically the projector emits a coded or uncoded pattern. One (or more) cameras, having a predetermined positions and alignment relative to the projector, which record images of the light pattern on the surface of an object. The three-dimensional coordinates of elements in the light pattern can be determined by trigonometric methods, such as by using epipolar geometry. Other types of noncontact devices may also be used to measure 3D coordinates, such as those that use time of flight techniques (e.g. laser trackers, laser scanners or time of flight cameras) for measuring the amount of time it takes for light to travel to the surface and return to the device.

These 3D imagers often have an additional color camera that is used for tracking the position of the 3D imager, the colorizing (texture) of the 3D point cloud or provide a visual feedback to the operator during scanning. It should be appreciated that to reduce processing and storage loading, the color camera acquires images at a relatively low resolution, such as 1.3 megapixels for example. The low resolution images allow the 3D imager to perform the intended function (e.g. tracking, colorizing or visualization feedback) without unnecessarily using large amounts of memory or slowing down the processor of the 3D imager. Further, typically, the color camera uses a global shutter camera sensor where the smallest achievable pixel size is larger than for a rolling shutter camera.

In some applications, it may be desirable to acquire high resolution or high quality images of the scene or portions of the scanned area. For example, in a crime scene investigation, the investigator may use a high resolution DSLR camera to take photographs of areas they believe needs to be documented further, such as evidence for example. As a result, the high quality photographs are acquired and stored separately from the 3D point cloud generated by the 3D imager.

Accordingly, while existing 3D imagers are suitable for their intended purpose the need for improvement remains, particularly in providing a system for acquiring high quality images during a scanning process.

BRIEF DESCRIPTION

According to one aspect of the disclosure, a three-dimensional (3D) measurement system is provided. The measurement system includes a noncontact measurement device operable to measure a distance from the noncontact measurement device to a surface. The noncontact measurement device includes a projector that emits a light pattern. A measurement camera is operably coupled to the noncontact measurement device. A first color camera is provided having a first quality parameter. A second color camera is operably coupled to the noncontact measurement device, the second color camera having a second quality parameter, the second quality parameter being larger than the first quality parameter. One or more processors are operably coupled to the noncontact measurement device, the one or more processors operable to execute computer instructions when executed on the processor for determining 3D coordinates of at least one point in a field of view and selectively acquiring an image with the second color camera.

According to another aspect of the disclosure, a method is provided. The method includes emitting a pattern of light with a projector of a noncontact measurement device. Point data is acquired about a plurality of points on a surface with a measurement camera of a noncontact measurement device. A first color image of the surface is acquired with a first color camera of the noncontact measurement device, the first color camera having a first quality parameter. 3D coordinates of the plurality of points are determined based at least in part on the point data and the baseline distance between the projector and the measurement camera. A second color image of the surface is selectively acquired with a second color camera in response to an input from an operator, the high quality camera being operably coupled to the noncontact measurement device and having a second quality parameter, the second quality parameter being different than the first quality parameter.

According to yet another aspect of the disclosure, a computer program product for determining three dimensional coordinates using a noncontact measurement device is provided. The computer program product comprises a nontransitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform: emitting a pattern of light with a projector of a noncontact measurement device; acquiring point data about a plurality of points on a surface with a measurement camera of a noncontact measurement device, acquiring a first color image of the surface with a first color camera of the noncontact measurement device, the first color camera having a first quality parameter; determining 3D coordinates of the plurality of points based at least in part on the point data and the baseline distance between the projector and the measurement camera; selectively acquiring a second color image of the surface with a second color camera in response to an input from an operator, the high quality camera being operably coupled to the noncontact measurement device and having a second quality parameter, the second quality parameter being different than the first quality parameter.

According to yet another aspect of the disclosure, a three-dimensional (3D) measurement system is provided. The measurement system includes a noncontact measurement device operable to measure a distance from the noncontact measurement device to a surface. The noncontact measurement device includes a projector that emits a light pattern. A measurement camera is operably coupled to the noncontact measurement device. A color camera is provided. One or more processors are operably coupled to the noncontact measurement device, the one or more processors operable to execute computer instructions when executed on the processor for: selectively acquiring an image with the color camera; determining 3D coordinates of at least one point in a field of view and determining a position registration through optical tracking based at least in part on the image, wherein the at least one point includes a plurality of points, the plurality of points includes a first plurality of points and a second plurality of points; determining an area of interest from the image and registers the first plurality of points with the area of interest; and deleting the second plurality of points.

According to yet another aspect of the disclosure, a three-dimensional (3D) measurement system is provided. The measurement system includes a noncontact measurement device operable to measure a distance from the noncontact measurement device to a surface. The noncontact measurement device includes a projector that emits a light pattern; a measurement camera operably coupled to the noncontact measurement device; a color camera operable to selectively operate at a first quality parameter and a second quality parameter; a user-interface; and one or more processors operably coupled to the noncontact measurement device, the one or more processors operable to execute computer instructions when executed on the processor for selectively acquiring a first image with the color camera operating at the first quality parameter; selectively acquiring a second image with the color camera operating at the second quality parameter; determining 3D coordinates of at least one point in a field of view and determining a position registration through optical tracking based at least in part on the image.

DETAILED DESCRIPTION

Embodiments of the invention provide for a three-dimensional (3D) measurement device that acquires high quality images during a scanning process. Embodiments disclosed herein further provide for integrating high quality images into a point cloud acquired by the 3D imager. Further embodiments disclosed herein provide for selectively coloring (texture) of the point cloud using high quality images. Still further embodiments disclosed herein provide for the automatic removal of points in the point cloud that are outside of an area of interest based on high quality images acquired by the 3D imager. Still further embodiments provide feedback to a user on the density of the point cloud based on high quality images acquired by the 3D imager.

Referring now toFIG. 1, a measurement device, such as 3D imager system20, is shown for determining 3D coordinates of surfaces in an environment. The system20includes an image scanner22having a projector24, a first camera26and a second camera28. In the exemplary embodiment, the projector24, and cameras26,28are each disposed in separate arms30,32,34of a housing36respectively. A color camera40may be centrally disposed on the housing36between the arms30,32,34. In the exemplary embodiment, the color camera40has a field of view that acquires images, including color information, of the environment being scanned. In an embodiment, the color camera40may be used to provide color (texture) information for incorporation into the 3D image. In some embodiments, the camera40acquires a video image may be used to register multiple 3D images through the use of videogrammetry. The color camera40is sometimes referred to as an RGB camera.

In this embodiment, the color camera40is configured to be selectively changed between a first quality parameter that acquires a low quality image, such as an image having a resolution equal to or less than about 1.3-5 megapixels, and a second quality parameter that acquires a high quality image, such as an image having a resolution equal to or greater than about 10 megapixels. It should be appreciated that while embodiments herein describe the quality parameter based on image or sensor resolution, this is for exemplary purposes and the claims should not be so limited. In other embodiments the quality parameter may be based on other attributes of the color camera40.

As discussed in more detail herein, in an embodiment the projector24projects a pattern of light onto a surface in the environment. As used herein, the term “projector” is defined to generally refer to a device for producing a pattern. The generation of the pattern can take place by means of deflecting methods, such as generation by means of diffractive optical elements or micro-lenses (or single lasers), or by shading methods, for example the production by means of shutters, transparencies (as they would be used in a transparency projector) and other masks. The deflecting methods have the advantage of less light getting lost and consequently a higher intensity being available.

The cameras26,28acquire images of the pattern and in some instances able to determine the 3D coordinates of points on the surface using trigonometric principles, e.g. epipolar geometry. In an embodiment, the cameras26,28are sensitive to monochromatic light, such as light in the infrared (IR) spectrum.

It should be appreciated that while the illustrated embodiments show and describe the device that determines 3D coordinates as being an image scanner, this is for exemplary purposes and the claimed invention should not be so limited. In other embodiments, devices that use other noncontact means for measuring 3D coordinates may also be used, such as a laser scanner device that uses time-of-flight to determine the distance to the surface.

A controller48is coupled for communication to the projector24, cameras26,28,40and in an embodiment the high quality camera47. The connection may be a wired connection50or a wireless connection. The controller48is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. Controller48may accept instructions through user interface52, or through other means such as but not limited to electronic data card, voice activation means, manually-operable selection and control means, radiated wavelength and electronic or electrical transfer.

Controller48uses signals act as input to various processes for controlling the system20. The digital signals represent one or more system20data including but not limited to images acquired by cameras26,28,40, temperature, ambient light levels, operator inputs via user interface52and the like.

Controller48is operably coupled with one or more components of system20by data transmission media50. Data transmission media50includes, but is not limited to, twisted pair wiring, coaxial cable, and fiber optic cable. Data transmission media50also includes, but is not limited to, wireless, radio and infrared signal transmission systems. Controller48is configured to provide operating signals to these components and to receive data from these components via data transmission media50.

In general, controller48accepts data from cameras26,28,40, projector24and a light source, and is given certain instructions for the purpose of determining the 3D coordinates of points on surfaces being scanned. The controller48may compare the operational parameters to predetermined variances and if the predetermined variance is exceeded, generates a signal that may be used to indicate an alarm to an operator or to a remote computer via a network. Additionally, the signal may initiate other control methods that adapt the operation of the system20such as changing the operational state of cameras26,28,40, projector24or light source42to compensate for the out of variance operating parameter. Still other control methods may display, highlight in the display or otherwise notify the operator when a low point density is detected.

The data received from cameras26,28,40may be displayed on a user interface52. The user interface52may be an LED (light-emitting diode) display, an LCD (liquid-crystal diode) display, a CRT (cathode ray tube) display, a touch-screen display or the like. A keypad may also be coupled to the user interface for providing data input to controller38. In an embodiment, the controller48displays in the user interface52a point cloud to visually represent the acquired 3D coordinates.

In addition to being coupled to one or more components within system20, controller48may also be coupled to external computer networks such as a local area network (LAN) and the Internet. A LAN interconnects one or more remote computers, which are configured to communicate with controller48using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet(̂) Protocol), RS-232, ModBus, and the like. Additional systems20may also be connected to LAN with the controllers48in each of these systems20being configured to send and receive data to and from remote computers and other systems20. The LAN is connected to the Internet. This connection allows controller48to communicate with one or more remote computers connected to the Internet.

Controller48includes a processor54coupled to a random access memory (RAM) device56, a non-volatile memory (NVM) device58, a read-only memory (ROM) device60, one or more input/output (I/O) controllers, and a LAN interface device62via a data communications bus.

LAN interface device62provides for communication between controller48and a network in a data communications protocol supported by the network. ROM device60stores an application code, e.g., main functionality firmware, including initializing parameters, and boot code, for processor54. Application code also includes program instructions as shown inFIGS. 9-11andFIG. 13for causing processor54to execute any system20operation control methods, including starting and stopping operation, changing operational states of projector24and color camera40, monitoring predetermined operating parameters, and generation of alarms. In an embodiment, the application code creates an onboard telemetry system may be used to transmit operating information between the system20and one or more remote computers or receiving locations. The information to be exchanged remote computers and the controller48include but are not limited to 3D coordinate data and images.

NVM device58is any form of non-volatile memory such as an EPROM (Erasable Programmable Read Only Memory) chip, a disk drive, or the like. Stored in NVM device58are various operational parameters for the application code. The various operational parameters can be input to NVM device58either locally, using a user interface52or remote computer, or remotely via the Internet using a remote computer. It will be recognized that application code can be stored in NVM device58rather than ROM device60.

Controller48includes operation control methods embodied in application code such as that shown inFIGS. 9-11andFIG. 13. These methods are embodied in computer instructions written to be executed by processor54, typically in the form of software. The software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, Visual C++, C#, Objective-C, Java, Javascript ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (HyperText Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing. Additionally, an operator can use an existing software application such as a spreadsheet or database and correlate various cells with the variables enumerated in the algorithms. Furthermore, the software can be independent of other software or dependent upon other software, such as in the form of integrated software.

In an embodiment, the controller48further includes an energy source, such as battery64. The battery64may be an electrochemical device that provides electrical power for the controller48. In an embodiment, the battery64may also provide electrical power to the cameras26,28,40, the projector24and the high quality camera47. In some embodiments, the battery64may be separate from the controller (e.g. a battery pack). In an embodiment, a second battery (not shown) may be disposed in the housing36to provide electrical power to the cameras26,28,40and projector24. In still further embodiments, the light source42may have a separate energy source (e.g. a battery pack).

It should be appreciated that while the controller48is illustrated as being separate from the housing36, this is for exemplary purposes and the claimed invention should not be so limited. In other embodiments, the controller48is integrated into the housing36.

Referring now toFIG. 2, another embodiment of the image scanner22is shown having a second color camera47. In this embodiment, the color camera40acquires color images at a first quality parameter/level and the second color camera47acquires color images at a second quality parameter/level. The quality parameter of the cameras40,47may be defined in terms of resolution (megapixels). In this embodiment, the second color camera47is integrated into the housing36. In some embodiments, the position of the second color camera47is in a predetermined geometrical relationship to the first color camera40, the projector24or the cameras26,28.

Referring now toFIG. 3, yet another embodiment of the image scanner22is shown having a second color camera47. In this embodiment, the color camera40acquires color images at a first quality parameter/level and the second color camera47acquires color images at a second quality parameter/level. The quality parameter of the cameras40,47may be defined in terms of resolution (megapixels). In this embodiment, the second color camera47is an external camera that is rigidly coupled to the housing36. The second color camera47may be coupled to the housing36by a mechanical structure, such as a bracket for example. In an embodiment the second color camera47is removably coupled to the housing.

As will be discussed in more detail herein, the second color camera47, or the color camera40in high quality mode of operation, may be used in combination with the measured 3D coordinate data to improve or enhance the point cloud data. As used herein, a point cloud is a collection or a set of data points in a coordinate system. In a three-dimensional coordinate system, these points are usually defined by X, Y, and Z coordinates, and represent the external surface of an object that is scanned with the system20.

Referring now toFIG. 4andFIG. 5, an embodiment is shown of a method of determining the 3D coordinates of the data points in the point cloud using the image scanner22. In the illustrated embodiment, the projector24and cameras26,28are arranged spaced apart in a triangular arrangement where the relative distances and positions between the components is known. The triangular arrangement is advantageous in providing information beyond that available for two cameras and a projector arranged in a straight line or from a system with a projector and a single camera. The additional information may be understood in reference toFIG. 4, which explain the concept of epipolar constraints, andFIG. 5that explains how epipolar constraints are advantageously applied to the triangular arrangement of the system20. InFIG. 4, a 3D triangulation instrument84includes a device1and a device2on the left and right sides as view from the viewpoint ofFIG. 4, respectively. Device1and device2may be two cameras or device1and device2may be one camera and one projector. Each of the two devices, whether a camera or a projector, has a perspective center, O1and O2, and a representative plane,86or88. The perspective centers are separated by a baseline distance B, which is the length of the line90. The perspective centers O1, O2are points through which rays of light may be considered to travel, either to or from a point on a surface in the area of the environment being scanned. These rays of light either emerge from an illuminated projector pattern or impinge on a photosensitive array. The placement of the reference planes86,88is applied inFIG. 4, which shows the reference planes86,88between the object point and the perspective centers O1, O2.

InFIG. 4, for the reference plane86angled toward the perspective center O2and the reference plane88angled toward the perspective center O1, a line90drawn between the perspective centers O1and O2crosses the planes86and88at the epipole points E1, E2, respectively. Consider a point UD on the plane86. If device1is a camera, it is known that an object point that produces the point UD on the image lies on the line92. The object point might be, for example, one of the points VA, VB, VC, or VD. These four object points correspond to the points WA, WB, WC, WD, respectively, on the reference plane88of device2. This is true whether device2is a camera or a projector. It is also true that the four points lie on a straight line94in the plane88. This line, which is the line of intersection of the reference plane88with the plane of O1-O2-UD, is referred to as the epipolar line94. It follows that any epipolar line on the reference plane88passes through the epipole E2. Just as there is an epipolar line on the reference plane of device2for any point on the reference plane of device1, there is also an epipolar line96on the reference plane of device1for any point on the reference plane of device2.

FIG. 5illustrates the epipolar relationships for a 3D imager100corresponding to triangulation instrument84ofFIG. 4in which two cameras and one projector are arranged in a triangular pattern. In general, the device1, device2, and device3may be any combination of cameras and projectors as long as at least one of the devices is a camera. Each of the three devices102,104,106has a perspective center O1, O2, O3, respectively, and a reference plane108,110, and112, respectively. Each pair of devices has a pair of epipoles. Device1and device2have epipoles E12, E21on the planes108,110, respectively. Device1and device3have epipoles E13, E31, respectively on the planes108,112, respectively. Device2and device3have epipoles E23, E32on the planes110,112, respectively. In other words, each reference plane includes two epipoles. The reference plane for device1includes epipoles E12and E13. The reference plane for device2includes epipoles E21and E23. The reference plane for device3includes epipoles E31and E32.

Consider the embodiment ofFIG. 5in which device3is a projector, device1is a first camera, and device2is a second camera. Suppose that a projection point P3, a first image point P1, and a second image point P2are obtained in a measurement. These results can be checked for consistency in the following way.

To check the consistency of the image point P1, intersect the plane P3-E31-E13with the reference plane108to obtain the epipolar line114. Intersect the plane P2-E21-E12to obtain the epipolar line116. If the image point P1has been determined consistently, the observed image point P1will lie on the intersection of the determined epipolar line114and line116.

To check the consistency of the image point P2, intersect the plane P3-E32-E23with the reference plane110to obtain the epipolar line105. Intersect the plane P1-E12-E21to obtain the epipolar line107. If the image point P2has been determined consistently, the observed image point P2will lie on the intersection of the determined epipolar lines107and105.

To check the consistency of the projection point P3, intersect the plane P2-E23-E32with the reference plane110to obtain the epipolar line118. Intersect the plane P1-E13-E31to obtain the epipolar line120. If the projection point P3has been determined consistently, the projection point P3will lie on the intersection of the determined epipolar line118and line120.

The redundancy of information provided by using a 3D imager100having a triangular arrangement of projector and cameras may be used to reduce measurement time, to identify errors, and to automatically update compensation/calibration parameters. It should be appreciated that based on the epipolar geometry relationships described herein, the distance from the image scanner22to points on the surface being scanned may be determined. By moving the image scanner22, the determination of the pose/orientation of the image scanner, and a registration process the three dimensional coordinates of locations (point data) on a surface may be determined and the point cloud generated.

It should be appreciated that since the cameras26,28are sensitive to monochromatic light (e.g. in the infrared spectrum), the measured 3D coordinate points do not include color or texture information. In an embodiment, as the 3D coordinates are measured and determined, the image scanner22also acquires color images of the scene being scanned with the color camera40. In one embodiment, the color data from the images acquired by color camera40is merged with the measured 3D coordinate data. In order to obtain a 3D point cloud of the scanned object, each image shot/frame is registered, in other words the three-dimensional coordinates obtained in each image frame is inserted in a common coordinate system. Registration is possible, for example, by videogrammetry, i.e., for example, “structure from motion” (SFM) or “simultaneous localisation and mapping” (SLAM). The natural texture of the scanned objects or environment can also be used for common points of reference, or a separate stationary pattern can be produced. The natural texture can be captured by the color camera113in addition to obtaining the color information. This allows the color information to be associated with each of the measured 3D coordinates. In one embodiment, the pose of the first color camera40and the second color camera47is known in relation to the pose of the projector24and the cameras26,28and since the characteristics/parameters of the color camera are known (e.g. focal length), the two-dimensional (2D) color data may be merged with the measured 3D coordinate data/points by projecting the rays that enter the 2D color cameras40,47onto the measured 3D coordinate data/points.

It should be appreciated that since the color camera40acquires images at a rapid rate during the scanning operation, the quality level of the color image (e.g. resolution) may be lower than that desired for the purpose of documenting the scene. For example, in an embodiment the system20is used to scan a crime scene to document the position of different objects or evidence in the area where the crime was committed. In addition to the positions, sizes and coordinates of the objects, the investigator is also likely to desire high quality (e.g. high resolution) images to document small details for later analysis. It should be appreciated that the color camera is operated at a lower quality level in order to reduce the amount of electronic memory, storage used or processing resources (e.g. CPU) for the color images. Further, in some embodiments, the color camera40has a global-shutter type of sensor (i.e. all pixels exposed at the same time) and the color camera47has a rolling-shutter type of sensor (i.e. time shift between the exposures of different pixel rows). Compared to a rolling shutter type of sensor, sensors with a global shutter are more limited regarding the minimum achievable pixel size. However, the global-shutter sensor is desired for improved position registration through optical tracking. Thus in some embodiments the use of a second color camera with a rolling-shutter type sensor provides advantages in allowing higher resolution.

Referring now toFIG. 6andFIG. 7an embodiment is shown of system20using an imager22having a color camera, such as color camera40,47for example, that may be configured to acquire high quality images, such as images having a resolution equal to or greater than about 10 megapixels. In an embodiment, the system20is configured to selectively acquire a high quality (e.g. resolution) image using the color camera47or the color camera40in a high quality mode. For example, during the scanning of an area, the operator desires to document an area with a high resolution image, such as the object130. The operator then changes the position and pose of the image scanner22to place the object within the field of view132of the color camera40,47. The operator then initiates the high quality color camera40,47and acquires an image of the area within the field of view132. In one embodiment, the activation of the high quality camera40,47may be through an actuator on the image scanner22or via the user interface52.

When the image is acquired, the position and pose information of the image scanner22is stored and associated with the image. This allows the image134(FIG. 7) to be displayed in the user interface52with the image134arranged in the proper orientation and position based on the position and pose of the image scanner22during acquisition of the image. In some embodiments, the system20will display an icon or other symbol at the location in the point cloud to indicate that a high quality image of that location is available and may be viewed by selecting the icon or symbol with the user interface52. It should be appreciated that while embodiments herein refer to viewing of the point cloud or the high quality images using the user interface52of controller48, this is for exemplary purposes and the claims should not be so limited. In other embodiments the point cloud data, including the high quality images, may be transmitted to one or more computers, remote computers, computer servers, or computer nodes. In an embodiment the point cloud data, including any high quality images, may be viewed by any computing device that receives the point cloud data.

Referring now toFIG. 8, another embodiment is shown of the system20where an area or object of interest may be identified by the user and measured 3D coordinate points outside of the area or object of interest may be eliminated from the point cloud. Is should be appreciated that the removal of points that are outside the area or object of interest will improve the visualization for the user, reduce the amount of electronic memory or storage used for the point cloud, and also improve the performance of the controller48. In this embodiment, the image scanner22is moved from a first position to a second position while scanning the object130. In one embodiment, the operator activates the high quality color camera47at the first position to acquire a first image136of the area or object of interest (e.g. object130), and then acquires one or more further high quality images of the area or object of interest, such as image138for example, during the scan.

From these high quality images, the area or object of interest, such as object130for example, may be identified. In one embodiment, the object130is manually identified by the user on the user interface, such as by tracing the outline of the object130on the user interface52. In other embodiments, the object130may be automatically identified by the controller48using edge matching processes (e.g. the Canny edge detection method), greyscale matching methods, gradient matching methods, blob detection, or object template matching for example. In other embodiments, the object130may be identified using feature-based methods (e.g. surface patches, corners, linear edges). In still other embodiments, the object130may be compared to electronic object models, such as computer-aided-design (CAD) models for example. Once the object130is detected, the measured 3D coordinates of points that are not on the surfaces of object130, such as points on the surface140,142,144for example, are removed from the point cloud data.

In another embodiment, an area of interest is identified based on identifying common surfaces that are located in each of the high quality color camera147images (e.g. a Boolean combined operation), such as images136,138for example. In still another embodiment, an area of interest is identified from the accumulation of surfaces that are located within the high quality color camera147images (e.g. a Boolean union operation).

Referring now toFIG. 9, with continuing reference toFIG. 6andFIG. 7, a method150is shown for acquiring high quality color images and merging the images with the measured 3D coordinate data in a point cloud. The method150starts in block152by initiating a scan. The method then proceeds to query block154where it is determined whether there has been a user input indicating that a high quality image was desired, such as by activating a button on the image scanner22or via the user interface52for example. When query block154returns a positive, the method150proceeds to block156where the high quality image is acquired by the camera40,47. The method150then proceeds to store the high quality image along with the position and pose of the image scanner22in block158.

After storing the position and pose, or when the query block154returns a negative, the method150proceeds to block160where the measured 3D coordinate points are determined based at least in part on the light pattern projected by the projector24and from the images of the light pattern on surfaces acquired by cameras26,28. The method150then proceeds to block162where the high quality image acquired in block156is registered to the measured 3D coordinate points based on position and pose of the image scanner22determined in block158. The method150then proceeds to query block164where it is determined whether the user wishes to continue scanning. When the query block164returns a positive, the method150loops back to block152. When the query block164returns a negative, the method150proceeds to stop block166.

It should be appreciated that whileFIG. 9illustrates the steps of method150sequentially, this was for exemplary purposes and the claims should not be so limited. In other embodiments, the steps may be performed in another order, or some steps may be performed in parallel. For example, the determination of the 3D coordinates in block160may be performed in parallel with the acquisition of the high quality image in block156.

Referring now toFIG. 10, another method170is shown for using high quality color images to selectively colorize measured 3D coordinate points in a point cloud. The method170is similar to the steps described inFIG. 9with reference to method160. The steps of blocks152-162that are the same between method160and method170will not be repeated here for brevity. In this embodiment, the acquisition of the high quality image in block156may be performed when the image scanner22is directed at an area of interest and a high quality color rendition is desired.

After the registration of the high quality image in block162, the method170proceeds to block172where the colors from the high quality image are registered to the measured 3D coordinates. As discussed herein, the merging of the color data with the measured 3D coordinate points may be performed by projecting the rays that enter the 2D color camera40,47onto the measured 3D coordinate data/points.

Referring now toFIG. 11, with continuing reference toFIG. 8, another method180is shown for using high quality color images to define an area or object or interest and remove undesired 3D coordinate points from the point cloud. The method170is similar to the steps described inFIG. 9with reference to method160. The steps of blocks152-162that are the same between method160and method180will not be repeated here for brevity. In this embodiment, the acquisition of the high quality image in block156may be performed in multiple locations when the user has the image scanner pointing at the area or object of interest.

After the user has completed their scan and acquired at least one, and in some embodiments a plurality, of high quality images of the area or object of interest, the query block164returns a negative and the method180proceeds to block182where the area or object or interest (e.g. object130ofFIG. 8) is identified from the high quality images. In an embodiment the area or object of interest may be identified manually by the user, such as by tracing an outline of the area or object on the user interface52for example. In other embodiments the area or object of interest may be identified automatically as described herein (e.g. Canny edge detection). Once the area or object has been identified, the method180proceeds to block184where the measured 3D coordinates that are outside of the area or interest, or not on a surface of an object of interest, are identified. The identified external points are then removed or deleted from the point cloud.

Referring now toFIG. 12andFIG. 13, with continuing reference toFIG. 8, an embodiment is shown wherein high quality images may be used as a feedback to the user during a scanning operation of the point density of the measured 3D coordinates. The method190starts with acquiring high quality images of surfaces of interest to the operator in block192. In the illustrated embodiment, the operator acquires images of surfaces186,188(FIG. 8) before the scanning is initiated. The operator then proceeds to scan the area with the system20in block194and determine the measured 3D coordinate points in block196.

With the measured 3D coordinate points determined the method190proceeds to query block198where it is determined if the scan includes the surfaces of interest, such as surfaces186,188for example. When query block198returns a positive, the method190proceeds to block200where the point density of the measured 3D coordinates on the surface of interest is determined. This point density is then compared to a threshold in query block202. In an embodiment, the threshold represents a level, such as a minimum level for example, of density of the measured 3D coordinate points. In one embodiment, the threshold may be user-defined. It should be appreciated that in some embodiments, the point density may be compared to several thresholds (e.g. low, medium, high point density thresholds). When the query block202returns a positive, meaning that the point density is below a threshold, the method190then proceeds to block204where the user interface52is changed to provide feedback to the operator on the point density level(s) of the surfaces of interest.

Referring toFIG. 12, an example is shown wherein the operator acquired high quality images of surfaces186,188(FIG. 8) prior to the scan. When the scanning was completed, it was determined that the surface188had a point density above the desired threshold. As such, on the user interface the surface188may include a first indicator, such as a first color (as indicated by the dotted cross-hatching206inFIG. 12) for example. In one embodiment, when the surface of interest has a point density greater than a threshold, the surface is colored green so that the operator is provided feedback that no additional scanning of this surface is needed to obtain the desired results. In the embodiment ofFIG. 12, at least a portion of the surface186has a point density below a threshold. When the point density is below a threshold, the user interface52may be changed to display the surface186with a second indicator, such as a second color (as indicated by the first cross-hatched area208). In an embodiment, when the surface of interest has a point density below a threshold, the surface is colored red so that the operator is provided feedback that additional scanning may be performed to obtain the desired point density.

It should be appreciated that in some embodiments, there may be multiple thresholds. In the embodiment illustrated inFIG. 12, areas of surface186may include areas of low density208that are below a first threshold, an area of medium density210that are between the first threshold and a second threshold, and an area of high density204that is above the second threshold. Each of the areas204,208,210may be indicated on the user interface52using a different color.

Once the feedback indicator of point density is displayed on user interface52, or when the query block202returns a negative, the method190proceeds to query block212. In query block212it is determined whether the operator desires to continue with additional scanning, such as to scan additional areas or to increase the point density of a surface of interest for example. When query block212returns a positive, the method190loops back to block194and the process continues. When query block190returns a negative, the method190proceeds to stop block214.

Technical effects and benefits of some embodiments include the noncontact measurement of three-dimensional coordinates of points on a surface and the acquisition of color images having a high quality parameter/level. The high quality image may be integrated into the point cloud to allow the user to view the measurement data with an additional (e.g. higher) level of detail. The high quality image may be used to improve the merging of color information into portions of the point cloud. The high quality image may further be used to identify the object or area of interest to allow points outside of the object/area of interest to be deleted from the point cloud. The deletion of points from the point cloud may improve the performance of the controller and improve the visualization of the point cloud for the user. The high quality image may still further be used to provide feedback to the operator during the scanning operation to determine whether a desired level of point density has been achieved.