Correction of current scan data using pre-existing data

A system and method for measuring coordinate values of an environment is provided. The system includes a coordinate measurement scanner that includes a light source that steers a beam of light to illuminate object points in the environment, and an image sensor arranged to receive light reflected from the object points to determine coordinates of the object points in the environment. The system also includes one or more processors for performing a method that includes receiving a previously generated map of the environment and causing the scanner to measure a plurality of coordinate values as the scanner is moved through the environment, the coordinate values forming a point cloud. The plurality of coordinate values are registered with the previously generated map into a single frame of reference. A current map of the environment is generated based at least in part on the previously generated map and the point cloud.

BACKGROUND

The present application is directed to optically scanning an environment, such as a building, and in particular to using pre-existing data to correct current scan data generated by a mobile scanning platform.

The automated three-dimensional (3D) scanning of an environment is desirable as a number of scans may be performed in order to obtain a complete scan of the area. 3D coordinate scanners include time-of-flight (TOF) coordinate measurement devices. A TOF laser scanner is a scanner in which the distance to a target point is determined based on the speed of light in air between the scanner and a target point. A laser scanner optically scans and measures objects in a volume around the scanner through the acquisition of data points representing object surfaces within the volume. Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value. This raw scan data is collected, stored and sent to a processor or processors to generate a 3D image representing the scanned area or object.

It should be appreciated that where an object (e.g. a wall, a column, or a desk) blocks the beam of light, that object will be measured but any objects or surfaces on the opposite side will not be scanned since they are in the shadow of the object relative to the scanner. Therefore, to obtain a more complete scan of the environment, the TOF scanner is moved to different locations and separate scans are performed. Subsequent to the performing of the scans, the 3D coordinate data (i.e. the point cloud) from each of the individual scans are registered to each other and combined to form a 3D image or model of the environment.

Some existing measurement systems have been mounted to a movable structure, such as a cart, and are moved on a continuous basis through a building, or other environment, to generate a digital representation of the building. However, these types of systems generally provide lower data quality than stationary scans. For example, mobile scanning devices, including those mounted on a movable structure and hand-held devices, can become inaccurate over distances due to error accumulation referred to as drift. When drift occurs, the model of the environment may not reflect the actual environment. Walls in hallways may appear as having a bend, extend on an angle, and/or edges of two walls forming a corner may not match up exactly.

Accordingly, while existing scanners are suitable for their intended purposes, what is needed is a system for having certain features of embodiments of the present invention.

BRIEF DESCRIPTION

According to one aspect of the invention, a system for measuring coordinate values of an environment is provided. The system includes a coordinate measurement scanner that includes a light source, an image sensor, and a controller. The light source steers a beam of light to illuminate object points in the environment and the image sensor is arranged to receive light reflected from the object points to determine coordinates of the object points in the environment. The system also includes one or more processors operably coupled to the scanner, the one or more processors being responsive to executable instructions for performing a method. The method includes receiving a previously generated map of the environment, the previously generated map including a plurality of features. The method also includes causing the scanner to measure a plurality of coordinate values as the scanner is moved through the environment, the coordinate values forming a point cloud. The method also includes registering the plurality of coordinate values and at least a subset of the features of the previously generated map into a single frame of reference. The method further includes generating a current map of the environment based at least in part on the previously generated map and the point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include the generating including updating the previously generated map based on detecting differences between the previously generated map and the point cloud and outputting the updated previously generated map as the current map.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include the updating including overlaying portions of the previously generated map with corresponding portions of the point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the scanner is a two-dimensional (2D) scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the scanner is a three-dimensional (3D) scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the previously generated map includes an existing point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the previously generated map includes a computer aided design (CAD) model.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the previously generated map includes a floor plan.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the system is configured to be carried by an operator without stopping the measurements of the plurality of coordinates.

According to another aspect of the invention, a method for measuring coordinate values of an environment is provided. The method includes moving a scanner through an environment, the scanner having a light source, an image sensor and a controller. The light source steers a beam of light to illuminate object points in the environment, the image sensor is arranged to receive light reflected from the object points, and the controller is operable to determine coordinates of the object points in the environment. The method also includes receiving, at the scanner, a previously generated map of the environment, the previously generated map including a plurality of features. The method also includes causing the scanner to measure a plurality of coordinate values as it moves through the environment, the coordinate values forming a point cloud. The method further includes registering the point cloud with at least a subset of the features of the previously generated map into a single frame of reference. The method further includes generating a current map of the environment based at least in part on the previously generated map and the point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include the generating including updating the previously generated map based on detecting differences between the previously generated map and the point cloud and outputting the updated previously generated map as the current map.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include the updating including overlaying portions of the previously generated map with corresponding portions of the point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include that the scanner is a 2D scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include that the scanner is a 3D scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include that the previously generated map includes an existing point cloud.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include that the previously generated map includes a CAD model.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include that the previously generated map includes a floor plan.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include moving the scanner continuously through the environment.

According to another aspect of the invention, a method for measuring coordinate values of an environment is provided. The method includes moving a base unit through an environment. The base unit includes a 2D scanner and a 3D scanner. The 2D scanner has a light source, an image sensor and a controller. The light source steers a beam of light within a first plane to illuminate object points in the environment, the image sensor is arranged to receive light reflected from the object points, and the controller is operable to determine a distance value to at least one of the object points. The 2D scanner measures an angle and a distance value. The 3D scanner is configured to operate in a compound mode, and the 3D scanner has a color camera. The method also includes, as the base unit is moving, causing the 2D scanner to generate a 2D map of the environment, the 2D map being based at least in part on the angle, the distance value, and a previously generated map of the environment. The method further includes, as the base unit is moving, causing the 3D scanner to operate in compound mode to measure a plurality of 3D coordinate values. The method further includes registering the plurality of 3D coordinate values into a single frame of reference based at least in part on the 2D map.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the method may include moving the base unit continuously through the environment.

According to another aspect of the invention, a system for measuring coordinate values of an environment is provided. The method includes a movable base unit, a 2D scanner coupled to the base unit, a 3D scanner coupled to the base unit, and one or more processors operably coupled to the base unit, the 2D scanner, and the 3D scanner. The 2D scanner includes a light source, an image sensor and a controller, the light source steering a beam of light within a first plane to illuminate object points in the environment. The image sensor is arranged to receive light reflected from the object points, and the controller is operable to determine a distance value to at least one of the object points. The 2D scanner measures an angle and a distance value. The 3D scanner is operable to selectively measure 3D coordinates and grey values of surfaces in the environment. The 3D scanner is configured to operate in one of a compound mode or a helical mode, and the 3D scanner has a color camera. The one or more processors are responsive to executable instructions for performing a method. The method includes causing the 3D scanner to measure a first plurality of 3D coordinate values while operating in one of the compound mode or the helical mode as the base unit is moved from a first position to a second position. The method also includes causing the 3D scanner to measure a second plurality of 3D coordinate values while operating in compound mode when the base unit is stationary between the first position and second position. The method further includes registering the first plurality of 3D coordinate values and second plurality of 3D coordinate values into a single frame of reference. The method further includes generating a 2D map of the environment using the 2D scanner as the base unit is moved from a first position to a second position, the 2D map being based at least in part on the angle, the distance value, and a previously generated map of the environment.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the registration is based at least in part on the 2D scanner data.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the registration is based at least in part on the previously generated map of the environment.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the 3D scanner is a time-of-flight (TOF) coordinate measurement device configured to measure the 3D coordinate values in a volume about the 3D scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the 3D scanner is a structured light area scanner.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the 2D scanner and the 3D scanner are removably coupled to the base unit, and the 2D scanner and 3D scanner may be operated as an independent device separate from the base unit or each other.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the system is configured to be carried by an operator without stopping the measurement of the first plurality of 3D coordinates.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the method further includes performing a compound compensation and optimizing by automatically fusing sensor data acquired while operating the system, wherein the compound compensation includes positions and orientations of the sensors.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the method further includes causing the color camera to acquire color data and colorizing the 3D scan data.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the method further includes analyzing a tracking quality attribute of the first plurality of 3D coordinates and providing feedback to the operator.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include that the feedback includes instructing the operator to perform the stationary scan.

In addition to one or more of the features described herein, or as an alternative, further embodiments of the system may include a display operably coupled to the 2D scanner and 3D scanner, the display being configured to display the registered plurality of 3D coordinates or the 2D map in the single frame of reference.

DETAILED DESCRIPTION

Embodiments of the present invention relate to using pre-existing data to correct current scan data generated by a mobile scanning platform. In accordance with one or more embodiments of the present invention, registration of a continuously scanning mobile system is supported by recognition of known patterns during the data acquisition. This can allow for both improved positioning during the continuous recording of three-dimensional (3D) and/or two-dimensional (2D) data using a mobile scanning platform and for the simplified overlay of temporal data recordings.

The known patterns, or features, can include information from pre-existing data sources such as, but not limited to: computer aided design (CAD) drawings, or models, of floor plans; golden point clouds captured with a high quality system; natural features, or landmarks, captured by local stationary laser scans (e.g., corners or rooms in buildings); and known measurements (e.g., distances between walls, length of a hallway, ninety degree building structure, etc.). As used herein, the term “golden point cloud” refers to a highly accurate point cloud that is registered based on an accurate reference system and/or generated by a high-resolution stationary scanner. In an embodiment, the golden point cloud is created by a laser scanner and includes several scan positions (low occlusion), a high point density, and high registration quality. The values of each of these factors can be different based on characteristics of the object(s) of interest in the point cloud.

Overlaying the data captured by the mobile scanning platform onto a pre-existing data source, or previously generated map, such as a CAD model or a golden point cloud, can eliminate drift and result in a model that accurately reflects the current environment.

In addition, unlike contemporary techniques, embodiments of the present environment do not require the use of loop closure techniques and/or targets (e.g., spheres, checkers) inserted into the environment in order to register a current scan to specific locations in the environment. One or more embodiments of the present invention can register landmarks (e.g., walls) that are shown in a pre-existing data source such as a floor plan of the environment with corresponding coordinate values in the scan data. This allows the registration of the scan data to be performed accurately and with fewer data points than contemporary techniques.

Referring now toFIGS.1-4, an embodiment is shown of a mobile scanning platform100. The platform100includes a frame102having a tripod portion104thereon. The frame102further includes a plurality of wheels106that allow the platform100to be moved about an environment. The frame102further includes a handle portion107that provides a convenient place for the operator to push and maneuver the platform100.

The tripod portion104includes a center post109. In an embodiment, the center post109generally extends generally perpendicular to the surface that the platform100is on. Coupled to the top of the post109is a 3D measurement device110. In the exemplary embodiment, the 3D measurement device110is a time-of-flight type scanner (either phase-based or pulse-based) that emits and receives a light to measure a volume about the scanner. In the exemplary embodiment, the 3D measurement device110is implemented by the scanner610that is described in reference toFIGS.27-29herein.

Also attached to the center post109is a 2D scanner108. In an embodiment, the 2D scanner108is the same type of scanner as is described in reference toFIGS.9-26herein. In the exemplary embodiment, the 2D scanner emits light in a plane and measures a distance to an object, such as a wall for example. As described in more detail herein, these distance measurements may be used to generate a 2D map of an environment when the 2D scanner108is moved therethrough. The 2D scanner108is coupled to the center post by an arm112that includes an opening to engage at least the handle portion of the 2D scanner108.

In an embodiment, one or both of the 3D measurement device110and the 2D scanner108are removably coupled from the platform100. In an embodiment, the platform100is configured to operate (e.g. operate the scanners108,110) while the platform100is being carried by one or more operators.

In an embodiment, the mobile scanning platform100may include a controller (not shown) that is coupled to communicate with both the 2D scanner108and the 3D measurement device110.

Referring now toFIG.5, another embodiment is shown of a mobile scanning platform200. The scanning platform200is similar to the platform100in that it has a frame202with a tripod204mounted thereon. The frame includes a plurality of wheels206and a handle portion207.

In this embodiment, the center post209includes a holder212mounted between the post209and a 3D measurement device210. The holder212includes a pair of arms214that define an opening therebetween. Mounted within the opening a 2D scanner208. In an embodiment, the 2D scanner208is mounted coaxial with the post209and the axis of rotation of the 3D measurement device210.

Is should be appreciated that the platforms100,200are manually pushed by an operator through the environment. As will be discussed in more detail herein, as the platform100,200is moved through the environment, both the 2D scanner108,208and the 3D measurement device110,210are operated simultaneously, with the data of the 2D measurement device being used, at least in part, to register the data of the 3D measurement system.

If should further be appreciated that in some embodiments, it may be desired to the measurement platform to be motorized in a semi-autonomous or fully-autonomous configuration. Referring now toFIGS.6-8, an embodiment is shown of a mobile scanning platform300. The mobile scanning platform100includes a base unit302having a plurality of wheels304. The wheels304are rotated by motors305(FIG.8). In an embodiment, an adapter plate307is coupled to the base unit302to allow components and modules to be coupled to the base unit302. The mobile scanning platform300further includes a 2D scanner308and a 3D scanner310. In the illustrated embodiment, each scanner308,310is removably coupled to the adapter plate306. The 2D scanner308may be the scanner illustrated and described in reference toFIGS.9-26. As will be described in more detail herein, in some embodiments the 2D scanner308is removable from the adapter plate306and is used to generate a map of the environment, plan a path for the mobile scanning platform to follow, and define 3D scanning locations. In the illustrated embodiment, the 2D scanner308is slidably coupled to a bracket311that couples the 2D scanner308to the adapter plate307.

In an embodiment, the 3D scanner310is a time-of-flight (TOF) laser scanner such as that shown and described in reference toFIGS.27-29. The scanner310may be that described in commonly owned U.S. Pat. No. 8,705,012, which is incorporated by reference herein. In an embodiment, the 3D scanner310mounted on a pedestal or post309that elevates the 3D scanner310above (e.g. further from the floor than) the other components in the mobile scanning platform300so that the emission and receipt of the light beam is not interfered with. In the illustrated embodiment, the pedestal or post309is coupled to the adapter plate307by a u-shaped frame314.

In an embodiment, the mobile scanning platform300further includes a controller316. The controller316is a computing device having one or more processors and memory. The one or more processors are responsive to non-transitory executable computer instructions for performing operational methods, such as that shown and described with respect toFIGS.30and35for example. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one or more processors have access to memory for storing information.

Coupled for communication to the controller316is a communications circuit318and an input/output hub320. In the illustrated embodiment, the communications circuit318is configured to transmit and receive data via a wireless radio-frequency communications medium, such as WiFi or Bluetooth for example. In an embodiment, the 2D scanner308communicates with the controller316via the communications circuit318

In an embodiment, the mobile scanning platform300further includes a motor controller322that is operably coupled to the control the motors305(FIG.5). In an embodiment, the motor controller322is mounted to an external surface of the base unit302. In another embodiment, the motor controller322is arranged internally within the base unit302. The mobile scanning platform300further includes a power supply324that controls the flow of electrical power from a power source, such as batteries326for example. The batteries326may be disposed within the interior of the base unit302. In an embodiment, the base unit302includes a port (not shown) for coupling the power supply to an external power source for recharging the batteries326. In another embodiment, the batteries326are removable or replaceable.

Referring now toFIGS.9-26, an embodiment of a 2D scanner408is shown having a housing432that includes a body portion434and a removable handle436. It should be appreciated that while the embodiment ofFIGS.9-26illustrate the 2D scanner408with the handle436attached, the handle436may be removed before the 2D scanner408is coupled to the base unit302when used in the embodiment ofFIGS.6-8. In an embodiment, the handle436may include an actuator438that allows the operator to interact with the scanner408. In the exemplary embodiment, the body portion434includes a generally rectangular center portion435with a slot440formed in an end442. The slot440is at least partially defined by a pair walls444that are angled towards a second end448. As will be discussed in more detail herein, a portion of a 2D laser scanner450is arranged between the walls444. The walls444are angled to allow the 2D laser scanner450to operate by emitting a light over a large angular area without interference from the walls444. As will be discussed in more detail herein, the end442may further include a three-dimensional camera or RGBD camera.

Extending from the center portion435is a mobile device holder441. The mobile device holder441is configured to securely couple a mobile device443to the housing432. The holder441may include one or more fastening elements, such as a magnetic or mechanical latching element for example, that couples the mobile device443to the housing432. In an embodiment, the mobile device443is coupled to communicate with a controller468(FIG.13). The communication between the controller468and the mobile device443may be via any suitable communications medium, such as wired, wireless or optical communication mediums for example.

In the illustrated embodiment, the holder441is pivotally coupled to the housing432, such that it may be selectively rotated into a closed position within a recess446. In an embodiment, the recess446is sized and shaped to receive the holder441with the mobile device443disposed therein.

In the exemplary embodiment, the second end448includes a plurality of exhaust vent openings456. In an embodiment, shown inFIGS.14-17, the exhaust vent openings456are fluidly coupled to intake vent openings458arranged on a bottom surface462of center portion435. The intake vent openings458allow external air to enter a conduit464having an opposite opening466in fluid communication with the hollow interior467of the body portion434. In an embodiment, the opening466is arranged adjacent to a controller468which has one or more processors that is operable to perform the methods described herein. In an embodiment, the external air flows from the opening466over or around the controller468and out the exhaust vent openings456.

In an embodiment, the controller468is coupled to a wall470of body portion434. In an embodiment, the wall470is coupled to or integral with the handle436. The controller468is electrically coupled to the 2D laser scanner450, the 3D camera460, a power source472, an inertial measurement unit (IMU)474, a laser line projector476(FIG.13), and a haptic feedback device477.

Referring now toFIG.18with continuing reference toFIGS.9-17, elements are shown of the scanner408with the mobile device443installed or coupled to the housing432. Controller468is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. The controller468includes one or more processing elements, or processors478. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one or more processors478have access to memory480for storing information.

Controller468is capable of converting the analog voltage or current level provided by 2D laser scanner450, camera460and IMU474into a digital signal to determine a distance from the scanner408to an object in the environment. In an embodiment, the camera460is a 3D or RGBD type camera. Controller468uses the digital signals that act as input to various processes for controlling the scanner408. The digital signals represent one or more scanner408data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation and roll orientation. As will be discussed in more detail, the digital signals may be from components internal to the housing432or from sensors and devices located in the mobile device443.

In general, when the mobile device443is not installed, controller468accepts data from 2D laser scanner450and IMU474and is given certain instructions for the purpose of generating a two-dimensional map of a scanned environment. Controller468provides operating signals to the 2D laser scanner450, the camera460, laser line projector476and haptic feedback device477. Controller468also accepts data from IMU474, indicating, for example, whether the operator is operating in the system in the desired orientation. The controller468compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, generates a signal that activates the haptic feedback device477. The data received by the controller468may be displayed on a user interface coupled to controller468. The user interface may be one or more LEDs (light-emitting diodes)482, an LCD (liquid-crystal diode) display, a CRT (cathode ray tube) display, or the like. A keypad may also be coupled to the user interface for providing data input to controller468. In one embodiment, the user interface is arranged or executed on the mobile device443.

The controller468may also be coupled to external computer networks such as a local area network (LAN), the Internet, and/or a cloud computing environment such as that shown below inFIG.38. A LAN interconnects one or more remote computers, which are configured to communicate with controllers468using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), RS-232, ModBus, and the like. additional scanners408may also be connected to LAN with the controllers468in each of these scanners408being configured to send and receive data to and from remote computers and other scanners408. The LAN may be connected to the Internet. This connection allows controller468to communicate with one or more remote computers connected to the Internet and/or to a cloud computing environment.

The processors478are coupled to memory480. The memory480may include random access memory (RAM) device484, a non-volatile memory (NVM) device487, a read-only memory (ROM) device488. In addition, the processors478may be connected to one or more input/output (I/O) controllers490and a communications circuit492. In an embodiment, the communications circuit492provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN discussed above, the communications circuit418, and/or the CLOUD.

Controller468includes operation control methods embodied in application code such as that shown or described with reference toFIGS.19-22. These methods are embodied in computer instructions written to be executed by processors478, typically in the form of software. The software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C#, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (HyperText Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing.

Coupled to the controller468is the 2D laser scanner450. The 2D laser scanner450measures 2D coordinates in a plane. In the exemplary embodiment, the scanning is performed by steering light within a plane to illuminate object points in the environment. The 2D laser scanner450collects the reflected (scattered) light from the object points to determine 2D coordinates of the object points in the 2D plane. In an embodiment, the 2D laser scanner450scans a spot of light over an angle while at the same time measuring an angle value and corresponding distance value to each of the illuminated object points.

Examples of 2D laser scanners450include, but are not limited to, Model LMS100 scanners manufactured by Sick, Inc of Minneapolis, Minn. and scanner Models URG-04LX-UG01 and UTM-30LX manufactured by Hokuyo Automatic Co., Ltd of Osaka, Japan. The scanners in the Sick LMS100 family measure angles over a 270 degree range and over distances up to 20 meters. The Hoyuko model URG-04LX-UG01 is a low-cost 2D scanner that measures angles over a 240 degree range and distances up to 4 meters. The Hoyuko model UTM-30LX is a 2D scanner that measures angles over a 270 degree range and to distances up to 30 meters. It should be appreciated that the above 2D scanners are exemplary and other types of 2D scanners are also available.

In an embodiment, the 2D laser scanner450is oriented so as to scan a beam of light over a range of angles in a generally horizontal plane (relative to the floor of the environment being scanned). At instants in time the 2D laser scanner450returns an angle reading and a corresponding distance reading to provide 2D coordinates of object points in the horizontal plane. In completing one scan over the full range of angles, the 2D laser scanner returns a collection of paired angle and distance readings. As the platform100,200,300is moved from place to place, the 2D laser scanner450continues to return 2D coordinate values. These 2D coordinate values are used to locate the position of the scanner408thereby enabling the generation of a two-dimensional map or floor plan of the environment.

Also coupled to the controller486is the IMU474. The IMU474is a position/orientation sensor that may include accelerometers494(inclinometers), gyroscopes496, a magnetometers or compass498, and altimeters. In the exemplary embodiment, the IMU474includes multiple accelerometers494and gyroscopes496. The compass498indicates a heading based on changes in magnetic field direction relative to the earth's magnetic north. The IMU474may further have an altimeter that indicates altitude (height). An example of a widely used altimeter is a pressure sensor. By combining readings from a combination of position/orientation sensors with a fusion algorithm that may include a Kalman filter, relatively accurate position and orientation measurements can be obtained using relatively low-cost sensor devices. In the exemplary embodiment, the IMU474determines the pose or orientation of the scanner108about three-axis to allow a determination of a yaw, roll and pitch parameter.

In the embodiment shown inFIGS.14-17, the scanner408further includes a camera460that is a 3D or RGB-D camera. As used herein, the term 3D camera refers to a device that produces a two-dimensional image that includes distances to a point in the environment from the location of scanner408. The 3D camera460may be a range camera or a stereo camera. In an embodiment, the 3D camera460includes an RGB-D sensor that combines color information with a per-pixel depth information. In an embodiment, the 3D camera460may include an infrared laser projector431(FIG.17), a left infrared camera433, a right infrared camera439, and a color camera437. In an embodiment, the 3D camera460is a RealSense™ camera model R200 manufactured by Intel Corporation.

In an embodiment, when the mobile device443is coupled to the housing432, the mobile device443becomes an integral part of the scanner408. In an embodiment, the mobile device443is a cellular phone, a tablet computer or a personal digital assistant (PDA). The mobile device443may be coupled for communication via a wired connection, such as ports500,502. The port500is coupled for communication to the processor478, such as via I/O controller690for example. The ports500,502may be any suitable port, such as but not limited to USB, USB-A, USB-B, USB-C, IEEE 1394 (Firewire), or Lightning™ connectors.

The mobile device443is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. The mobile device443includes one or more processing elements, or processors504. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one or more processors504have access to memory506for storing information.

The mobile device443is capable of converting the analog voltage or current level provided by sensors508and processor478. Mobile device443uses the digital signals that act as input to various processes for controlling the scanner408. The digital signals represent one or more platform100,200,300data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation, roll orientation, global position, ambient light levels, and altitude for example.

In general, mobile device443accepts data from sensors508and is given certain instructions for the purpose of generating or assisting the processor478in the generation of a two-dimensional map or three-dimensional map of a scanned environment. Mobile device443provides operating signals to the processor478, the sensors508and a display510. Mobile device443also accepts data from sensors508, indicating, for example, to track the position of the mobile device443in the environment or measure coordinates of points on surfaces in the environment. The mobile device443compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, may generate a signal. The data received by the mobile device443may be displayed on display510. In an embodiment, the display510is a touch screen device that allows the operator to input data or control the operation of the scanner408.

The controller468may also be coupled to external networks such as a local area network (LAN), a cellular network, a cloud, and/or the Internet. A LAN interconnects one or more remote computers, which are configured to communicate with controller68using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internee) Protocol), RS-232, ModBus, and the like. additional scanners408may also be connected to LAN with the controllers468in each of these scanners408being configured to send and receive data to and from remote computers and other scanners408. The LAN may be connected to the Internet. This connection allows controller468to communicate with one or more remote computers connected to the Internet.

The processors504are coupled to memory506. The memory506may include random access memory (RAM) device, a non-volatile memory (NVM) device, and a read-only memory (ROM) device. In addition, the processors504may be connected to one or more input/output (I/O) controllers512and a communications circuit514. In an embodiment, the communications circuit514provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN or the cellular network discussed above.

Controller468includes operation control methods embodied in application code shown or described with reference toFIGS.19-22. These methods are embodied in computer instructions written to be executed by processors478,504, typically in the form of software. The software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C#, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (HyperText Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing.

Also coupled to the processor504are the sensors508. The sensors508may include but are not limited to: a microphone516; a speaker518; a front or rear facing camera520; accelerometers522(inclinometers), gyroscopes524, a magnetometers or compass526; a global positioning satellite (GPS) module528; a barometer530; a proximity sensor532; and an ambient light sensor534. By combining readings from a combination of sensors508with a fusion algorithm that may include a Kalman filter, relatively accurate position and orientation measurements can be obtained.

It should be appreciated that the sensors460,474integrated into the scanner408may have different characteristics than the sensors508of mobile device443. For example, the resolution of the cameras460,520may be different, or the accelerometers494,522may have different dynamic ranges, frequency response, sensitivity (mV/g) or temperature parameters (sensitivity or range). Similarly, the gyroscopes496,524or compass/magnetometer may have different characteristics. It is anticipated that in some embodiments, one or more sensors508in the mobile device443may be of higher accuracy than the corresponding sensors474in the scanner408. As described in more detail herein, in some embodiments the processor478determines the characteristics of each of the sensors508and compares them with the corresponding sensors in the scanner408when the mobile device. The processor478then selects which sensors474,508are used during operation. In some embodiments, the mobile device443may have additional sensors (e.g. microphone516, camera520) that may be used to enhance operation compared to operation of the scanner408without the mobile device443. In still further embodiments, the scanner408does not include the IMU474and the processor478uses the sensors508for tracking the position and orientation/pose of the scanner408. In still further embodiments, the addition of the mobile device443allows the scanner408to utilize the camera520to perform three-dimensional (3D) measurements either directly (using an RGB-D camera) or using photogrammetry techniques to generate 3D maps. In an embodiment, the processor478uses the communications circuit (e.g. a cellular 4G internet connection) to transmit and receive data from remote computers or devices.

In an embodiment, the scanner408determines a quality attribute/parameter for the tracking of the scanner408and/or the platform100. In an embodiment, the tracking quality attribute is a confidence level in the determined tracking positions and orientations to actual positions and orientations. When the confidence level crosses a threshold, the platform100may provide feedback to the operator to perform a stationary scan. It should be appreciated that a stationary scan will provide a highly accurate measurements that will allow the determination of the position and orientation of the scanner or platform with a high level of confidence. In an embodiment, the feedback is provided via a user interface. The user interface may be on the platform100, the scanner408, or the scanner610for example.

In the exemplary embodiment, the scanner408is a handheld portable device that is sized and weighted to be carried by a single person during operation. Therefore, the plane536(FIG.22) in which the 2D laser scanner450projects a light beam may not be horizontal relative to the floor or may continuously change as the computer moves during the scanning process. Thus, the signals generated by the accelerometers494, gyroscopes496and compass498(or the corresponding sensors508) may be used to determine the pose (yaw, roll, tilt) of the scanner108and determine the orientation of the plane451.

In an embodiment, it may be desired to maintain the pose of the scanner408(and thus the plane536) within predetermined thresholds relative to the yaw, roll and pitch orientations of the scanner408. In an embodiment, a haptic feedback device477is disposed within the housing432, such as in the handle436. The haptic feedback device477is a device that creates a force, vibration or motion that is felt or heard by the operator. The haptic feedback device477may be but is not limited to: an eccentric rotating mass vibration motor or a linear resonant actuator for example. The haptic feedback device is used to alert the operator that the orientation of the light beam from 2D laser scanner450is equal to or beyond a predetermined threshold. In operation, when the IMU474measures an angle (yaw, roll, pitch or a combination thereof), the controller468transmits a signal to a motor controller538that activates a vibration motor540. Since the vibration originates in the handle436, the operator will be notified of the deviation in the orientation of the scanner408. The vibration continues until the scanner408is oriented within the predetermined threshold or the operator releases the actuator438. In an embodiment, it is desired for the plane536to be within 10-15 degrees of horizontal (relative to the ground) about the yaw, roll and pitch axes.

In an embodiment, the 2D laser scanner450makes measurements as the platform100,200,300is moved about an environment, such from a first position542to a second registration position544as shown inFIG.19. In an embodiment, 2D scan data is collected and processed as the scanner408passes through a plurality of 2D measuring positions546. At each measuring position546, the 2D laser scanner450collects 2D coordinate data over an effective FOV548. Using methods described in more detail below, the controller468uses 2D scan data from the plurality of 2D scans at positions546to determine a position and orientation of the scanner408as it is moved about the environment. In an embodiment, the common coordinate system is represented by 2D Cartesian coordinates x, y and by an angle of rotation θ relative to the x or y axis. In an embodiment, the x and y axes lie in the plane of the 2D scanner and may be further based on a direction of a “front” of the 2D laser scanner450.

FIG.21shows the 2D scanner408collecting 2D scan data at selected positions546over an effective FOV548. At different positions546, the 2D laser scanner450captures a portion of the object550marked A, B, C, D, and E (FIG.20).FIG.21shows 2D laser scanner450moving in time relative to a fixed frame of reference of the object550.

FIG.21includes the same information asFIG.20but shows it from the frame of reference of the scanner408rather than the frame of reference of the object550.FIG.21illustrates that in the scanner408frame of reference, the position of features on the object change over time. Therefore, the distance traveled by the scanner408can be determined from the 2D scan data sent from the 2D laser scanner450to the controller468.

As the 2D laser scanner450takes successive 2D readings and performs best-fit calculations, the controller468keeps track of the translation and rotation of the 2D laser scanner450, which is the same as the translation and rotation of the scanner408. In this way, the controller468is able to accurately determine the change in the values of x, y, θ as the scanner408moves from the first position542to the second position544.

In an embodiment, the controller468is configured to determine a first translation value, a second translation value, along with first and second rotation values (yaw, roll, pitch) that, when applied to a combination of the first 2D scan data and second 2D scan data, results in transformed first 2D data that closely matches transformed second 2D data according to an objective mathematical criterion. In general, the translation and rotation may be applied to the first scan data, the second scan data, or to a combination of the two. For example, a translation applied to the first data set is equivalent to a negative of the translation applied to the second data set in the sense that both actions produce the same match in the transformed data sets. An example of an “objective mathematical criterion” is that of minimizing the sum of squared residual errors for those portions of the scan data determined to overlap. Another type of objective mathematical criterion may involve a matching of multiple features identified on the object. For example, such features might be the edge transitions552,554, and556shown inFIG.19. The mathematical criterion may involve processing of the raw data provided by the 2D laser scanner450to the controller468, or it may involve a first intermediate level of processing in which features are represented as a collection of line segments using methods that are known in the art, for example, methods based on the Iterative Closest Point (ICP). Such a method based on ICP is described in Censi, A., “An ICP variant using a point-to-line metric,” IEEE International Conference on Robotics and Automation (ICRA) 2008, which is incorporated by reference herein.

In an embodiment, assuming that the plane536of the light beam from 2D laser scanner450remains horizontal relative to the ground plane, the first translation value is dx, the second translation value is dy, and the first rotation value dθ. If the first scan data is collected with the 2D laser scanner450having translational and rotational coordinates (in a reference coordinate system) of (x1, y1, θ1), then when the second 2D scan data is collected at a second location the coordinates are given by (x2, y2, θ2)=(x1+dx, y1+dy, θ1+dθ). In an embodiment, the controller468is further configured to determine a third translation value (for example, dz) and a second and third rotation values (for example, pitch and roll). The third translation value, second rotation value, and third rotation value may be determined based at least in part on readings from the IMU474.

The 2D laser scanner450collects 2D scan data starting at the first position542and more 2D scan data at the second position544. In some cases, these scans may suffice to determine the position and orientation of the scanner408at the second position544relative to the first position542. In other cases, the two sets of 2D scan data are not sufficient to enable the controller468to accurately determine the first translation value, the second translation value, and the first rotation value. This problem may be avoided by collecting 2D scan data at intermediate scan positions546. In an embodiment, the 2D scan data is collected and processed at regular intervals, for example, once per second. In this way, features in the environment are identified in successive 2D scans at positions546. In an embodiment, when more than two 2D scans are obtained, the controller468may use the information from all the successive 2D scans in determining the translation and rotation values in moving from the first position542to the second position544. In another embodiment, only the first and last scans in the final calculation, simply using the intermediate 2D scans to ensure proper correspondence of matching features. In most cases, accuracy of matching is improved by incorporating information from multiple successive 2D scans.

It should be appreciated that as the scanner408is moved beyond the second position544, a two-dimensional image or map of the environment being scanned may be generated. It should further be appreciated that in addition to generating a 2D map of the environment, the data from scanner408may be used to generate (and store) a 2D trajectory of the scanner408as it is moved through the environment. In an embodiment, the 2D map and/or the 2D trajectory may be combined or fused with data from other sources in the registration of measured 3D coordinates. It should be appreciated that the 2D trajectory may represent a path followed by the 2D scanner408.

Referring now toFIG.22, a method560is shown for generating a two-dimensional map with annotations. The method560starts in block562where the facility or area is scanned to acquire scan data570, such as that shown inFIG.23. The scanning is performed by carrying the scanner408through the area to be scanned. The scanner408measures distances from the scanner408to an object, such as a wall for example, and also a pose of the scanner408in an embodiment the user interacts with the scanner408via actuator538. In the illustrated embodiments, the mobile device443provides a user interface that allows the operator to initiate the functions and control methods described herein. Using the registration process desired herein, the two dimensional locations of the measured points on the scanned objects (e.g. walls, doors, windows, cubicles, file cabinets etc.) may be determined. It is noted that the initial scan data may include artifacts, such as data that extends through a window572or an open door574for example. Therefore, the scan data570may include additional information that is not desired in a 2D map or layout of the scanned area.

The method560then proceeds to block564where a 2D map576is generated of the scanned area as shown inFIG.24. The generated 2D map576represents a scan of the area, such as in the form of a floor plan without the artifacts of the initial scan data. It should be appreciated that the 2D map576represents a dimensionally accurate representation of the scanned area that may be used to determine the position and pose of the mobile scanning platform100,200,300in the environment to allow the registration of the 3D coordinate points measured by the 3D measurement device110. In the embodiment ofFIG.22, the method560then proceeds to block566where optional user-defined annotations are made to the 2D maps576to define an annotated 2D map that includes information, such as dimensions of features, the location of doors, the relative positions of objects (e.g. liquid oxygen tanks, entrances/exits or egresses or other notable features such as but not limited to the location of automated sprinkler systems, knox or key boxes, or fire department connection points (“FDC”). In an embodiment, the annotation may also be used to define scan locations where the mobile scanning platform300stops and uses the 3D scanner310to perform a stationary scan of the environment.

Once the annotations of the 2D annotated map are completed, the method560then proceeds to block568where the 2D map is stored in memory, such as non-volatile memory device487for example. The 2D map may also be stored in a network accessible storage device or server so that it may be accessed by the desired personnel.

Referring now toFIG.25andFIG.26an embodiment is illustrated with the mobile device443coupled to the scanner408. As described herein, the 2D laser scanner450emits a beam of light in the plane536. The 2D laser scanner450has a field of view (FOV) that extends over an angle that is less than 360 degrees. In the exemplary embodiment, the FOV of the 2D laser scanner is about 270 degrees. In this embodiment, the mobile device443is coupled to the housing432adjacent the end where the 2D laser scanner450is arranged. The mobile device443includes a forward facing camera520. The camera520is positioned adjacent a top side of the mobile device and has a predetermined field of view580. In the illustrated embodiment, the holder441couples the mobile device443on an obtuse angle582. This arrangement allows the mobile device443to acquire images of the floor and the area directly in front of the scanner408(e.g. the direction the operator is moving the platform100,200).

In embodiments where the camera520is an RGB-D type camera, three-dimensional coordinates of surfaces in the environment may be directly determined in a mobile device coordinate frame of reference. In an embodiment, the holder441allows for the mounting of the mobile device443in a stable position (e.g. no relative movement) relative to the 2D laser scanner450. When the mobile device443is coupled to the housing432, the processor478performs a calibration of the mobile device443allowing for a fusion of the data from sensors508with the sensors of scanner408. As a result, the coordinates of the 2D laser scanner may be transformed into the mobile device coordinate frame of reference or the 3D coordinates acquired by camera520may be transformed into the 2D scanner coordinate frame of reference.

In an embodiment, the mobile device is calibrated to the 2D laser scanner450by assuming the position of the mobile device based on the geometry and position of the holder441relative to 2D laser scanner450. In this embodiment, it is assumed that the holder that causes the mobile device to be positioned in the same manner. It should be appreciated that this type of calibration may not have a desired level of accuracy due to manufacturing tolerance variations and variations in the positioning of the mobile device443in the holder441. In another embodiment, a calibration is performed each time a different mobile device443is used. In this embodiment, the user is guided (such as via the user interface/display510) to direct the scanner408to scan a specific object, such as a door, that can be readily identified in the laser readings of the scanner408and in the camera-sensor520using an object recognition method.

Referring now toFIGS.27-29, an embodiment is shown of a laser scanner610. In this embodiment, the laser scanner610has a measuring head622and a base624. The measuring head622is mounted on the base624such that the laser scanner610may be rotated about a vertical axis623. In one embodiment, the measuring head622includes a gimbal point627that is a center of rotation about the vertical axis623and a horizontal axis625. The measuring head622has a rotary mirror626, which may be rotated about the horizontal axis625. The rotation about the vertical axis may be about the center of the base624. In one embodiment, the vertical axis623is coaxial with the center axis of the post109,209,309. The terms vertical axis and horizontal axis refer to the scanner in its normal upright position. It is possible to operate a 3D coordinate measurement device on its side or upside down, and so to avoid confusion, the terms azimuth axis and zenith axis may be substituted for the terms vertical axis and horizontal axis, respectively. The term pan axis or standing axis may also be used as an alternative to vertical axis.

The measuring head622is further provided with an electromagnetic radiation emitter, such as light emitter628, for example, that emits an emitted light beam630. In one embodiment, the emitted light beam630is a coherent light beam such as a laser beam. The laser beam may have a wavelength range of approximately 300 to 1600 nanometers, for example 790 nanometers, 905 nanometers, 1550 nm, or less than 400 nanometers. It should be appreciated that other electromagnetic radiation beams having greater or smaller wavelengths may also be used. The emitted light beam630is amplitude or intensity modulated, for example, with a sinusoidal waveform or with a rectangular waveform. The emitted light beam630is emitted by the light emitter628onto a beam steering unit, such as mirror626, where it is deflected to the environment. A reflected light beam632is reflected from the environment by an object634. The reflected or scattered light is intercepted by the rotary mirror626and directed into a light receiver636. The directions of the emitted light beam630and the reflected light beam632result from the angular positions of the rotary mirror626and the measuring head622about the axes625,623, respectively. These angular positions in turn depend on the corresponding rotary drives or motors.

Coupled to the light emitter628and the light receiver636is a controller638. The controller638determines, for a multitude of measuring points X, a corresponding number of distances d between the laser scanner610and the points X on object634. The distance to a particular point X is determined based at least in part on the speed of light in air through which electromagnetic radiation propagates from the device to the object point X. In one embodiment the phase shift of modulation in light emitted by the laser scanner610and the point X is determined and evaluated to obtain a measured distance d.

The speed of light in air depends on the properties of the air such as the air temperature, barometric pressure, relative humidity, and concentration of carbon dioxide. Such air properties influence the index of refraction n of the air. The speed of light in air is equal to the speed of light in vacuum c divided by the index of refraction. In other words, cair=c/n. A laser scanner of the type discussed herein is based on the time-of-flight (TOF) of the light in the air (the round-trip time for the light to travel from the device to the object and back to the device). Examples of TOF scanners include scanners that measure round trip time using the time interval between emitted and returning pulses (pulsed TOF scanners), scanners that modulate light sinusoidally and measure phase shift of the returning light (phase-based scanners), as well as many other types. A method of measuring distance based on the time-of-flight of light depends on the speed of light in air and is therefore easily distinguished from methods of measuring distance based on triangulation. Triangulation-based methods involve projecting light from a light source along a particular direction and then intercepting the light on a camera pixel along a particular direction. By knowing the distance between the camera and the projector and by matching a projected angle with a received angle, the method of triangulation enables the distance to the object to be determined based on one known length and two known angles of a triangle. The method of triangulation, therefore, does not directly depend on the speed of light in air.

In one mode of operation, the scanning of the volume around the scanner610takes place by rotating the rotary mirror626relatively quickly about axis625while rotating the measuring head622relatively slowly about axis623, thereby moving the assembly in a spiral pattern. This is sometimes referred to as a compound mode of operation. In an exemplary embodiment, the rotary mirror rotates at a maximum speed of 5820 revolutions per minute. For such a scan, the gimbal point627defines the origin of the local stationary reference system. The base624rests in this local stationary reference system. In other embodiments, another mode of operation is provided wherein the scanner610rotates the rotary mirror626about the axis625while the measuring head622remains stationary. This is sometimes referred to as a helical mode of operation.

In an embodiment, the acquisition of the 3D coordinate values further allows for the generation of a 3D trajectory, such as the 3D trajectory (e.g. 3D path) of the gimbal point627for example. This 3D trajectory may be stored and combined or fused with other data, such as data from the 2D scanner and/or from an inertial measurement unit for example and used to register 3D coordinate data. It should be appreciated that the 3D trajectory may be transformed from the gimbal point627to any other location on the system, such as the base unit.

In addition to measuring a distance d from the gimbal point627to an object point X, the laser scanner610may also collect gray-scale information related to the received optical power (equivalent to the term “brightness.”) The gray-scale value may be determined at least in part, for example, by integration of the bandpass-filtered and amplified signal in the light receiver636over a measuring period attributed to the object point X.

The measuring head622may include a display device640integrated into the laser scanner610. The display device640may include a graphical touch screen641, which allows the operator to set the parameters or initiate the operation of the laser scanner610. For example, the screen641may have a user interface that allows the operator to provide measurement instructions to the device, and the screen may also display measurement results.

The laser scanner610includes a carrying structure642that provides a frame for the measuring head622and a platform for attaching the components of the laser scanner610. In one embodiment, the carrying structure642is made from a metal such as aluminum. The carrying structure642includes a traverse member644having a pair of walls646,648on opposing ends. The walls646,648are parallel to each other and extend in a direction opposite the base624. Shells650,652are coupled to the walls646,648and cover the components of the laser scanner610. In the exemplary embodiment, the shells650,652are made from a plastic material, such as polycarbonate or polyethylene for example. The shells650,652cooperate with the walls646,648to form a housing for the laser scanner610.

On an end of the shells650,652opposite the walls646,648a pair of yokes654,656are arranged to partially cover the respective shells650,652. In the exemplary embodiment, the yokes654,656are made from a suitably durable material, such as aluminum for example, that assists in protecting the shells650,652during transport and operation. The yokes654,656each includes a first arm portion658that is coupled, such as with a fastener for example, to the traverse member644adjacent the base624. The arm portion658for each yoke654,656extends from the traverse member644obliquely to an outer corner of the respective shell650,652. From the outer corner of the shell, the yokes654,656extend along the side edge of the shell to an opposite outer corner of the shell. Each yoke654,656further includes a second arm portion that extends obliquely to the walls646,648. It should be appreciated that the yokes654,656may be coupled to the traverse member644, the walls646,648and the shells650,654at multiple locations.

The pair of yokes654,656cooperate to circumscribe a convex space within which the two shells650,652are arranged. In the exemplary embodiment, the yokes654,656cooperate to cover all of the outer edges of the shells650,654, while the top and bottom arm portions project over at least a portion of the top and bottom edges of the shells650,652. This provides advantages in protecting the shells650,652and the measuring head622from damage during transportation and operation. In other embodiments, the yokes654,656may include additional features, such as handles to facilitate the carrying of the laser scanner610or attachment points for accessories for example.

In an embodiment, on top of the traverse member644, a prism660is provided. The prism extends parallel to the walls646,648. In the exemplary embodiment, the prism660is integrally formed as part of the carrying structure642. In other embodiments, the prism660is a separate component that is coupled to the traverse member644. When the mirror626rotates, during each rotation the mirror626directs the emitted light beam630onto the traverse member644and the prism660. In some embodiments, due to non-linearities in the electronic components, for example in the light receiver636, the measured distances d may depend on signal strength, which may be measured in optical power entering the scanner or optical power entering optical detectors within the light receiver636, for example. In an embodiment, a distance correction is stored in the scanner as a function (possibly a nonlinear function) of distance to a measured point and optical power (generally unscaled quantity of light power sometimes referred to as “brightness”) returned from the measured point and sent to an optical detector in the light receiver636. Since the prism660is at a known distance from the gimbal point627, the measured optical power level of light reflected by the prism660may be used to correct distance measurements for other measured points, thereby allowing for compensation to correct for the effects of environmental variables such as temperature. In the exemplary embodiment, the resulting correction of distance is performed by the controller638.

In an embodiment, the base624is coupled to a swivel assembly (not shown) such as that described in commonly owned U.S. Pat. No. 8,705,012 ('012), which is incorporated by reference herein. The swivel assembly is housed within the carrying structure642and includes a motor that is configured to rotate the measuring head622about the axis623. In an embodiment, the angular/rotational position of the measuring head622about the axis623is measured by angular encoder. In the embodiments disclosed herein, the base (with or without the swivel assembly) may be mounted to the post109,209, or309.

An auxiliary image acquisition device666may be a device that captures and measures a parameter associated with the scanned area or the scanned object and provides a signal representing the measured quantities over an image acquisition area. The auxiliary image acquisition device666may be, but is not limited to, a pyrometer, a thermal imager, an ionizing radiation detector, or a millimeter-wave detector. In an embodiment, the auxiliary image acquisition device666is a color camera.

In an embodiment, a central color camera (first image acquisition device612) is located internally to the scanner and may have the same optical axis as the 3D scanner device. In this embodiment, the first image acquisition device612is integrated into the measuring head622and arranged to acquire images along the same optical pathway as emitted light beam630and reflected light beam632. In this embodiment, the light from the light emitter628reflects off a fixed mirror616and travels to dichroic beam-splitter618that reflects the light617from the light emitter628onto the rotary mirror626. In an embodiment, the mirror626is rotated by a motor699and the angular/rotational position of the mirror is measured by angular encoder697. The dichroic beam-splitter618allows light to pass through at wavelengths different than the wavelength of light617. For example, the light emitter628may be a near infrared laser light (for example, light at wavelengths of 780 nm or 1150 nm), with the dichroic beam-splitter618configured to reflect the infrared laser light while allowing visible light (e.g., wavelengths of 400 to 700 nm) to transmit through. In other embodiments, the determination of whether the light passes through the beam-splitter618or is reflected depends on the polarization of the light. The digital camera612obtains 2D images of the scanned area to capture color data to add to the scanned image. In the case of a built-in color camera having an optical axis coincident with that of the 3D scanning device, the direction of the camera view may be easily obtained by simply adjusting the steering mechanisms of the scanner—for example, by adjusting the azimuth angle about the axis623and by steering the mirror626about the axis625. One or both of the color cameras612,666may be used to colorize the acquired 3D coordinates (e.g. the point cloud).

In an embodiment, when the 3D scanner is operated in compound mode, a compound compensation may be performed to optimize the registration of date by combining or fusing sensor data (e.g. 2D scanner, 3D scanner and/or IMU data) using the position and orientation (e.g. trajectory) of each sensor.

It should be appreciated that while embodiments herein refer to the 3D scanner610as being a time-of-flight (phase shift or pulsed) scanner, this is for exemplary purposes and the claims should not be so limited. In other embodiments, other types of 3D scanners may be used, such as but not limited to structured light scanners, area scanners, triangulation scanners, photogrammetry scanners, or a combination of the foregoing.

Referring now toFIGS.30-32, an embodiment is shown of a method700for scanning an environment with the mobile scanning platform100,200, or300. The method700starts in block702where the platform is configured. In the embodiment where the platform is platform100or200, the configuring may include attaching the 2D scanner108or208to the respective arm or holder, and the 3D measurement device110or210to the post109, or209. In an embodiment where the platform is platform300, the configuring may include determining a path for the platform300to follow and defining stationary scan locations (if desired). In an embodiment, the path may be determined using the system and method described in commonly owned U.S. patent application Ser. No. 16/154,240, the contents of which are incorporated by reference herein. Once the path is defined, the 2D scanner308and 3D scanner310may be coupled to the platform300. It should be appreciated that in some embodiments, the platform300may be remotely controlled by an operator and the step of defining a path may not be performed.

Once the platform100,200, or300is configured, the method700proceeds to block704where the 2D scanner108,208,308, or408is initiated and the 3D measurement device110,210,310, or610is initiated in block706. It should be appreciated that when operation of the 2D scanner108,208,308, or408is initiated, the 2D scanner starts to generate a 2D map of the environment as described herein. Similarly, when operation of the 3D measurement device110,210,310, or610is initiated, the coordinates of 3D points in the environment are acquired in a volume about the 3D scanner.

The method700then proceeds to block708where the platform100,200, or300is moved through the environment. As the platform100,200, or300is moved, both the 2D scanner108,208,308, or408and the 3D measurement device110,210,310, or610continue to operate. This results in the generation of both a 2D map710(FIG.31) and the acquisition of 3D points711. In an embodiment, as the 2D map is generated, the location or path712of the platform100,200,300is indicated on the 2D map. In an embodiment, the platform100may include a user interface that provides feedback to the operator during the performing of the scan. In an embodiment, a quality attribute (e.g. scan density) of the scanning process may be determined during the scan. When the quality attribute crosses a threshold (e.g. scan density too low), the user interface may provide feedback to the operator. In an embodiment, the feedback is for the operator to perform a stationary scan with the 3D scanner.

The method700then proceeds to block714where the acquired 3D coordinate points are registered into a common frame of reference. It should be appreciated that since the platform100,200, or300is moving while the 3D measurement device110,210,310, or610is acquiring data, the local frame of reference of the 3D scanner is also changing. Using the position and pose data from the 2D scanner108,208,308, or408, the frame of reference of the acquired 3D coordinate points may be registered into a global frame of reference. In an embodiment, the registration is performed as the platform100,200, or300is moved through the environment. In another embodiment, the registration is done when the scanning of the environment is completed.

The registration of the 3D coordinate points allows the generation of a point cloud716(FIG.32) in block718. In an embodiment, a representation of the path720of the platform100,200, or300is shown in the point cloud716. In some embodiments, the point cloud716is generated and displayed to the user as the platform100,200,300moves through the environment being scanned. In these embodiments, blocks708,714,718may loop continuously until the scanning is completed. With the scan complete, the method700ends in block722where the point cloud716and 2D map710are stored in memory of a controller or processor system.

As described previously, mobile 2D and 3D capturing devices can become inaccurate over distance due to the accumulation or error or drift. For example, contemporary approaches that utilize simultaneous localization and mapping (SLAM) techniques can drift over distance (e.g., long hallways) and their accuracy depends on surrounding conditions. When they are available, local reference systems such as spheres or points or checkers can be used as reference points by contemporary systems to reduce or minimize the drift. In addition, natural features, or landmarks, such as walls or windows can be used as reference points. When reference systems are not available and natural features are relied on for mapping, it can be difficult to map areas such as, but not limited to: large areas; large surfaces (planar or curved); glass; irregular features; and long distances.

Turning now toFIGS.33A and33B, a schematic illustration of an image802of a portion of a structure as generated by a mobile mapping system that utilizes the techniques described herein is generally shown. The image shown inFIG.33Aillustrates a map generated using a contemporary algorithm that results in a portion804experiencing drift.FIG.33Bshows portion804magnified to illustrate how the map806generated by the contemporary mobile system deviates from the true structure808.

Turning now toFIGS.34A and34B, a schematic illustration of an image810of the portion of the structure shown inFIGS.33A and33Bas generated by a mobile scanning platform that utilizes pre-existing data to correct current scan data is generally shown in accordance with an embodiment. In accordance with an embodiment the current scan data is registered with a previously generated map and detected differences between the current scan data and the previously generated map are overlaid onto the existing map. The image shown inFIG.34Aillustrates a map generated by overlaying pre-existing data (e.g., a previously generated map) such as a CAD model or a golden point cloud with the scanned data.FIG.34Bshows portion804magnified to illustrate how the map806generated by the mobile scanning platform closely approximates (or is the same as) the true structure808.

Turning now toFIG.35, a flow diagram of a method for using pre-existing data to correct current scan data generated by a mobile scanner is generally shown in accordance with an embodiment. The processing shown inFIG.35can be performed, for example, by software executed on a scanning system such as FARO® ScanPlan™ for example or executing on a scanning system such as mobile scanning platform100,200, and/or300. At block3502, a previously generated map of the environment is uploaded to the scanning system. The previously generated map can be used in place of, or as a supplement to, generating the map at block710ofFIG.30. In accordance with one or more embodiments, the previously generated map is an existing CAD floor plan such as the original floor plan820shown inFIG.36.FIG.36depicts a schematic illustration of a drawing (DWG) formatted original floor plan820in accordance with an embodiment of the present invention. The DWG format includes lines and vectors, and one or more embodiments of the present invention converts the original floor plan820from DWG format into an extracted mapping outline822that is compatible with the scanning software executing on the scanning system. In accordance with an embodiment, the extracted mapping outline822includes extracted landmarks and/or wall lines that can be used for improved tracking. Formats of the extracted mapping outline822can include, but are not limited to “WRL”, “COR”, “CSV”, and “OBJ.” WRL and OBJ are CAD data formats that represent the layout as objects. COR and CSV are data files including coordinates that can deliver, for example, the edges of the walls and therefore represent the layout as lines. In accordance with one or more embodiments, the scanning software creates a new project and imports the converted mapping outline822.

At block3504, the scanning system scans the environment by moving through the environment and measuring a plurality of coordinate values that form a point cloud. The scanning system starts mapping within the now known environment of the extracted mapping outline822. At block3506, the scan is registered with the previously generated map. In accordance with one or more embodiments, the registration can be performed by the user tapping on a location on the uploaded map where the scanning will start. This location can correspond to a marker such as, but not limited to: an optional room quick response (QR) code; and a radio frequency identifier (RFID) tag with room information.

In accordance with one or more embodiments, coordinate values measured by the scanner as it moves through the environment are registered to features (e.g., landmarks such as walls and windows) in the previously generated map using landmarks such as walls and windows. Feature registration can be used to support the use of a SLAM algorithm by providing constraints such as ninety-degree corners and/or straight walls for use in tracking. As described above, the start location of the mobile mapping system is known and marked on the map. Therefore, a rough registration is already done by the user, and the 3D point cloud acquired by the mobile system can be projected in one plane using a top-view algorithm for registration. The top-view algorithm projects the 3D points into one layer or plane and compares these planes (e.g., walls) of different scan positions with each other. In accordance with an embodiment, the top-view algorithm is used to register a point cloud by the mobile system and the map instead of two point clouds. In this manner, the 3D point cloud can be transformed into a 2D layout in data formats such as, but not limited to WRL, COR, CSV, and OBJ.

Processing continues at block3508where the previously generated map is updated with detected differences between the data generated by the scanner and the data in the previously generated map. Static deviations from the uploaded map to the mapping algorithm used by the scanning system can be adapted according to existing algorithms After registration, it is assumed that the point cloud and the map are correctly positioned. In an embodiment, the map is used as a reference, so that an iterative closest point (ICP) algorithm can detect and correct the drifting parts of the point cloud. An ICP algorithm that is used for cloud-to-cloud registration can be adapted and used by one or more embodiments. As shown inFIG.37, a rectangular floor having a long length (e.g., 15 feet, 25 feet, 40 feet) is scanned. The mapping result without using prior information828is compared to the mapping result where prior information830such as a previously generated map is utilized.

At block3508, new 360-degree images of the environment (e.g., a building) are created as the scanning software collects scan data (e.g., a plurality of coordinate values making up a point cloud) and updates the previously generated map if required. The updating can be performed, for example, by overlaying portions of the previously generated map with corresponding portions of the point cloud. In accordance with one or more embodiments, when the scanning is completed, the updated extracted mapping outline822is converted back into a DWG format with the updated information from the scanning.

In accordance with one or more other embodiments, the current scan data, or point cloud, is updated based on information in the previously generated map and the previously generated map is not modified. The updating can be performed, for example, by overlaying portions of the point cloud with corresponding portions of the previously generated map. In accordance with one or more embodiments, when the scanning is completed, the updated point cloud is converted into a DWG format with the updated information from the previously generated map.

In one or embodiments of the present invention, the mobile mapping system includes a 2D scanner and/or a 3D scanner.

In one or more embodiments of the present invention, the previously generated map is a 2D or 3D point cloud, a CAD model, and/or a floor plan.

In an embodiment, a 2D point cloud may be used as the previously generated map. In an embodiment, the 2D point cloud may be generated from a 3D point cloud generated by a scanner such as scanner610for example. The 2D point cloud may be extracted by extracting points from a plane that is parallel to (or substantially parallel to) the floor of the structure for example.

In one or more embodiments of the present invention, the mobile mapping system is configured to be carried by an operator without stopping the measurement of the plurality of 2D coordinates.

One or more embodiments include facilitating scanning of an environment using a mobile platform while simultaneously generating a 2D map of the environment and a point cloud. The base unit is moved (e.g., continuously) through the environment and includes a 2D scanner for measuring an angle and a distance value, and a 3D scanner having a color camera and operating in a compound mode. As the base unit is moving, the 2D scanner generates a 2D map of the environment based at least in part on the angle, the distance value, and a previously generated map of the environment. As the base unit is moving through the environment, the 3D scanner is operating in a compound mode to measure a plurality of 3D coordinate values. The 3D coordinate values are registered into a single frame of reference based at least in part on the 2D map.

Technical effects and benefits of some embodiments include providing a system and a method that facilitate the rapid scanning of an environment using a movable platform that utilizes previously generated maps of the environment to correct drifting errors.

It should be appreciated that while embodiments herein describe a coordinate measurement device in reference to laser scanner this is for exemplary purposes and the claims should not be so limited. In other embodiments, the scan processing software may be executed on, or receive data from, any coordinate measurement device capable of measuring and determining either 2D or 3D coordinates of an object or the environment while moving. The coordinate measurement device may be but is not limited to: an articulated arm coordinate measurement machine, a laser tracker, an image scanner, a photogrammetry device, a triangulation scanner, a laser line probe, or a structured light scanner for example.

It is understood in advance that although this disclosure describes using pre-existing data to correct current scan data generated by a mobile scanner in reference to cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. In essence, cloud computing is an infrastructure made up of a network of interconnected nodes.

Referring now toFIG.38, an illustrative cloud computing environment is depicted. As shown, cloud computing environment comprises one or more cloud computing nodes10with which local computing devices used by cloud consumers, such as, for example, coordinate measurement device13and computers1115may communicate. In an embodiment, the correction of current scan data using pre-existing data is performed through the cooperation of computer15or11, and the coordinate measurement device13. For example, the previously generated map may be accessed from computers1115and/or one or more of nodes10. Nodes10may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices shown inFIG.38are intended to be illustrative only and that computing nodes10and cloud computing environment can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Referring now toFIG.39, a set of functional abstraction layers provided by cloud computing environment (FIG.38) is shown. It should be understood in advance that the components, layers, and functions shown inFIG.39are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided: hardware and software layer12includes hardware and software components. Examples of hardware components include: mainframes14; RISC (Reduced Instruction Set Computer) architecture based servers16; servers18; blade servers20; storage devices22; and networks and networking components24. In some embodiments, software components include network application server software26, and database software28; virtualization layer30provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers32; virtual storage34; virtual networks36, including virtual private networks; virtual applications and operating systems38; and virtual clients40.

In one example, management layer42may provide the functions described below. Resource provisioning44provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and pricing46provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may comprise application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal48provides access to the cloud computing environment for consumers and system administrators. Service level management50provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment52provides pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

Workloads layer54provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation56; software development and lifecycle management58; transaction processing60; scan processing software62; point cloud to virtual reality data processing64; and user defined content to point cloud processing66.

Turning now toFIG.40, a schematic illustration of a system900is depicted upon which aspects of one or more embodiments of correcting current scan data using pre-existing data may be implemented. In an embodiment, all or a portion of the system900may be incorporated into one or more of the 3D scanner device and processors described herein. In one or more exemplary embodiments, in terms of hardware architecture, as shown inFIG.40, the computer901includes a processing device905and a memory910coupled to a memory controller915and an input/output controller935. The input/output controller935can be, for example, but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The input/output controller935may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the computer901may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.

In one or more exemplary embodiments, a keyboard950and mouse955or similar devices can be coupled to the input/output controller935. Alternatively, input may be received via a touch-sensitive or motion sensitive interface (not depicted). The computer901can further include a display controller925coupled to a display930.

The processing device905is a hardware device for executing software, particularly software stored in secondary storage920or memory910. The processing device905can be any custom made or commercially available computer processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the computer901, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macro-processor, or generally any device for executing instructions.

The memory910can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), flash drive, disk, hard disk drive, diskette, cartridge, cassette or the like, etc.). Moreover, the memory910may incorporate electronic, magnetic, optical, and/or other types of storage media. Accordingly, the memory910is an example of a tangible computer readable storage medium940upon which instructions executable by the processing device905may be embodied as a computer program product. The memory910can have a distributed architecture, where various components are situated remote from one another, but can be accessed by the processing device905.

The instructions in memory910may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. In the example ofFIG.40, the instructions in the memory910include a suitable operating system (OS)911and program instructions916. The operating system911essentially controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. When the computer901is in operation, the processing device905is configured to execute instructions stored within the memory910, to communicate data to and from the memory910, and to generally control operations of the computer901pursuant to the instructions. Examples of program instructions916can include instructions to implement the processing described herein in reference toFIGS.1-39.

The computer901ofFIG.40also includes a network interface960that can establish communication channels with one or more other computer systems via one or more network links. The network interface960can support wired and/or wireless communication protocols known in the art. For example, when embodied in a user system, the network interface960can establish communication channels with an application server.

It will be appreciated that aspects of the present invention may be embodied as a system, method, or computer program product and may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.), or a combination thereof. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

The computer readable medium may contain program code embodied thereon, which may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. In addition, computer program code for carrying out operations for implementing aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

In addition, some embodiments described herein are associated with an “indication”. As used herein, the term “indication” may be used to refer to any indicia and/or other information indicative of or associated with a subject, item, entity, and/or other object and/or idea. As used herein, the phrases “information indicative of” and “indicia” may be used to refer to any information that represents, describes, and/or is otherwise associated with a related entity, subject, or object. Indicia of information may include, for example, a code, a reference, a link, a signal, an identifier, and/or any combination thereof and/or any other informative representation associated with the information. In some embodiments, indicia of information (or indicative of the information) may be or include the information itself and/or any portion or component of the information. In some embodiments, an indication may include a request, a solicitation, a broadcast, and/or any other form of information gathering and/or dissemination.

“Determining” something can be performed in a variety of manners and therefore the term “determining” (and like terms) includes calculating, computing, deriving, looking up (e.g., in a table, database or data structure), ascertaining and the like.

It will be readily apparent that the various methods and algorithms described herein may be implemented by, e.g., appropriately and/or specially-programmed general purpose computers and/or computing devices. Typically, a processor (e.g., one or more microprocessors) will receive instructions from a memory or like device, and execute those instructions, thereby performing one or more processes defined by those instructions. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of media (e.g., computer readable media) in a number of manners. In some embodiments, hard-wired circuitry or custom hardware may be used in place of, or in combination with, software instructions for implementation of the processes of various embodiments. Thus, embodiments are not limited to any specific combination of hardware and software.

A “processor” generally means any one or more microprocessors, CPU devices, GPU devices, computing devices, microcontrollers, digital signal processors, or like devices, as further described herein. A CPU typically performs a variety of tasks while a GPU is optimized to display images.

Terms such as processor, controller, computer, DSP, FPGA are understood in this document to mean a computing device that may be located within an instrument, distributed in multiple elements throughout an instrument, or placed external to an instrument.