PHOTOGRAMMETRY SYSTEM FOR GENERATING STREET EDGES IN TWO-DIMENSIONAL MAPS

A computer-implemented method is provided that includes retrieving at least one selected image from a plurality of aerial images of an environment, the at least one selected image comprising surface regions that are concurrently in a three-dimensional (3D) point cloud of the environment. The method further includes detecting areas of the surface regions in the at least one selected image, such that coordinates of the areas of the surface regions are extracted from the at least one selected image. The method further includes comparing the at least one selected image to the 3D point cloud to align common locations in both the at least one selected image and the 3D point cloud. The method further includes displaying an integration of a drawing of the coordinates of the areas of the surface regions in a representation of the 3D point cloud.

BACKGROUND

The present invention relates generally to a photogrammetry system for generating street edges in two-dimensional (2D) maps, and more specifically, to automatic 2D sketch generation using photogrammetry.

The points in a three-dimensional (3D) point cloud, such as that generated by a 3D laser scanner time-of-flight (TOF) coordinate measurement device or created by algorithms that takes data from photogrammetry, are very useful. A 3D TOF laser scanner of this type steers a beam of light to a non-cooperative target such as a diffusely scattering surface of an object. A distance meter in the device measures a distance to the object, and angular encoders measure the angles of rotation of two axles in the device. The measured distance and two angles enable a processor in the device to determine the 3D coordinates of the target.

A TOF laser scanner is a scanner in which the distance to a target point is determined based on the speed of light in air between the scanner and a target point. Laser scanners are typically used for scanning closed or open spaces such as interior areas of buildings, industrial installations and tunnels. They may be used, for example, in industrial applications and accident reconstruction applications. A laser scanner optically scans and measures objects in a volume around the scanner through the acquisition of data points representing object surfaces within the volume. Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value. This raw scan data is collected, stored and sent to a processor or processors to generate a 3D image representing the scanned area or object.

Generating an image requires at least three values for each data point. These three values may include the distance and two angles, or may be transformed values, such as the x, y, z coordinates. In an embodiment, an image is also based on a fourth gray-scale value, which is a value related to irradiance of scattered light returning to the scanner.

Most TOF scanners direct the beam of light within the measurement volume by steering the light with a beam steering mechanism. The beam steering mechanism includes a first motor that steers the beam of light about a first axis by a first angle that is measured by a first angular encoder (or other angle transducer). The beam steering mechanism also includes a second motor that steers the beam of light about a second axis by a second angle that is measured by a second angular encoder (or other angle transducer).

Many contemporary laser scanners include a camera mounted on the laser scanner for gathering camera digital images of the environment and for presenting the camera digital images to an operator of the laser scanner. By viewing the camera images, the operator of the scanner can determine the field of view of the measured volume and adjust settings on the laser scanner to measure over a larger or smaller region of space. In addition, the camera digital images may be transmitted to a processor to add color to the scanner image. To generate a color scanner image, at least three positional coordinates (such as x, y, z) and three color values (such as red, green, blue “RGB”) are collected for each data point.

A 3D point cloud of data points is formed by the set of three positional coordinates (such as x, y, z) and three color values (such as red, green, blue “RGB”). Processing is generally performed on the 3D point cloud of data points which can include millions of data points. However, additional software processing tools for 3D data points in a 3D point cloud can be helpful to a user.

Accordingly, while existing 3D scanners and existing processing for 3D point clouds are suitable for their intended purposes, what is needed is a 3D point cloud processing tool having certain features of embodiments disclosed herein.

BRIEF DESCRIPTION

According to one embodiment, a computer-implemented method using a photogrammetry system for generating street edges in two-dimensional (2D) maps is provided. The method includes retrieving at least one selected image from a plurality of aerial images of an environment, the at least one selected image comprising surface regions that are concurrently in a three-dimensional (3D) point cloud of the environment. The method includes detecting areas/objects (having edges) of the surface regions in the at least one selected image, such that coordinates of the areas/objects (including the edges) of the surface regions are extracted from the at least one selected images. The method includes comparing the at least one selected image to the 3D point cloud to align common locations in both the selected image and the 3D point cloud, and displaying an integration of a drawing of the coordinates of the edges of the surface regions in a representation of the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the 3D point cloud is generated from the plurality of aerial images using photogrammetry.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the at least one selected image is selected from the plurality of aerial images having been used to generate the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the areas/objects (including the edges) of the surface regions are detected using machine learning.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the plurality of aerial images are orthoimages.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the coordinates of the areas/objects (including the edges) of the surface regions are connected by lines to form the drawing of the areas/objects (having the edges), the lines being formed along edges of the surface regions.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein forward projection is utilized to integrate the drawing into the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein back projection is utilized to integrate 3D data of the 3D point cloud into the at least one selected images.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include wherein the drawing of the coordinates of the edges of the surface regions is displayed in an orthographic view of the 3D point cloud.

According to an embodiment, a system is provided. The system includes a memory having computer readable instructions and one or more processors for executing the computer readable instructions. The computer readable instructions control the one or more processors to perform operations. The operations include retrieving at least one selected image from a plurality of aerial images of an environment, the at least one selected image comprising surface regions that are concurrently in a three-dimensional (3D) point cloud of the environment. The operations further include detecting areas of the surface regions in the at least one selected image, such that coordinates of the areas of the surface regions are extracted from the at least one selected image. The operations further include comparing the at least one selected image to the 3D point cloud to align common locations in both the at least one selected image and the 3D point cloud. The operations further include displaying an integration of a drawing of the coordinates of the areas of the surface regions in a representation of the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the 3D point cloud is generated from the plurality of aerial images using photogrammetry.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the at least one selected image is selected from the plurality of aerial images having been used to generate the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the areas of the surface regions are detected using machine learning.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the plurality of aerial images are orthoimages.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the coordinates of the areas of the surface regions are connected by lines to form the drawing of the areas, the lines being formed along edges of the surface regions.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that forward projection is utilized to integrate the drawing into the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that back projection is utilized to integrate 3D data of the 3D point cloud into the at least one selected image; and smoothing the at least one selected image by finding 3D points for a line in which the 3D points have a minimal squared sum of a back projection error into a plurality of images.

In addition to one or more features described herein, or as an alternative, further embodiments of the system may include that the drawing of the coordinates of the areas of the surface regions is displayed in an orthographic view of the 3D point cloud.

According to an embodiment, a method is provided. The method includes retrieving at least one selected image from a plurality of aerial images of an environment, the at least one selected image comprising surface regions. The method further includes detecting areas of the surface regions in the at least one selected image, such that coordinates of the areas of the surface regions are extracted from the at least one selected image. The method further includes comparing the at least one selected image to a 3D point cloud to align common locations in both the at least one selected image and the 3D point cloud. The method further includes displaying an integration of a drawing of the coordinates of the areas of the surface regions in a representation of the 3D point cloud.

In addition to one or more features described herein, or as an alternative, further embodiments of the method may include that the 3D point cloud is generated from the plurality of aerial images using photogrammetry.

DETAILED DESCRIPTION

The present invention relates to automatic two-dimensional (2D) overview sketch (e.g., a plan view of a scene) generation using photogrammetry which will provide the accurately sketched lines into a three-dimensional (3D) point cloud. Unmanned autonomous vehicles, commonly referred to as “drones,” are quickly becoming essential to public safety professionals who work with crash scene reconstruction. With an inexpensive entrance level in pricing, drones provide fast capture on crash scenes and provide advantages when compared to older methods of data collection. Safety professionals, including the police, need to see street edges marked in the 3D point cloud and/or to filter 3D points lying outside the road edges, as disclosed in one or more embodiments. This can help the police make determinations during a crash scene reconstruction along with providing documentation to support their analysis. Although a safety professional could attempt to make user-drawn edges of the street using a software such a FARO® Zone 3D software, FARO® Zone 2D software, etc., the user drawn edges may not be precise as desired and are time consuming. One or more embodiments are configured to utilize photogrammetry (including image locations and image positions) to automatically generate a 2D sketch overview in the 2D images of photogrammetry and then project the 2D sketch into the 3D point cloud, such that the street edges and/or any other roadway delineations are accurately viewable in the 3D point cloud.

Technical effects and benefits of one or more embodiments include the efficient and automatic reproduction of street edges of streets and roadways for a 3D point cloud generated using 2D images acquired by photogrammetry, thereby saving time and providing a higher level of accuracy over user-drawn lines/edges in a 3D point cloud. Drones, also referred to as unmanned aerial vehicles (UAVs) and remotely piloted aircraft (RPA), have been used to measure two-dimensional coordinates and/or three-dimensional coordinates, and they provide a cost effective way to measure objects or environments without incurring the effort and expense of building structures to support the scanning devices. These systems allow for the rapid acquisition of coordinates in a wide variety of environments.

Referring now toFIG.1, an example drone1020is depicted. The drone1020can include a scanner670such as the laser scanner20as discussed inFIGS.2A,2B,3, and4and/or another suitable three-dimensional coordinate scanning device. Similarly, the drone1020can include a camera680, for example, having features of the cameras66,112(of laser scanner20depicted inFIGS.2A-4) and/or another suitable camera. The drone1020includes a fuselage1022that supports at least one thrust device1024. In an embodiment, the drone1020includes a plurality of thrust devices1024A,1024B, such as four thrust devices arranged about the periphery of the fuselage1022. In an embodiment, the thrust devices1024include a propeller member that rotates to produce thrust. The thrust devices1024may be configurable to provide both lift (vertical thrust) and lateral thrust (horizontal thrust). The vertical and horizontal components of the thrust allow the changing of the altitude, lateral movement, and orientation (attitude) of the drone1020.

In the exemplary embodiment, the fuselage1022and thrust devices1024are sized and configured to carry a payload such as an optical scanner670that is configured to measure three-dimensional coordinates of points in the environment or on an object. Particularly, the drone1020can carry the camera680and/or the scanner670. The scanner670may be a time-of-flight scanner, a triangulation scanner, an area scanner, a structured light scanner, or a laser tracker for example. In an embodiment, the scanner670may be releasably coupled to the fuselage1022by a coupling1028. The camera680may be releasably coupled to the fuselage1022by a coupling1029.

In another embodiment, the scanner670may be integral with or fixedly coupled to the fuselage1022. As will be discussed in more detail herein, the scanner670may also be coupled to a scanner controller38by a communication and power connection1030. Similarly, the camera680may be coupled to the controller1038by a communication and power connection1031. It should be appreciated that the scanner controller38may be located in the scanner20, within the fuselage1022, or include multiple processing units that are distributed between the scanner20, the fuselage1022, or are remotely located from the drone1020. The scanner controller38may be coupled to communicate with a drone controller1038.

Both the drone controller1038and the scanner controller38may include processors that are responsive to operation control methods embodied in application code. These methods are embodied in computer instructions written to be executed by the processor, such as in the form of software. The controller1038is coupled to the thrust devices1024and configured to transmit and receive signals from the thrust devices1024. The controller1038may further be coupled to one or more sensor devices that enable to the controller to determine the position, orientation, and altitude of the drone1020. In an embodiment, these sensors may include an altimeter1040, a gyroscope or accelerometers1042or a global positioning satellite (GPS) system1044. In other embodiments, the controller1038may be coupled to other sensors, such as force sensors. The drone controller1038may be coupled to a communication adapter1037to transmit and receive signals from the computer system602over the network650, such that the drone controller1038can execute instructions/commands from the computer system602as depicted inFIG.6. Also, the drone controller1038can send images to the computer system602.

FIGS.2A,2B, and3depict a coordinate measurement device, such as a laser scanner20for optically scanning and measuring the environment surrounding the laser scanner20. The laser scanner20has a measuring head22and a base24. The measuring head22is mounted on the base24such that the laser scanner20may be rotated about a vertical axis23. In one embodiment, the measuring head22includes a gimbal point27that is a center of rotation about the vertical axis23and a horizontal axis25. The measuring head22has a rotary mirror26, which may be rotated about the horizontal axis25. The rotation about the vertical axis may be about the center of the base24. The terms vertical axis and horizontal axis refer to the scanner in its normal upright position. It is possible to operate a 3D coordinate measurement device on its side or upside down, and so to avoid confusion, the terms azimuth axis and zenith axis may be substituted for the terms vertical axis and horizontal axis, respectively. The term pan axis or standing axis may also be used as an alternative to vertical axis.

The measuring head22is further provided with an electromagnetic radiation emitter, such as light emitter28, for example, that emits an emitted light beam30. In one embodiment, the emitted light beam30is a coherent light beam such as a laser beam. The laser beam may have a wavelength range of approximately 300 to 1600 nanometers, for example 790 nanometers, 905 nanometers, 1550 nm, or less than 400 nanometers. It should be appreciated that other electromagnetic radiation beams having greater or smaller wavelengths may also be used. The emitted light beam30is amplitude or intensity modulated, for example, with a sinusoidal waveform or with a rectangular waveform. The emitted light beam30is emitted by the light emitter28onto a beam steering unit, such as mirror26, where it is deflected to the environment. A reflected light beam32is reflected from the environment by an object34. The reflected or scattered light is intercepted by the rotary minor26and directed into a light receiver36. The directions of the emitted light beam30and the reflected light beam32result from the angular positions of the rotary mirror26and the measuring head22about the axes25and23, respectively. These angular positions in turn depend on the corresponding rotary drives or motors.

Coupled to the light emitter28and the light receiver36is a controller38. The controller38determines, for a multitude of measuring points X, a corresponding number of distances d between the laser scanner20and the points X on object34. The distance to a particular point X is determined based at least in part on the speed of light in air through which electromagnetic radiation propagates from the device to the object point X. In one embodiment the phase shift of modulation in light emitted by the laser scanner20and the point X is determined and evaluated to obtain a measured distance d.

The speed of light in air depends on the properties of the air such as the air temperature, barometric pressure, relative humidity, and concentration of carbon dioxide. Such air properties influence the index of refraction n of the air. The speed of light in air is equal to the speed of light in vacuum c divided by the index of refraction. In other words, cair=c/n. A laser scanner of the type discussed herein is based on the time-of-flight (TOF) of the light in the air (the round-trip time for the light to travel from the device to the object and back to the device). Examples of TOF scanners include scanners that measure round trip time using the time interval between emitted and returning pulses (pulsed TOF scanners), scanners that modulate light sinusoidally and measure phase shift of the returning light (phase-based scanners), as well as many other types. A method of measuring distance based on the time-of-flight of light depends on the speed of light in air and is therefore easily distinguished from methods of measuring distance based on triangulation. Triangulation-based methods involve projecting light from a light source along a particular direction and then intercepting the light on a camera pixel along a particular direction. By knowing the distance between the camera and the projector and by matching a projected angle with a received angle, the method of triangulation enables the distance to the object to be determined based on one known length and two known angles of a triangle. The method of triangulation, therefore, does not directly depend on the speed of light in air.

In one mode of operation, the scanning of the volume around the laser scanner20takes place by rotating the rotary minor26relatively quickly about axis25while rotating the measuring head22relatively slowly about axis23, thereby moving the assembly in a spiral pattern. In an exemplary embodiment, the rotary mirror rotates at a maximum speed of 5820 revolutions per minute. For such a scan, the gimbal point27defines the origin of the local stationary reference system. The base24rests in this local stationary reference system.

In addition to measuring a distance d from the gimbal point27to an object point X, the scanner20may also collect gray-scale information related to the received optical power (equivalent to the term “brightness.”) The gray-scale value may be determined at least in part, for example, by integration of the bandpass-filtered and amplified signal in the light receiver36over a measuring period attributed to the object point X.

The measuring head22may include a display device40integrated into the laser scanner20. The display device40may include a graphical touch screen41, as shown inFIG.2A, which allows the operator to set the parameters or initiate the operation of the laser scanner20. For example, the screen41may have a user interface that allows the operator to provide measurement instructions to the device, and the screen may also display measurement results.

The laser scanner20includes a carrying structure42that provides a frame for the measuring head22and a platform for attaching the components of the laser scanner20. In one embodiment, the carrying structure42is made from a metal such as aluminum. The carrying structure42includes a traverse member44having a pair of walls46,48on opposing ends. The walls46,48are parallel to each other and extend in a direction opposite the base24. Shells50,52are coupled to the walls46,48and cover the components of the laser scanner20. In the exemplary embodiment, the shells50,52are made from a plastic material, such as polycarbonate or polyethylene for example. The shells50,52cooperate with the walls46,48to form a housing for the laser scanner20.

On an end of the shells50,52opposite the walls46,48a pair of yokes54,56are arranged to partially cover the respective shells50,52. In the exemplary embodiment, the yokes54,56are made from a suitably durable material, such as aluminum for example, that assists in protecting the shells50,52during transport and operation. The yokes54,56each includes a first arm portion58that is coupled, such as with a fastener for example, to the traverse44adjacent the base24. The arm portion58for each yoke54,56extends from the traverse44obliquely to an outer corner of the respective shell50,52. From the outer corner of the shell, the yokes54,56extend along the side edge of the shell to an opposite outer corner of the shell. Each yoke54,56further includes a second arm portion that extends obliquely to the walls46,48. It should be appreciated that the yokes54,56may be coupled to the traverse42, the walls46,48and the shells50,54at multiple locations.

The pair of yokes54,56cooperate to circumscribe a convex space within which the two shells50,52are arranged. In the exemplary embodiment, the yokes54,56cooperate to cover all of the outer edges of the shells50,54, while the top and bottom arm portions project over at least a portion of the top and bottom edges of the shells50,52. This provides advantages in protecting the shells50,52and the measuring head22from damage during transportation and operation. In other embodiments, the yokes54,56may include additional features, such as handles to facilitate the carrying of the laser scanner20or attachment points for accessories for example.

On top of the traverse44, a prism60is provided. The prism extends parallel to the walls46,48. In the exemplary embodiment, the prism60is integrally formed as part of the carrying structure42. In other embodiments, the prism60is a separate component that is coupled to the traverse44. When the mirror26rotates, during each rotation the mirror26directs the emitted light beam30onto the traverse44and the prism60. Due to non-linearities in the electronic components, for example in the light receiver36, the measured distances d may depend on signal strength, which may be measured in optical power entering the scanner or optical power entering optical detectors within the light receiver36, for example. In an embodiment, a distance correction is stored in the scanner as a function (possibly a nonlinear function) of distance to a measured point and optical power (generally unscaled quantity of light power sometimes referred to as “brightness”) returned from the measured point and sent to an optical detector in the light receiver36. Since the prism60is at a known distance from the gimbal point27, the measured optical power level of light reflected by the prism60may be used to correct distance measurements for other measured points, thereby allowing for compensation to correct for the effects of environmental variables such as temperature. In the exemplary embodiment, the resulting correction of distance is performed by the controller38.

In an embodiment, the base24is coupled to a swivel assembly (not shown) such as that described in commonly owned U.S. Pat. No. 8,705,012 ('012), which is incorporated by reference herein. The swivel assembly is housed within the carrying structure42and includes a motor138that is configured to rotate the measuring head22about the axis23. In an embodiment, the angular/rotational position of the measuring head22about the axis23is measured by angular encoder134.

An auxiliary image acquisition device66may be a device that captures and measures a parameter associated with the scanned area or the scanned object and provides a signal representing the measured quantities over an image acquisition area. The auxiliary image acquisition device66may be, but is not limited to, a pyrometer, a thermal imager, an ionizing radiation detector, or a millimeter-wave detector. In an embodiment, the auxiliary image acquisition device66is a color camera.

In an embodiment, a central color camera (first image acquisition device)112is located internally to the scanner and may have the same optical axis as the 3D scanner device. In this embodiment, the first image acquisition device112is integrated into the measuring head22and arranged to acquire images along the same optical pathway as emitted light beam30and reflected light beam32. In this embodiment, the light from the light emitter28reflects off a fixed minor116and travels to dichroic beam-splitter118that reflects the light117from the light emitter28onto the rotary minor26. In an embodiment, the mirror26is rotated by a motor136and the angular/rotational position of the mirror is measured by angular encoder134. The dichroic beam-splitter118allows light to pass through at wavelengths different than the wavelength of light117. For example, the light emitter28may be a near infrared laser light (for example, light at wavelengths of 780 nm or 1150 nm), with the dichroic beam-splitter118configured to reflect the infrared laser light while allowing visible light (e.g., wavelengths of 400 to 700 nm) to transmit through. In other embodiments, the determination of whether the light passes through the beam-splitter118or is reflected depends on the polarization of the light. The digital camera112obtains 2D images of the scanned area to capture color data to add to the scanned image. In the case of a built-in color camera having an optical axis coincident with that of the 3D scanning device, the direction of the camera view may be easily obtained by simply adjusting the steering mechanisms of the scanner—for example, by adjusting the azimuth angle about the axis23and by steering the minor26about the axis25.

Referring now toFIG.4with continuing reference toFIGS.2A-3, elements are shown of the laser scanner20. Controller38is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. The controller38includes one or more processing elements122. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one or more processors122have access to memory124for storing information.

Controller38is capable of converting the analog voltage or current level provided by light receiver36into a digital signal to determine a distance from the laser scanner20to an object in the environment. Controller38uses the digital signals that act as input to various processes for controlling the laser scanner20. The digital signals represent one or more laser scanner20data including but not limited to distance to an object, images of the environment, images acquired by panoramic camera126, angular/rotational measurements by a first or azimuth encoder132, and angular/rotational measurements by a second axis or zenith encoder134.

In general, controller38accepts data from encoders132,134, light receiver36, light source28, and panoramic camera126and is given certain instructions for the purpose of generating a 3D point cloud of a scanned environment. Controller38provides operating signals to the light source28, light receiver36, panoramic camera126, zenith motor136and azimuth motor138. The controller38compares the operational parameters to predetermined variances and if the predetermined variance is exceeded, generates a signal that alerts an operator to a condition. The data received by the controller38may be displayed on a user interface40coupled to controller38. The user interface40may be one or more LEDs (light-emitting diodes), an LCD (liquid-crystal diode) display, a CRT (cathode ray tube) display, a touch-screen display or the like. A keypad may also be coupled to the user interface for providing data input to controller38. In one embodiment, the user interface is arranged or executed on a mobile computing device that is coupled for communication, such as via a wired or wireless communications medium (e.g., Ethernet, serial, USB, Bluetooth™ or WiFi) for example, to the laser scanner20.

The controller38may also be coupled to external computer networks such as a local area network (LAN) and the Internet. A LAN interconnects one or more remote computers, which are configured to communicate with controller38using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet({circumflex over ( )}) Protocol), RS-232, ModBus, and the like. Additional systems may also be connected to LAN with the controllers38in each of these systems being configured to send and receive data to and from remote computers and other systems. The LAN may be connected to the Internet. This connection allows controller38to communicate with one or more remote computers connected to the Internet.

The processors122are coupled to memory124. The memory124may include random access memory (RAM) device140, a non-volatile memory (NVM) device142, and a read-only memory (ROM) device144. In addition, the processors122may be connected to one or more input/output (I/O) controllers146and a communications circuit148. In an embodiment, the communications circuit148provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN discussed above.

Controller38includes operation control methods embodied in application code. These methods are embodied in computer instructions written to be executed by processors122, typically in the form of software. The software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (HyperText Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing.

It should be appreciated that while some embodiments herein describe a point cloud that is generated by a TOF scanner, this is for example purposes and the claims should not be so limited. In other embodiments, the point cloud may be generated or created using other types of scanners, such as but not limited to triangulation scanners, area scanners, structured-light scanners, laser line scanners, flying dot scanners, and photogrammetry devices for example.

Turning now toFIG.5, a computer system500is generally shown in accordance with one or more embodiments of the invention. The computer system500can be an electronic, computer framework comprising and/or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein. The computer system500can be easily scalable, extensible, and modular, with the ability to change to different services or reconfigure some features independently of others. The computer system500can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system500can be a cloud computing node. Computer system500can be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system500can be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

As shown inFIG.5, the computer system500has one or more central processing units (CPU(s))501a,501b,501c, etc., (collectively or generically referred to as processor(s)501). The processors501can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations. The processors501, also referred to as processing circuits, are coupled via a system bus502to a system memory503and various other components. The system memory503can include a read only memory (ROM)504and a random access memory (RAM)505. The ROM504is coupled to the system bus502and can include a basic input/output system (BIOS) or its successors like Unified Extensible Firmware Interface (UEFI), which controls certain basic functions of the computer system500. The RAM is read-write memory coupled to the system bus502for use by the processors501. The system memory503provides temporary memory space for operations of said instructions during operation. The system memory503can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.

The computer system500comprises an input/output (I/O) adapter506and a communications adapter507coupled to the system bus502. The I/O adapter506can be a small computer system interface (SCSI) adapter that communicates with a hard disk508and/or any other similar component. The I/O adapter506and the hard disk508are collectively referred to herein as a mass storage510.

Software511for execution on the computer system500can be stored in the mass storage510. The mass storage510is an example of a tangible storage medium readable by the processors501, where the software511is stored as instructions for execution by the processors501to cause the computer system500to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail. The communications adapter507interconnects the system bus502with a network512, which can be an outside network, enabling the computer system500to communicate with other such systems. In one embodiment, a portion of the system memory503and the mass storage510collectively store an operating system, which can be any appropriate operating system to coordinate the functions of the various components shown inFIG.5.

Additional input/output devices are shown as connected to the system bus502via a display adapter515and an interface adapter516. In one embodiment, the adapters506,507,515, and516can be connected to one or more I/O buses that are connected to the system bus502via an intermediate bus bridge (not shown). A display519(e.g., a screen or a display monitor) is connected to the system bus502by the display adapter515, which can include a graphics controller to improve the performance of graphics intensive applications and a video controller. A keyboard521, a mouse522, a speaker523, etc., can be interconnected to the system bus502via the interface adapter516, which can include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit. Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI) and the Peripheral Component Interconnect Express (PCIe). Thus, as configured inFIG.5, the computer system500includes processing capability in the form of the processors501, storage capability including the system memory503and the mass storage510, input means such as the keyboard521and the mouse522, and output capability including the speaker523and the display519.

In some embodiments, the communications adapter507can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network512can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device can connect to the computer system500through the network512. In some examples, an external computing device can be an external webserver or a cloud computing node.

It is to be understood that the block diagram ofFIG.5is not intended to indicate that the computer system500is to include all of the components shown inFIG.5. Rather, the computer system500can include any appropriate fewer or additional components not illustrated inFIG.5(e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the embodiments described herein with respect to computer system500can be implemented with any appropriate logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware, in various embodiments.

FIG.6is a block diagram of a computer system602for automatic 2D overview drawing or sketch generation using photogrammetry according to one or more embodiments. Elements of computer system500may be used in and/or integrated in computer system602.

Data in database690in memory608can include a 3D point cloud, also referred to as 3D point cloud data, point cloud, a 3D image, etc. The 3D point cloud includes 3D point cloud data points. Data in database690in memory608can include 2D images. In an embodiment, the 2D images were acquired while performing photogrammetry at a scene, such as an automobile accident scene or a crime scene for example. In the embodiments described herein, the data in database690can be generated by a camera via photogrammetry; however other types of coordinate measurement devices may be used for generating the 3D point cloud data, such as but not limited to a TOF laser scanner a structured light scanner or a triangulation scanner, and/or another suitable three-dimensional coordinate scanning device. In an embodiment, the drone being used for photogrammetry may include an optical scanner670, such as a TOF laser scanner for example. Software application604can be used with, integrated in, call, and/or be called by other software applications, such as machine learning model606, photogrammetry software612, drawing software, etc., for processing 3D point cloud data and 2D images as understood by one of ordinary skill in the art.

In one or more embodiments, software application604can be employed by a user for processing and manipulating 2D images and 3D point cloud data using a user interface such as, for example, a keyboard, mouse, touch screen, stylus, etc. Software application604can include and/or work with a graphical user interface (GUI), and features of the software application604can receive the output from the machine learning model606to edit, draw, and animate 2D images and/or 3D point cloud data as discussed herein. As understood by one of ordinary skill in the art, software application604includes functionality for processing any 2D image and 3D image including a 3D point cloud. In one or more embodiments, the software application604can include features of, be representative of, and/or be implemented in FARO® Zone 2D Software, FARO® Zone 3D Software, FARO® PhotoCore Software, and/or FARO® Scene Software, all of which are provided by FARO® Technologies, Inc. Software application604can call and/or include the features and functionality of photogrammetry software612. Photogrammetry is a technique to obtain reliable data of real-world objects in the environment by creating 3D models from photos. 2D and 3D data is extracted from an image and, with overlapping photos of an object, building, scene, or terrain, converted into a digital 3D model.

Photogrammetry is a technique for modeling objects using images, such as photographic images acquired by a digital camera for example. Photogrammetry can make 3D models from 2D images or photographs. When two or more images are acquired at different positions that have an overlapping field of view, common points or features may be identified on each image. By projecting a ray from the camera location to the feature/point on the object, the 3D coordinate of the feature/point may be determined using trigonometry or triangulation. In some examples, photogrammetry may be based on markers/targets (e.g., lights or reflective stickers) or based on natural features. To perform photogrammetry, for example, images are captured, such as with a camera (e.g., the camera680) having a sensor, such as a photosensitive array for example. By acquiring multiple images of an object, or a portion of the object, from different positions or orientations, 3D coordinates of points on the object may be determined based on common features or points and information on the position and orientation of the camera when each image was acquired. In order to obtain the desired information for determining 3D coordinates, the features are identified in two or more images. Since the images are acquired from different positions or orientations, the common features are located in overlapping areas of the field of view of the images. It should be appreciated that photogrammetry techniques are described in commonly-owned U.S. Pat. No. 10,597,753, the contents of which are incorporated by reference herein. With photogrammetry, two or more images are captured and used to determine 3D coordinates of features.

The various components, modules, engines, etc. described regarding the computer system602can be implemented as instructions stored on a computer-readable storage medium, as hardware modules, as special-purpose hardware (e.g., application specific hardware, application specific integrated circuits (ASIC s), application specific special processors (ASSPs), field programmable gate arrays (FPGAs), as embedded controllers, hardwired circuitry, etc.), or as some combination or combinations of these. According to aspects of the present disclosure, the engine(s) described herein can be a combination of hardware and programming. The programming can be processor executable instructions stored on a tangible memory, and the hardware can include the computer system602for executing those instructions. Thus, a system memory (e.g., the memory608) can store program instructions that when executed by the computer system602implement the engines described herein. Other engines can also be utilized to include other features and functionality described in other examples herein.

A network adapter (not shown) provides for the computer system602to transmit data to and/or receive data from other sources, such as other processing systems, data repositories, and the like. As an example, the computer system602can transmit data to and/or receive data from the camera680, the scanner670, and/or a user device660directly and/or via a network670.

The camera680can be a 2D camera or a 3D camera (RGBD or time-of-flight for example). The camera680captures an image (or multiple images), such as of an environment160. The camera680transmits the images to the computer system602. In one or more embodiments, the camera680encrypts the image before transmitting it to the computer system602. Although not shown, the camera680can include components such as a processing device, a memory, a network adapter, and the like, which may be functionally similar to those included in the computer system500,602as described herein.

In some examples, the camera680is mounted to a mobile base, which can be moved about the environment160. In some examples, the camera680is disposed in or mounted to an unmanned aerial vehicle. In various examples, the camera680is mounted on a manned aerial vehicle and/or unmanned aerial vehicle, generally referred to as a drone. In some examples, the camera680is mounted to a fixture, which is user-configurable to rotate about a roll axis, a pan axis, and a tilt axis. In such examples, the camera680is mounted to the fixture to rotate about the roll axis, the pan axis, and the tilt axis. Other configurations of mounting options for the camera680also are possible.

A coordinate measurement device, such as scanner670for example, is any suitable device for measuring 3D coordinates or points in an environment, such as the environment160, to generate data about the environment. The scanner670may be implemented as a TOF laser scanner20. A collection of 3D coordinate points is sometimes referred to as a point cloud. According to one or more embodiments described herein, the scanner670is a three-dimensional (3D) laser scanner time-of-flight (TOF) coordinate measurement device. It should be appreciated that while embodiments herein may refer to a laser scanner, this is for example purposes and the claims should not be so limited. In other embodiments, other types of coordinate measurement devices or combinations of coordinate measurement devices may be used, such as but not limited to triangulation scanners, structured light scanners, laser line probes, photogrammetry devices, and the like. A 3D TOF laser scanner steers a beam of light to a non-cooperative target such as a diffusely scattering surface of an object. A distance meter in the scanner670measures a distance to the object, and angular encoders measure the angles of rotation of two axles in the device. The measured distance and two angles enable a processor in the scanner670to determine the 3D coordinates of the target.

A TOF laser scanner, such as the scanner670, is a scanner in which the distance to a target point is determined based on the speed of light in air between the scanner and a target point. Laser scanners are typically used for scanning closed or open spaces such as interior areas of buildings, industrial installations, and tunnels. They may be used, for example, in industrial applications and accident reconstruction applications. A laser scanner, such as the scanner670, optically scans and measures objects in a volume around the scanner670through the acquisition of data points representing object surfaces within the volume. Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value. This raw scan data is collected and stored as a point cloud, which can be transmitted to the computer system602and stored in the database690about the environment160.

In some examples, the scanner670is mounted to a mobile base, which can be moved about the environment160. In some examples, the scanner670is disposed in or mounted to an unmanned aerial vehicle. In various examples, the scanner670is mounted on a manned aerial vehicle and/or unmanned aerial vehicle, generally referred to as a drone. In some examples, the scanner670is mounted to a fixture, which is user-configurable to rotate about a roll axis, a pan axis, and a tilt axis. In such examples, the scanner670is mounted to the fixture to rotate about the roll axis, the pan axis, and the tilt axis. Other configurations of mounting options for the scanner670also are possible.

According to one or more embodiments described herein, the camera680captures 2D image(s) of the environment160and the scanner670captures 3D information of the environment160. In some examples, the camera680and the scanner670are separate devices; however, in some examples, the camera680and the scanner670are integrated into a single device. For example, the camera680can include depth acquisition functionality and/or can be used in combination with a 3D acquisition depth camera, such as a time of flight camera, a stereo camera, a triangulation scanner, LIDAR, and the like. In some examples, 3D information can be measured/acquired/captured using a projected light pattern and a second camera (or the camera680) using triangulation techniques for performing depth determinations. In some examples, a time-of-flight (TOF) approach can be used to enable intensity information (2D) and depth information (3D) to be acquired/captured. The camera680can be a stereo-camera to facilitate 3D acquisition. In some examples, a 2D image and 3D information (i.e., a 3D data set) can be captured/acquired at the same time; however, the 2D image and the 3D information can be obtained at different times.

The user device660(e.g., a smartphone, a laptop or desktop computer, a tablet computer, a wearable computing device, a smart display, and the like) can also be located within or proximate to the environment160. The user device660can display an image of the environment160, such as on a display of the user device660(e.g., the display519of the computer system500ofFIG.5) along with a digital visual element. In some examples, the user device660can include components such as a processor, a memory, an input device (e.g., a touchscreen, a mouse, a microphone, etc.), an output device (e.g., a display, a speaker, etc.), and the like.

For ease of understanding and not limitation, an example scenario is illustrated that using drones to assist public safety professionals with crash scene reconstruction. It should be appreciated that embodiments are not limited to the example scenario and other environments may be used.FIG.7depicts a flowchart700for processing images according to one or more embodiments. A drone, such an unmanned or manned aerial vehicle, may include the camera680to capture aerial images of a physical location which can be the environment160. The physical location can be a crash scene. Numerous images (e.g., typically several hundred images) may be captured and stored as data in database690in computer system602.

At block702, the software application604is configured to retrieve and/or receive images622of the environment160(e.g., crash scene) from the database690. The images622are (2D) aerial images of the environment160. The images622and other 2D images in database903may have been converted to orthoimages, orthophotos, or orthoimages. An orthophoto, orthophotograph, orthoimage, or orthoimagery is an aerial photograph or satellite imagery geometrically corrected (“orthorectified”) such that the scale is uniform: the photo or image follows a given map projection. Unlike an uncorrected aerial photograph, an orthophoto can be used to measure true distances, because it is an accurate representation of the earth's surface, having been adjusted for topographic relief, lens distortion, and camera tilt.

Although aerial images captured by drones may be utilized, images are not limited to those captured by drones. Other orthographic photos/images can be used including point cloud scanner data, Google® images, map photos/images, etc. Additionally, the images may be converted into orthoimages, using a suitable technique as understood by one of ordinary skill in the art.

For accuracy and distance validation, various techniques can be used. An agency capturing aerial images can typically use the FARO® scalebar (for NIST traceability). This is an accurate way to verify distance measurement (of the drawing/sketch discussed herein) because it is based on software measurements and not the user. If a scalebar is not present, one can use a known measurement in their scene. Typically, a yard stick can be used or a tape measure at a known length can be used in the scene of the captured image. If a known measurement is not used, the user can utilize fixed points, such as the lane width or a door width of a vehicle. There are many options about how many times to use a measurement for accuracy. Some may use the measure just once in a project. Others may use the measurement for accuracy at the first and last scan, while some may use it for each scan.

Referring toFIG.7at block704, the software application604is configured to call, use, and/or communicate with photogrammetry software612to perform photogrammetric processing or photogrammetry on the images622. Photogrammetry is a technique to obtain reliable data of real-world objects in the environment by creating 3D models from photos. Digital image capturing and photogrammetric processing includes several well defined stages, which allow the generation of 2D or 3D digital models of the object as an end product, as obtained by software application604at block706. Photogrammetry may include the following: feature detection in the images, feature matching between images, estimation of position and orientation for each image in 3D space, bundle adjustment to fine tune the position and orientation, and output of a 3D point cloud of the environment160.

As a result of the feature extraction, feature matching, and bundle adjust during the photogrammetry, the photogrammetry software612also returns the image positions and orientations (of the 2D images622utilized to create the 3D point cloud628) in the coordinate system of the calculated 3D point cloud, which can be stored in database690. This information of the image positions and orientations is relevant for further processing discussed inFIG.8. As stored in database690, the photogrammetry software612also returns the camera model and the internal camera parameters which can be utilized to refine the results for further processing discussed inFIG.8.FIG.9depicts a combination of (2D) aerial images of an example environment according to one or more embodiments. In this example, aerial images may be combined using photogrammetry to result in the 2D image depicted inFIG.9, as understood by one of ordinary skill in the art. Although a crash scene may be illustrated, other types of aerial images may be utilized.FIG.10Aillustrates example drone poses for capturing the aerial images according to one or more embodiments. Photogrammetry may be utilized to create a 3D point cloud from the aerial images at different poses inFIG.10. For example,FIG.18illustrates an example of multiple drone images1802where photogrammetry is utilized to generate a 3D point cloud1804of the drone images1802.

FIG.8depicts a flowchart of a computer-implemented method800for generating a drawing/sketch of street edges in 2D image(s) and generating the sketch in a 3D image according to one or more embodiments described herein. The computer-implemented method800can be performed by or implemented on any suitable processing system, for example, the computer system602inFIG.6, a cloud computing node, and/or combinations thereof.

At block802of the computer-implemented method800, the software application604is configured to perform image selection to receive/retrieve one or more 2D images622representing the environment160having been captured by the camera680. The environment160, for example, is the physical location of the crash scene having streets or roadways, where the 2D images622capture the environment. All (drone) images may be utilized for the photogrammetric process discussed inFIG.7or another process for photogrammetry. However, the software application604is configured to utilize a subset of images in database690to find 2D structures or regions of interest in the 3D point cloud. The software application604may be configured to select certain ones of the images622based on some of the photogrammetry statistics, such as (only) select images that do not share many common features (e.g., less than a predetermined number of common features), which is utilized to reduce the redundancy of selected images. In cases where some redundancy is desired, the software application604can be configured to select certain ones of the images622based on having line segments and/or street marking seen in several images for improved accuracy or robustness.

At block804, the software application604employs the machine learning model606to detect and segment objects/areas including surface regions/areas, for example, streets and/or roadways, in the selected (2D) images622. Surface regions are streets and roadways, and the streets and roadways have edges delineating the surface region from another region such as the grass, curb, etc. In one or more embodiments, the machine learning mode606can identify (output) which parts of the image622depict a street, a parking lot, a building, a vehicle, walkway, road/street markings, sign boards, etc., as illustrated inFIG.17. In one or more embodiments, the software application604may have rules-based instructions to determine the street edges of streets and/or roadways, which were identified/segmented as objects in the images622. In one or more embodiments, the software application604is configured to input the selected images622to the machine learning model606for processing to determine the street edges. It is noted that street markings and lane lines are for separating traffic traveling the opposite and/or same directions.

The machine learning model606is trained to identify structures in the images622as segments (or objects) which represent the street edges of the street and also identify street markings on the street. The machine learning model606is configured to generate an output data626as 2D coordinates (e.g., x, y coordinates) of the segments (or segmented objects) of the street edges and street markings in the image(s)622. The segments can also be vehicles, parking lots, building, etc. The software application604is configured to store the output data626.

FIGS.11A and12Adepict drone images as example aerial images. After processing by the machine learning model606,FIG.11Billustrates a monochromatic predicted image of the drone image inFIG.11A, whereFIG.11Bshows that the image has been segmented. After processing by the machine learning model606,FIG.12Billustrates a predicted image of the drone image inFIG.12A, whereFIG.12Bshows that the image has been segmented. InFIGS.11B and12B, the street or roadway (i.e., surface region) has been segmented along with street marking and other objects. Coordinates (x, y coordinates) of the street edges are captured and included in the output data626because each segmented object has coordinates. Also, the coordinates of traffic lines, markings, and points of interest may also be captured and saved in the output data626. Moreover, the output data626of the machine learning model606is a mask which identifies (i.e., segments) all objects from the present in the images, and machine learning model606may apply a unique color coding, pattern, shading, etc., to each segmented object as depicted inFIG.17. The mask indicates where each object is and what objects are present in the 2D image622. Streets are object which the trained machine learning model606(neural network) detects; the street edges may be determined/inferred using computer vision techniques or and/or rules-based algorithms.

Optionally, as segmentation improvements, the software application604may compare the street edges in the output data626to streets in the selected images622. In cases where there happen to be broken lines of the street edges, the software application604can fill or connect the broken lines of the street edges accordingly. For example, the software application604can identify a train of broken lines for street edges in the output data626and find the corresponding locations (x, y coordinates) in the selected images622, and then extrapolate to find the missing line pieces (and/or coordinates). The software application604is configured to include any additional x, y coordinates found during the segmentation improvement in the output data626.

In one or more embodiments, the machine learning model606can include various engines/classifiers and/or can be implemented on a neural network. The features of the engines/classifiers can be implemented by configuring and arranging the computer system602to execute machine learning algorithms. In general, machine learning algorithms, in effect, extract features from received data (e.g., inputs of 2D images) in order to “classify” the received data. Examples of suitable classifiers include but are not limited to neural networks, support vector machines (SVMs), logistic regression, decision trees, hidden Markov Models (HMMs), etc. The end result of the classifier's operations, i.e., the “classification,” is to predict a class for the data. The machine learning algorithms apply machine learning techniques to the received data in order to, over time, create/train/update a unique “model.” The learning or training performed by the engines/classifiers can be supervised, unsupervised, or a hybrid that includes aspects of supervised and unsupervised learning. Supervised learning is when training data is already available and classified/labeled. Unsupervised learning is when training data is not classified/labeled so must be developed through iterations of the classifier. Unsupervised learning can utilize additional learning/training methods including, for example, clustering, anomaly detection, neural networks, deep learning, and the like.

In one or more embodiments the engines/classifiers are implemented as neural networks (or artificial neural networks), which use a connection (synapse) between a pre-neuron and a post-neuron, thus representing the connection weight.

Neuromorphic systems are interconnected elements that act as simulated “neurons” and exchange “messages” between each other. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in neuromorphic systems such as neural networks carry electronic messages between simulated neurons, which are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making neuromorphic systems adaptive to inputs and capable of learning. After being weighted and transformed by a function (i.e., transfer function) determined by the network's designer, the activations of these input neurons are then passed to other downstream neurons, which are often referred to as “hidden” neurons. This process is repeated until an output neuron is activated. Thus, the activated output neuron determines (or “learns”) and provides an output or inference regarding the input.

Neural networks are usually created with base networks and based on requirements. Example base networks utilized include RESNET50, RESNET 10, Xception. It should be appreciated that other base network could be utilized for images. After the creation of the neural network, the dataset (training and testing) is fed to the model with the specific loss function and the training is started. Training consists of different hyperparameters that need to be set in order to achieve better accuracy. The dataset that is fed into the deep learning model is processed, and this called data preparation and augmentation. For illustration and not limitation, the training datasets include aerial (2D) images of streets, roadways, and crash scenes. In some cases, pretrained or partially trained neural networks were used which were further trained using the training datasets. Supervised learning was utilized in which the 2D images were manually segmented and fed to the neural network.

The raw dataset is collected and sorted manually. The sorted dataset is labeled (e.g., using the Amazon Web Services® (AWS®) labeling tool such as Amazon SageMaker® Ground Truth). The labeling tool creates segmentation masks. Although additional labels can be used, a few examples of the labeling used in training the machine learning model606included the main road, the side road, walkway, road marking, etc. Both the images and masks are sorted again in order to achieve data balancing and divided into training, testing, and validation datasets. Training and validation are used for training and evaluation, while testing is used after training to test the machine learning model on an unseen dataset. The training dataset may be processed through different data augmentation techniques. Training takes the labeled datasets, base networks, loss functions, and hyperparameters. Once these are all created and compiled, the training of the neural network occurs to eventually result in the trained machine learning model. Once the model is trained, the model (including the adjusted weights) is saved to a file for deployment and/or further testing on the test dataset.

Returning toFIG.8, at block806, the software application604is configured to perform photogrammetry and/or employ the photogrammetry software612to perform photogrammetry on the aerial images for the environment160(e.g., crash scene) in database690, resulting in a 3D point cloud628. Photogrammetry can be performed as discussed inFIG.7and/or any suitable photogrammetric process may be performed. In one or more embodiments, photogrammetry may have been previously performed.

At block808, the software application604is configured to perform a 2D to 3D comparison which comprises comparing the selected 2D images (e.g., in database690) of the environment160to the 3D point cloud628of the environment. The reason for this process is to transfer the 2D segmentation information to 3D space. The result of this comparison is a 3D representation of an edge (or multiple edges). At block810, this 3D edge can be projected again to 2D but with a defined orthogonal view and defined scale. This results in the 2D sketch of the street edges. Corresponding structures in the 2D images can be found and matched to corresponding structures in the 3D point cloud. The image positions and orientations of the 2D images622, utilized to create the 3D point cloud628in the coordinate system, can be utilized for comparing the selected 2D images622that had street edges and marking detected to the 3D point cloud628, and subsequently matching common features (and their coordinates) between the selected 2D images622and the 3D point cloud628. Moreover, the coordinates (x, y coordinates) of the 2D images (particularly the selected 2D images) can be matched and integrated to the corresponding coordinates (x, y, z) in the 3D point cloud628, and vice versa. To integrate the selected 2D images (having street edges and other marking detected) in the 3D point cloud, various techniques can be utilized including forward projection, back projection, and back projection with segmented images to point cloud.

Forward projection includes finding matching line segments (or features) in multiple 2D images and then triangulating the line segments into the 3D point cloud. This method may be performed for features that can be described in a parametric way. Forward projection (with segmented images to point cloud) may further include projecting the road segments/edges of the drone (2D) images into the 3D point cloud and connecting all the edges to create a curve (straight line/arcs).

Back projection includes projecting 3D data from the photogrammetry 3D point cloud into the 2D images and attributing 3D points to identified lines or segments in the 2D images. This includes ambiguity free back projection strategies. Ambiguities (per 2D image) can arise when there are multiple 3D points (with different distances to the camera) projected into a single pixel. In an example case, this can be resolved by selecting the 3D point with the correct distance, which is typically the closest distance to the camera as only this object would be visible.

At block810(which may be in combination with block808), the software application604is configured to draw and insert a 2D sketch/drawing of the output data626as street edges (and street markings) in the 3D point cloud628, where the 2D sketch/drawings of the street edges is formed by drawing lines connecting the coordinates (x, y coordinates) of street edges drawing lines connecting other street markings in the 3D point cloud628. In some examples, in addition to using the coordinates saved in the output data626, the drawing/sketch of the lines occurs by at least partially following the segmentations that have segmented in, for example, the predicted images11B and12B, resulting in the example drawing/sketch depicted inFIG.15. Each segmented area, which is a segmented object, has a value and implicitly incorporates the edge. The drawing/sketch is around/on the edges of the segmented object using the values (x, y coordinates) around the segmented object.

For example, the software application604is configured to identify the lines/edges as borders between semantic segments (as seen inFIG.11B or12B). In 2D images, the lines are therefore a collection of pixel coordinates (x, y coordinates). After the 2D to 3D conversion, the coordinates of the lines/edges are a collection of 3D coordinates (x, y, z coordinates) in object space. In both the 2D image and 3D images, the software application604is configured to describe the edges (especially the street edges) by geometric primitives. This means, the software application604(using one or more algorithms) works to piecewise fit a line, an arc, a spline, or something similar to the points (2D points or 3D points) describing the edge. This result in a smooth single-pixel width description (in the 2D image) and/or single-3D-point width description (in the 3D image) of the identified edge. This type of filtering can be done in the initial images, in the 3D point cloud, or in the final projection of the 3D edge into a top down view 2D image (discussed further below). The later option would be the optimal choice.

It should be appreciated that the software application604is configured to transform the 3D point cloud into a top-down view as a 2D representation of the 3D point cloud. The 2D representation630is a 2D image of the 3D point cloud. The software application604can determine that the selected 2D images622correspond to region(s) of interest in the 3D point cloud, or vice versa. In one or more embodiments, a user may identify a region of interest in the 2D images and/or in the 3D point cloud. In the example scenario, the region of interest could be a crash scene, an intersection at which the crash occurred, etc. The software application604is configured to collapse 3D points/coordinates (e.g., x, y, z coordinates) of the 3D point cloud along a defined axis (e.g., z axis which can be height, elevation, etc.) to generate the 2D representation630. For example, from a top-down view into the 3D point cloud, the coordinates of the z-axis can be reduced to zero (0). There can be filtering performed prior to or after the 2D generation of the 2D representation630of the 3D point cloud.

In transforming the 3D point cloud into the top-down 2D representation630, the 3D structures in the 3D point cloud628can be smoothened. Each transferred 3D structure may contain noise like double structures, and a suitable averaging could be used in 3D point could or directly in the final top-view 2D representation630. Smoothing and refinement can be done by means of redundancy. For example, line A is projected into image1and image2(or even more images). When different 3D points are to be back projected into the lines in the different images, a mean line can be calculated. Each projected 3D point into a 2D image can be attributed with an error value which may be based on the 2D distance of the projected point to the nearest line segment. Points are searched which produce a global minimum error, for example, a point is taken as a line point in 3D when the square sum of errors is below a fixed threshold and has a minimum value in its 3D neighborhood. Not only real 3D points may be taken this way, but also interpolated points can be found. To get the 3D point in the 2D image, the software application604can use a dynamic threshold (for distance between pixel and line segment) and/or fixed dynamic threshold to ensure that the 3D point lies on the edge segment (e.g., street line) in the 2D image.

For illustrations purposes and not limitation,FIG.13illustrates an example 2D representation630of the 3D point cloud628without the drawing/sketch of the street edges and street markings.FIG.14illustrates the graphical display of example 2D representation630of the 3D point cloud628with the 2D drawing/sketch of street edges and street markings according to one or more embodiments. The 2D representation630of the 3D point cloud628with the drawing/sketch can be displayed in a graphical user interface of the software application604. As discussed herein, the software application604is configured to generate/draw/sketch street edges (and other street marking) for the streets and roadways in the selected (2D) images622, using the output data626including image positions and orientations of the large number of images622utilized during photogrammetry. In one or more embodiments, the software application604may include, call, and/or employ features of a drawing software tool.

FIG.15illustrates the display of the drawing/sketch of street edges and street markings without the 2D representation630of the 3D point cloud628according to one or more embodiments. As discussed herein, the software application604, for example, using and/or employing a drawing software tool, is configured to automatically draw lines, arcs, etc., to connect the coordinates (x, y coordinates) of the output data626in order to draw/sketch the street edges and street markings of streets and roadways.

FIG.17illustrates visual display of the automatic 2D sketch generation according to one or more embodiments. Particularly,FIG.17illustrates display of an example drone image in view1702, example segmented images in view1704, and an example overlay of a segmented image on its corresponding drone image in view1706. In view1706, a drawing/sketch of the street edges on the street/roadway has been generated according to one or more embodiments.

FIG.16is a flowchart of a computer-implemented method1600for generating street edges in 2D maps using photogrammetry according to one or more embodiments. The software application604, in conjunction with the photogrammetry software612, the machine learning model606(e.g., a neural network), and software drawing tools, executes the method1600. The software application604includes a graphical user interface (GUI) for displaying the images and the sketch/drawing on a display, such as the display519or any other suitable display screen.

At block1602, the software application604is configured to retrieve at least one selected image from a plurality of aerial images (e.g., 2D images in database690) of an environment (e.g., environment160), the at least one selected image comprising surface regions that are concurrently in a three-dimensional (3D) point cloud (e.g., 3D point cloud628) of the environment. In some cases, multiple 2D images may be combined using photogrammetry into a larger view of a 2D image. In some cases, multiple 2D images can be selected for processing.

At block1604, the software application604is configured to detect areas/objects (having edges) of the surface regions in the at least one selected image, such that coordinates of the edges of the surface regions are extracted from the at least one selected images. The software application604can call on and/or use the machine learning model606(having machine learning algorithms) for area/object detection, which results in output data626. The output data626includes segmented images, identified street lines (i.e., street edges), road marking, etc., as well as their coordinates (x, y coordinates).

At block1606, the software application604is configured to (respectively) compare the at least one selected image(s) to the 3D point cloud628to align common locations/features in both the selected image and the 3D point cloud. The comparison can use forward projection, back projection, etc.

At block1608, the software application604is configured to display (e.g., on display519) an integration of a drawing/sketch of the coordinates (e.g., 2D coordinates such as x, y coordinates) of the edges of the surface regions in a representation (e.g., 2D representation630) of the 3D point cloud628. An example of the drawing/sketch is depicted inFIG.15. The integration of the drawing/sketch in the 2D representation630is illustrated inFIG.14, with street edges, street lines, and street markings. An example of the 2D representation without the drawing/sketch is illustrated inFIG.13.

The 3D point cloud628is generated from the plurality of aerial images using photogrammetry. The at least one selected image(s) (e.g., images622) is selected from the plurality of aerial images (e.g., database690) having been used to generate the 3D point cloud628, for example, using photogrammetry. The edges of the surface regions (e.g., streets, roadways, etc.) are detected using machine learning (e.g., trained machine learning model606). Street lines and road markings can also be identified.

The plurality of aerial images are orthoimages. The coordinates of the edges of the surface regions are connected by lines to form the drawing of the edges, for example, as depicted inFIGS.14and15. Forward projection is utilized to integrate the drawing into the 3D point cloud628. Back projection is utilized to match 2D information from the at least one selected images to 3D data of the 3D point, and smoothing the identified line in 3D by finding 3D points for a line in which the 3D points have a minimal squared sum of a back projection error into a plurality of images. For example, during back projection, the software application604is configured to identify segments, edges, or similar structures in the 2D images and transfer this knowledge to the 3D data. 3D points of the 3D point cloud are back projected into a plurality of images. It may be noted that a selection of 3D points can be used which was identified as candidate 3D points belonging to the line or being close to the line. For each of these points an error is computed for each image which depends on the distance of the back projected 2D point to the 2D line identified in this image. As an example, this error can be calculated as the pixel distance between the back projected point and the line in 2D. Now, the candidate 3D points can be filtered in an optional step by applying a threshold which removes points above a defined distance from the list of candidate points. Due to noise and sampling related artefacts, it is not expected that a 3D point is being back projected into multiple images and having a minimum error calculated for each image. To identify a global optimal point, we identify these candidate points which have a minimum global error in a plurality of images. This global error can be defined as the squared sum of the individual errors calculated per image, but also more advanced error combinations can be used e.g. which neglect single large errors when they are calculated as a median value of the individual errors. This results in a smoothed line which is expected to be less impacted by noise or individual image artefacts. Additionally, new 3D points which are not part of the list of candidate 3D points can be found by interpolating between two or more existing 3D points and this additional 3D point has a smaller global reprojection error than the 3D data of the original 3D point cloud. This can result in an even smoother line in 3D. The drawing of the coordinates of the edges of the surface regions is displayed (e.g., on display519) in an orthographic view of the 3D point cloud628, for example, as depicted inFIG.14.

While embodiments of the invention have been described in detail in connection with only a limited number of embodiments, it should be readily understood that embodiments of the invention are not limited to such disclosed embodiments. Rather, embodiments of the invention can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the invention. Additionally, while various embodiments of the invention have been described, it is to be understood that aspects of the invention may include only some of the described embodiments. Accordingly, embodiments of the invention are not to be seen as limited by the foregoing description, but is only limited by the scope of the appended claims.