Method and apparatus for identifying a vibrometry spectrum in imaging applications

A method includes obtaining image data associated with a specified area having one or more objects. The method also includes segmenting the image data into one or more segments associated with the one or more objects. The method further includes analyzing each of the one or more segments to identify a vibrometry spectrum associated with the corresponding object. In addition, the method includes generating an image of the specified area using the vibrometry spectrum associated with each object. The image of the specified area could illustrate each of the one or more objects with an intensity based on a total power of that target's total vibrational energy. The image of the specified area could also illustrate movement of at least one object over time.

TECHNICAL FIELD

This disclosure is generally directed to image processing systems. More specifically, this disclosure relates to a method and apparatus for identifying a vibrometry spectrum in imaging applications.

BACKGROUND

Laser vibrometry typically involves capturing non-contact vibration measurements of an object's surface. In a conventional laser vibrometer, a laser beam is reflected off a surface of interest, and the frequency and amplitude of the surface's vibrations can be identified based on the Doppler shift of the reflected laser beam. However, conventional laser vibrometers typically require the use of a coherent micro-Doppler source and sensor in order to capture fine frequency measurements. Unfortunately, various Laser Detection and Ranging (LADAR) systems and other systems are able to provide only ranging (distance) information to a target's surface and are unable to capture such fine frequency measurements.

SUMMARY

This disclosure provides a method and apparatus for identifying a vibrometry spectrum in imaging applications.

In a first embodiment, a method includes obtaining image data associated with a specified area having one or more objects. The method also includes segmenting the image data into one or more segments associated with the one or more objects. The method further includes analyzing each of the one or more segments to identify a vibrometry spectrum associated with the corresponding object. In addition, the method includes generating an image of the specified area using the vibrometry spectrum associated with each object.

In a second embodiment, an apparatus includes at least one memory configured to store image data associated with a specified area having one or more objects. The apparatus also includes at least one processing unit configured to segment the image data into one or more segments associated with the one or more objects, analyze each of the one or more segments to identify a vibrometry spectrum associated with the corresponding object, and generate an image of the specified area using the vibrometry spectrum associated with each object.

In a third embodiment, a non-transitory computer readable medium embodies a computer program. The computer program includes computer readable program code for obtaining image data associated with a specified area having one or more objects. The computer program also includes computer readable program code for segmenting the image data into one or more segments associated with the one or more objects. The computer program further includes computer readable program code for analyzing each of the one or more segments to identify a vibrometry spectrum associated with the corresponding object. In addition, the computer program includes computer readable program code for generating an image of the specified area using the vibrometry spectrum associated with each object.

DETAILED DESCRIPTION

FIG. 1illustrates an example LADAR system100in accordance with this disclosure. As shown inFIG. 1, a laser platform102includes at least one laser that directs radiation towards a given area104and at least one sensory array that receives reflections of that radiation from the given area104. The platform102includes any suitable structure on which at least one laser and at least one sensory array can be placed. In this example, the platform102includes an airplane, although other platforms (such as a satellite, unmanned drone, or other vehicle) could be used.

The given area104represents any suitable area being scanned using the laser platform102. The given area104could include zero or more targets or other objects, such as vehicles. Any objects in the given area104may be visible or obscured, such as when an object is located under trees or other foliage or is otherwise camouflaged. The given area104can have any suitable size, shape, and dimensions and can represent an area in any given environment.

The laser platform102directs radiation towards the given area104and receives radiation reflected from the given area104. By performing calculations such as time-of-flight calculations, it is possible to obtain ranging information to any targets within the given area104. The ranging information could then be analyzed to identify a vibrometry spectrum of each target. The vibrometry spectrum defines the vibrational characteristics of a target. The vibrometry spectrum of a target could be used in various ways, such as to automatically identify whether an engine in a target vehicle is running. The analysis can be done by a processing system on the platform102itself, or the analysis could be done remotely, such as by an analysis system106. The analysis system106could receive measurement data or other data from the platform102in any suitable manner, such as via satellite or other wireless communications. The analysis system106includes any suitable computing or other data processing system that analyzes data and identifies vibrometry spectra for targets in the given area104.

FIGS. 2 through 5illustrate example components in the LADAR system100ofFIG. 1in accordance with this disclosure. In particular,FIGS. 2 through 4illustrate an example laser platform102and its components, andFIG. 5illustrates an example analysis system106. These systems are simplified here for ease of explanation.

As shown inFIG. 2, the laser platform102includes a LADAR subsystem202, which directs radiation towards the given area104and measures radiation reflected from the given area104. In this example, the LADAR subsystem202includes a laser source204and transmit optics206. The laser source204represents any suitable laser source generating illumination at a desired wavelength or in a desired wavelength range, such as a very narrow wavelength range. Depending on the implementation, the laser source204could generate radiation in the near infrared, visible, or ultraviolet spectrum. The transmit optics206include lenses, mirrors, or other suitable optical devices for directing radiation from the laser source204towards a target area, such as towards the given area104to be scanned.

The LADAR subsystem202also includes receive optics208and an optical detector210. The receive optics208include lenses, mirrors, or other suitable optical devices for directing radiation reflected from the target area, such as from the given area104, to the optical detector210. The optical detector210measures the radiation received by the LADAR subsystem202. The optical detector210includes any suitable structure for measuring radiation, such as an array of photodetectors.

In some embodiments, the subsystem202implements a Geiger-mode Avalanche Photodiode Detector (GmAPD) LADAR. In a conventional coherent or linear mode LADAR system, a laser source generates broad laser pulses and scans over a wide area (shown as the scan path108inFIG. 1). Also, a detector captures a single position on the ground per pulse and digitizes the returned pulse, possibly detecting multiple return pulses (such as from tree leaves and the ground). The waveform is thresholded to determine the time of the returns, and the range is calculated from the time of flight. This allows the system to determine where, for example, the tree leaves and ground are located.

In a GmAPD LADAR system, the laser source204generates narrow laser pulses, and a sensor array forming the detector210is over-biased so that the sensor array records the time of the first single photon detected for each pixel in the array. For every laser shot, the scan can capture a grid of points (shown inFIG. 1as a 4×5 grid110) per pulse. The same point can be illuminated multiple times. This allows multiple returns from the same ground post (the same position) using successive laser shots. For instance, the first shot may detect a photon reflected off tree leaves, while the next shot may detect a photon from a vehicle under the tree. This allows the possibility of foliage and camouflage net penetration.

Example components of a GmAPD LADAR system are shown inFIGS. 3 and 4. InFIG. 3, a single pixel300of the optical detector210in a GmAPD LADAR system is shown. The pixel300includes a photodiode302having an output coupled to an inverter304, which is coupled to a digital timing circuit306. The photodiode302is biased for Geiger-mode of operation, meaning the photodiode302is biased above its breakdown voltage. As a result, the photodiode302generates a charge avalanche upon generation of a single photoelectron. This means that the photodiode302triggers its output upon detecting a single photon from the given area104, thereby yielding single-photon counting sensitivity. The inverter304inverts the output of the photodiode302, and the digital timing circuit306generates a signal identifying the time-of-arrival of the photon. This results in the pixel300identifying the digital time of photon arrival with no amplifier noise.

Multiple pixels300can be combined to form a focal plane array400in the optical detector210as shown inFIG. 4. Any suitable number of pixels300could be used in the array400, such as a 32×32 array up to a 32×128 array (although other arrangements could be used). As shown inFIG. 4, the focal plane array400includes a lens array402, which includes multiple small lenses (often called “lenslets”) that focus incoming light. The focal plane array400also includes an Avalanche Photodiode Detector (APD) array404, which includes an array of the diodes302in the pixels300. A complimentary metal-oxide semiconductor (CMOS) array406implements various other components of the pixels300, such as the inverters304and digital timing circuits306.

Returning toFIG. 2, the laser source204in a GmAPD LADAR system often illuminates the entire area of sensor coverage and has a lower power than that utilized by a coherent or linear mode LADAR system. Because of the lower power, the probability of detection is not necessarily 100%. Also, stray light and sensor internal thermal noise known as dark current (which can cause the circuit to avalanche because of over-biasing) contribute noise to the overall system, which can be filtered out using “coincidence processing” or other suitable processing technique. Additional details regarding an example GmAPD LADAR system are found in Albota et al., “Three-dimensional imaging laser radar with a photon-counting avalanche photodiode array and microchip laser,” Applied Optics, Vol. 41, No. 36, 2002 (which is hereby incorporated by reference). Coincidence processing is a statistical method that determines if a single return point is noise or a true return by counting the number of points in fixed-sized voxels. Neighborhood coincidence processing also considers points in neighboring voxels.

Information about operation of the laser subsystem202(such as the timing of laser shots and the receipt of reflected photons) can be used in any suitable manner. For example, the information can be provided to at least one processing system212, stored in at least one memory device214, and/or communicated to at least one external device or system (such as the analysis system106) via at least one communication interface216. The processing system212could simply receive the information and pass it on to the external device or system via the communication interface216, or the processing system212could analyze the information (such as to generate vibrometry spectra for targets). The processing system212could perform any other suitable operations as needed or desired, such as authentication or encryption operations.

The processing system212includes any suitable processing or computing device(s) configured to process information, such as at least one microprocessor, microcontroller, digital signal processor, field programmable gate array, application-specific integrated circuit, or other device(s). The memory device214includes any suitable storage and retrieval device(s), such as a volatile and/or non-volatile memory. The communication interface216includes any suitable interface(s) configured to transmit or receive data, such as at least one wireless transceiver.

As shown inFIG. 5, the analysis system106includes at least one communication interface502, at least one memory device504, and at least one processing system506. Information from the laser platform102could be received via the communication interface502, stored in the memory device504, and analyzed by the processing system506. The processing system506could analyze the information to generate vibrometry spectra for targets in the scanned area, store the vibrometry spectra in the memory device504, communicate the vibrometry spectra to an external device or system, or use the vibrometry spectra (such as to generate a display of the scanned area).

The processing system506includes any suitable processing or computing device(s) configured to process information, such as at least one microprocessor, microcontroller, digital signal processor, field programmable gate array, application-specific integrated circuit, or other device(s). The memory device504includes any suitable storage and retrieval device(s), such as a volatile and/or non-volatile memory. The communication interface502includes any suitable interface(s) configured to transmit or receive data, such as at least one wireless transceiver.

AlthoughFIGS. 1 through 5illustrate one example of a LADAR system100and examples of components in the LADAR system100, various changes may be made toFIGS. 1 through 5. For example, as noted above, the laser platform102could include any other suitable platform vehicle. Also, information collected by the laser platform102could be analyzed by the processing system212on the platform102, the processing system506in the analysis system106, or any other suitable device or system. If processed on the platform102, the resulting vibrometry spectra could be used locally (such as to generate displays for an operator on the platform102) or communicated to an external device or system (with or without the underlying data). Similarly, if processed on the analysis system106, the resulting vibrometry spectra could be used locally (such as to generate displays for an operator in an analysis center) or communicated to an external device or system (with or without the underlying data). In addition, these figures represent one example environment where vibrometry spectra can be determined using an imaging system. This functionality could be used with any other suitable imaging system.

FIGS. 6 and 7illustrate example types of point cloud data used by the LADAR system ofFIG. 1in accordance with this disclosure. When describing the generation of vibrometry spectra below, various types of data can be used. In some embodiments (such as GmAPD systems), there are three “levels” of data that contain information about a scene. “Level 0” data refers to raw data associated with the laser platform102and laser pulses directed to and reflected from the given area104. Examples of Level 0 data can include platform position, platform Euler angles (roll, pitch, and yaw), azimuth and elevation pointing angles of a sensor, and round trip time of laser light. “Level 1” data represents computed X/Y/Z positions, frame numbers, and pixel numbers of laser returns, including both real returns and noise. An example of Level 1 data is shown inFIG. 6. As can be seen inFIG. 6, a raw point cloud600(Level 1 data) does not provide much useful information to an operator. “Level 2” data represents smoothed and de-noised point cloud data. An example of Level 2 data is shown inFIG. 7. As can be seen inFIG. 7, a processed point cloud700(Level 2 data) provides much more useful information to an operator, such as topographical information or objects of interest.

As noted above, conventional laser vibrometers often require the use of coherent micro-Doppler sources and sensors in order to capture fine frequency measurements. However, some ranging systems (such as GmAPD or other LADAR systems) provide only ranging information to a target's surface and cannot capture such fine frequency measurements. Described below are various techniques for obtaining vibrometry spectra for targets in a given area104using data from systems like GmAPD or photon-counting LADAR systems. The vibrometry spectra can be used in various ways, such as to determine whether targets such as vehicles, industrial equipment, HVAC (heating, ventilation, air conditioning) systems, and generators are vibrating and therefore operating. Additionally, successive spectra and instantaneous height changes of a vehicle as it moves down a road can be compared with an expected response based on the road's contours to determine whether that vehicle is loaded or empty. This supports the ability to gather phenomenological information from existing and future sensors that are based on range-only measurement.

FIG. 8illustrates an example method800for identifying and using a vibrometry spectrum in accordance with this disclosure. For ease of explanation, the method800is described as being performed by the processing system212in the platform102or the processing system506in the analysis system106. The method800could be performed by any other or additional processing systems in one or more locations.

Image data is obtained from an imaging system at step802. This could include, for example, the processing system212or506receiving measurement data obtained by the optical detector210. The measurements could represent measurements associated with the first photons received by each pixel of the optical detector210in a GmAPD system. Other data could also be obtained, such as an identification of the time-of-flight for a laser pulse from the laser source204to a target and back. The data may involve one or more targets that could be traveling or otherwise operating (and therefore vibrating).

The data is separated into segmented targets at step804. This could include, for example, the processing system212or506dividing the image data into different segments, where each segment is associated with a different potential target or other object. In some embodiments, the processing system212or506could perform volumetric change detection, automated data segmentation, or target tracking. In general, any suitable technique could be used to identify a potential target of interest. In particular embodiments, the segmentation is done using Level 2 data, and each segment represents a collection of Level 2 data associated with a potential target.

Signal processing is performed to generate a vibration spectrum for each target at step806, and the vibration spectra are analyzed at step808. In some embodiments, the signal processing could vary depending on the mode of operation for the processing system212or506. For example, the processing system212or506could operate in a map mode or a target mode. In map mode, the system100scans a given area104over a shorter period of time and attempts to identify targets in the area that are vibrating. In target mode, the system100scans the given area104over a longer length of time and identifies movement of an object as well as the object's vibrational characteristics.

The results of the analysis are used at step810. This could include, for example, the processing system212or506generating a map that graphically illustrates the given area104and that identifies any targets currently vibrating. The intensity of each target in the map could depend on the level of that target's vibrations. This could also include the processing system212or506generating a map that graphically illustrates the given area104and shows movement of a target over time. The results of the analysis could be used in any other suitable manner.

FIGS. 9 through 11illustrate a more specific example method and related details for identifying and using a vibrometry spectrum in a first (single scan) mode of operation in accordance with this disclosure. In particular,FIGS. 9 through 11illustrate a method900and related details for identifying and using a vibrometry spectrum in a map mode of operation. In this mode, the system100scans a given area104over a shorter period of time, generating data in one or a limited number of frames.

Image data is obtained from an imaging system at step902. This could include, for example, the processing system212or506receiving measurement data obtained by the optical detector210. The data is separated into segmented targets at step904. This could include, for example, the processing system212or506performing any suitable technique to identify potential targets of interest and segmenting the image data based on the identified targets. Each segment can represent part of a collection of Level 2 data.

Differences between each segmented target and their corresponding Level 1 data are identified at step906. This could include, for example, the processing system212or506identifying the difference between each point in a Level 2 point cloud's segment and the corresponding point in a Level 1 point cloud. The vibration characteristics of each target are identified at step908. This could include, for example, the processing system212or506using Parseval's Theorem, which defines the conservation of power between frequency and time domains. In this case, the differences between each segmented target and the corresponding Level 1 data are in the time domain, and step908involves converting that information into the frequency domain.

An intensity-based image of the targets is generated and presented (or used in other ways) at step910. This could include, for example, the processing system212or506generating a three-dimensional image of the given area104and any targets in the given area104. This could also include the processing system212or506illuminating different targets in the image with different intensities based on different vibration levels of the targets. In particular embodiments, the intensities of the targets can be determined as follows. The “vibrational energy” of each target can be calculated by summing the squares of the amplitudes of the target's spectral or time-domain vibration waveform and dividing the value by the length of the waveform. The resulting value represents the total power of the target's total vibrational energy. The intensity of each target can be based on the total power of that target's total vibrational energy.

An example of this is shown inFIGS. 10 and 11. InFIG. 10, an image1000represents a three-dimensional image of a given area104and any topographical features and targets in the given area104. InFIG. 11, an image1100also represents a three-dimensional image of the given area104and any topographical features and targets in the given area104. However, the image1100has a lower intensity overall compared to the image1000. Also, two targets1102a-1102bin the image1100are shown as having lower intensities, while two targets1104a-1104bin the image1100are shown as having higher intensities. Here, the system100has determined that the targets1104a-1104bare vibrating more than the targets1102a-1102b. In this way, the image1100is able to quickly convey to an operator which targets in a given area104are currently operating or otherwise vibrating. The system100could generate one or both images1000and1100. As a particular example, the system100could present the image1000to the operator and then, in response to input from the operator, generate the image1100. However, the system100could also generate the image1100without first generating the image1000.

FIGS. 12 and 13illustrate a more specific example method and related details for identifying and using a vibrometry spectrum in a second (multiple scan) mode of operation in accordance with this disclosure. More specifically,FIGS. 12 and 13illustrate a method1200and related details for identifying and using a vibrometry spectrum in a target mode of operation. In this mode, the system100scans the given area104over a longer period of time, generating data in a larger number of frames.

Image data is obtained from an imaging system at step1202. This could include, for example, the processing system212or506receiving measurement data obtained by the optical detector210. The image data is corrected for any line-of-sight issues at step1204. This could include, for example, the processing system212or506modifying Level 1 point cloud data based on the location of the platform102with respect to the given area104. The data is separated into segmented targets at step1206. This could include, for example, the processing system212or506performing any suitable technique to identify potential targets of interest and segmenting the image data based on the identified targets. Each segment can represent part of a collection of Level 2 data.

The system processes each target's image data at step1208and processes ground data at step1210. For each target, this could include the processing system212or506identifying the difference between each point in a Level 2 point cloud's segment and the corresponding point in a Level 1 point cloud. This could also include the processing system212or506subtracting a ground spot's data from each target's segment. For the ground, this could include the processing system212or506identifying the differences between the Level 2 cloud points and the corresponding Level 1 cloud points for the ground spot.

The image data is cropped at step1212. This could include, for example, the processing system212or506performing a histogram-guided cropping that filters data furthest from peak values in each target's image data using standard deviation. Image data is arranged by ascending frame at step1214, and vectors for any targets are concatenated at step1216. This could include, for example, the processing system212or506arranging X/Y/Z and pixel data by ascending frame and concatenating vectors for any moving vehicles or other targets. An average height of each target is identified using the frames at step1218, and any DC or other bias is subtracted from the identified heights at step1220.

The resulting data is plotted in the frequency domain at step1222. This could include, for example, the processing system212or506generating an FFT power spectrum. The frequency domain data can then be used in any suitable manner at step1224. This could include, for example, the processing system212or506generating an image that shows movement of a target object over time. An example of this is shown inFIG. 13, where an image1300shows a ground spot1302and multiple instances1304a-1304cof a target moving. The ground spot1302represents an area lacking any targets, and the frequency domain data represents the vibrometry spectrum for the object,

FIG. 14A and 14Billustrate example vibrometry spectra in accordance with this disclosure. In particular,FIG. 14Aillustrates the vibrometry spectra1400of two operating vehicles and the ground over a 4 kHz range, while

FIG. 14Billustrates a portion1402of that vibrometry spectra over a narrower 1 kHz range. Here, line1404represents the vibrometry spectrum of the ground, line1406represents the vibrometry spectrum of a first vehicle, and line1408represents the vibrometry spectrum of a second vehicle. As shown inFIGS. 14A and 14B, the vibrometry spectra1400show that there is spectral content evident for the operating vehicles that is absent for the ground. These vibrometry spectra1400therefore clearly show that vibrational characteristics of a target can be determined using the frequency domain data captured using a GmAPD or other ranging system.

AlthoughFIGS. 6 through 14Billustrate example details of identifying and using a vibrometry spectrum, various changes may be made to

FIGS. 6 through 14B. For example, the point cloud data shown inFIGS. 6 and 7and the images shown inFIGS. 10,11, and13are for illustration only. Also, whileFIGS. 8,9, and12show example methods having serial steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In addition, the example spectra shown inFIGS. 14A and 14Bare for illustration only.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer code (including source code, object code, or executable code). The terms “transmit” and “receive,” as well as derivatives thereof, encompass both direct and indirect communication. The term “obtain” and its derivatives refer to any acquisition of data or other tangible or intangible item, whether acquired from an external source or internally (such as through internal generation of the item). The terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation. The term “obtain” and its derivatives refer to any acquisition of data or other tangible or intangible item, whether acquired from an external source or internally (such as through internal generation of the item). The term “or” is inclusive, meaning and/or. The phrase “associated with,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like. The phrase “at least one of,” when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list may be needed. For example, “at least one of: A, B, and C” includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.