Patent ID: 12260518

DETAILED DESCRIPTION

The aspects and features of the present invention summarized above can be embodied in various forms. The following description shows, by way of illustration, combinations and configurations in which the aspects and features can be put into practice. It is understood that the described aspects, features, and/or embodiments are merely examples, and that one skilled in the art may utilize other aspects, features, and/or embodiments or make structural and functional modifications without departing from the scope of the present disclosure.

For example, although the present invention is often described below in the context of imaging through fog, one skilled in the art will readily understand that the present invention can be applied to imaging through other dynamic scattering media such as water, rain, or aerosols, which can reduce transmission and/or scatter light in a time-dependent manner.

As noted above, imaging an object through fog presents significant difficulties, even without turbulence in the fog, with the fog often being responsible for image degradation over long distances. The present invention provides a method that uses a single-pixel camera or detector to reconstruct images of objects obscured by fog.

One prominent issue with imaging through fog using single-pixel cameras (SPCs) is the temporal variation of the detected signal that occurs due to unavoidable changes in the fog density with time. These changes can be confused with temporal variations in the detected signal associated with changes in the target reflectivity as the structure on the illumination beam is varied, thus reducing the fidelity of the reconstructed image. Another issue is the presence of light scattered by the fog itself, which can saturate the detector or add dynamic and shot noise.

The present invention provides a method for imaging objects through a dynamic scattering medium such as fog that mitigates the effects of temporal variations of the signal due to variation in the medium, e.g., fog density. It is based on the observation that the time scale of the intensity variations of the gated detector due to changes in fog density is much slower than the time scale for the variations due to changes in the structured beam illumination. As a result, the effects of fog density variations can be suppressed by using a combination of fast pulses, a fast detector and a high-pass filter in the detection system. As described in more detail below, the method of the present invention has been used to successfully reconstruct an image of the target in the presence of fog using both computational ghost imaging and compressive sensing which has the potential for shorter acquisition times.

The imaging technique in accordance with the present invention uses high-pass filters similar to the technique described in Zhang et al., supra. However, in Zhang et al., the source of the background fluctuations was external to the illumination system and can be mitigated by other means, such as optical narrowband filtering or use of a reference detector. In contrast, the temporal fluctuations in our application are introduced directly onto the returned signal, making it impossible to compensate for them with a simple reference detector.

As described in more detail below, the present invention provides an apparatus setup and a technique for mitigating the temporal variations of the signals received by a time-gated detector due to variation in the fog density. The imaging technique in accordance with the present invention is based on the observation that the time scale of the intensity variations of the signals due to changes in fog density is much slower than the time scale for the variations due to changes in the structured beam illumination. As a result, it becomes possible to suppress the effects of fog density variations by using short pulses, a fast detector and a high-pass filter in the detection system. Using this technique, we have demonstrated that the image of the target can be successfully reconstructed in the presence of fog using both computational ghost imaging and compressive sensing, which has the potential for shorter acquisition times.

The block schematic inFIG.1illustrates an exemplary apparatus setup and aspects of the method for single-pixel imaging in accordance with the present invention.

As shown inFIG.1, an exemplary apparatus setup in accordance with the present invention can include an illuminator100, a receiver200and a data processor/controller300. The illuminator can include a pulsed laser source101, a spatial light modulator (SLM)103, a beam splitter104and a reference detector105. The receiver can include a collecting optic system201(shown here as a simple lens), a fast “bucket” (non-imaging) detector202and a fast storage scope203. Data processor/controller300can include a computer programmed with appropriate software and having sufficient bandwidth and memory to control the apparatus described herein; process and analyze the data, including applying a high-pass filter as described below; and implement the algorithms necessary to reconstruct the image.

In accordance with the present invention and with reference to the exemplary apparatus illustrated inFIG.1a fast pulse illumination signal102is formed by a pulsed laser101. Laser101can be any suitable pulsed laser source that produces pulses typically from less than one nanosecond to a few nanoseconds in duration. For example, one exemplary laser system that can be used for imaging through fog is Bright Solutions SRL wedge XF 532 that produces 10 μJ 532-nm sub-nanosecond pulses at a repetition rate of 10 kHz.

The range of laser parameters such as pulse duration, pulse energy and repetition rate and the frame rate of the SLM are driven by a combination of the system requirements and the limits of the various technological capabilities. In general, the system requirement is for acquisition of an image with a specified minimum spatial resolution and field of view (FOV), which can give the number of resolution elements in the image. These parameters affect the number of elements in the SLM, with more elements allowing sharper image resolution or larger FOV. SLM's with 1920×1200 elements are generally applicable to a wide range of applications.

The laser pulse duration is affected by the requirement for sufficient range resolution and the ability to suppress scattering from regions of the perturbing medium or other potential targets located at distances that are different from the distance of the desired target. Generally, pulse durations ranging from sub nanoseconds to a few nanoseconds (2-15 nsec) are sufficient for most applications.

The SLM framing rate determines the time required for image acquisition, and higher rates allow faster acquisition times. In addition, the SLM framing rate should be faster than the time scale of the dynamic fluctuations of the scattering medium with the upper limit would generally be set by available technology. Depending on the dynamics of the scattering medium, SLM framing rates can vary from tens of Hz to tens of kHz. Phase SLM's can have frame rates up to 40 Hz, while DMD SLM's can have rates up to 40 kHz.

The laser pulse repetition rate can be matched to the SLM framing rate or can be higher to allow averaging of multiple laser pulses for a given spatial pattern. Higher SLM framing rates, along with higher laser repetition rates allow faster image acquisition times.

The pulse energy of the laser is determined by the requirement of having sufficient received energy to be detected by the bucket detector in a single pulse. Higher pulse energies generally allow imaging at longer distances or through a denser scattering medium. However, higher pulse energies at a given pulse repetition rate lead to a higher average power requirement for the laser, which is usually limited by available laser power. Pulse energies of the order of 10 μJ can be sufficient, with higher energies allowing faster acquisition times or longer ranges.

The range of laser wavelengths is determined by the needs of the specific application, along with the availability of lasers with the combination of parameters as discussed above. For example, imaging through water would generally require laser wavelengths in the blue green region of the spectrum because of the spectral absorption of water. Imaging through air can accommodate longer wavelengths, ranging from the visible into the infrared (0.5 μm up to 1.5 μm or longer). Longer wavelengths are associated with less scattering in the medium, allowing longer ranges. However, a lack of availability of suitable detectors at the longer wavelengths can limit the wavelength extent in the infrared.

The illumination beam produced by fast pulse102is directed into a spatial light modulator (SLM)103which imposes a pseudo-random pattern on the illumination beam. In some embodiments, SLM103can be a phase SLM that produces random phase patterns on the illumination beam, while in other embodiments, such as described below with respect toFIG.9andFIG.10, it can be a digital micro-mirror device (DMD) with an imaging lens that produces intensity patterns in the beams. In some embodiments, the patterns imposed on the beam by SLM103can be generated during the imaging process, while in other embodiments, the patterns can be pre-calculated, stored in data processor300, and then transferred into memory in SLM103for faster operation.

Beam splitter104directs a portion of the illumination light onto a reference detector105. Reference detector105defines the initial time 0 and therefore the distance for pulses that can fluctuate in time. For lasers that have a very stable pulse generation time with provided trigger signals or ones that can be externally triggered, time 0 can come from that trigger. The time from the trigger to the chosen digital time gate can be converted to distance.

The signal from reference detector105is then fed into fast storage scope203in order to normalize laser pulse to pulse energy fluctuations and to trigger the scope, setting the zero reference time for subsequent selection of target ranges. The illumination beam then propagates through the scattering medium such as fog region400towards a target object600to be imaged, with the imposed pseudo-random phase pattern produced by SLM103causing the illumination beam to develop a speckled intensity distribution such as speckle pattern500in illuminated area600as it hits the target object601. In accordance with the present invention, the patterns imposed on illumination beam by SLM103and the corresponding speckle patterns on the target are updated at a rate of, e.g., 40 Hz commensurate with the considerations given earlier.

The illumination beam with the speckle pattern is then reflected off target601, propagates back through fog region400, and is focused by collecting optics201onto a fast single-pixel “bucket” detector202. Collecting optics201can include any suitable components, such as a single lens or mirror or a system containing multiple lenses or mirrors or a combination of both mirrors and lenses, that can focus the light onto the bucket detector. Bucket detector202then converts the detected photons from the reflected beams into an electrical signal that is recorded on a fast storage scope203.

Fast storage scope203is triggered by a reference detector and stores a time trace of the detected signal for each speckle pattern. The stored time traces are digitally gated to record a single value for each trace at a particular time corresponding to the distance of the object of interest from the reference detector and discarding the signals due to the fog or other potential targets that are located at different distances and hence occur at different times. Since the speckle patterns are produced sequentially in time, the gated bucket signals then are correspondingly stored as a time sequence.

Fast storage scope203also receives and digitizes the signal from reference detector105. The digital data from both bucket detector202and reference detector105can then be input directly into a data processor such as data processor/controller300coupled to the signal receiving apparatus or can be input into a data storage device which can then be used to input the signal data into data processor/controller300. In either case, the digital signals are passed to data processor/controller300for analysis, where the data processor normalizes the digital signal from bucket detector202according to the signal from the reference detector105and applies a digital high-pass filter to remove the slower signal variations produced by dynamic changes in fog density over time. The filtered bucket values are then used together with their corresponding speckle patterns to generate the images using any appropriate reconstruction algorithm such as CGI or CSI.

EXAMPLE

In experiments to demonstrate the efficacy and utility of the present invention, the inventors at the Naval Research Laboratory (NRL) used as the target601a cutout of the letters “NRL” on white paper, placed just outside a 4-foot-long fog chamber forming a fog region400. In these experiments the fog chamber was in the form of an 8×10×48-inch oblong chamber fabricated using Lexan® polycarbonate sheets, Anti-reflective coated optical windows were used to control the beam path and minimize optical loss as the light travels through the chamber. The fog was generated by a cyclone ultrasonic fogger, with a ball valve being used to regulate the rate of introduction of the fog. The fog entered the chamber through holes equidistantly spaced in a polyvinyl chloride (PVC) pipe extending the length of the fog chamber. This technique results in an approximately uniform fog distribution throughout the entire chamber.

A Thorlabs® DET10A2 bucket detector was used to collect backscattered light from the target, while a PicoScope® 6407 Digitizer was used to collect the multiple traces from the bucket detector. Gated values of each trace, corresponding to the distance to the object, were used together with the stored speckle patterns for reconstruction of the image (NRL) with computational ghost imaging (CGI) and compressive sensing imaging (CSI).

One hundred (100) laser shots were averaged for each speckle pattern on the target to further reduce laser fluctuations. The double pass transmission of light in the fog was exp(−6), measured using a separate 532 nm continuous wave (CW) laser propagating through the fog chamber (not shown in figure). Higher attenuations could be used in principle, but under those conditions the fog scatter near the target exceeded the dynamic range of our digital oscilloscope.

An example of a typical time signal obtained without fog (solid curve) along with a reference signal (dashed curve) for normalization and range is shown inFIG.2. The recorded reference signal is used to normalize the pulse to pulse laser fluctuations that are present in all lasers and can negatively impact image reconstruction. Collecting the full trace enables the ability to differentiate and reconstruct multiple targets located at different distances, providing a potential of three-dimensional scene reconstruction.

The image is reconstructed using the SLM and a single-pixel (bucket) detector. The SLM is used to project unique 2D intensity patterns Ii(x, y) onto the object and the reflection intensity is measured using a single-pixel detector by means of the relation
bi=∫∫Ii(x,y)O(x,y)dxdy(1)
where biis the peak of the ith integrated time signal and O(x,y) is the reflection function of the object/target. Provided that the bucket detector is collecting the scattered light from the object illuminated by the 2D intensity pattern, biis effectively a time-gated weighting factor for each unique intensity pattern in the image reconstruction algorithm.

To reconstruct the object's reflection function using CGI, the 2D intensity patterns are weighted with the bucket detector measurements, as described in O. Katz, et al., “Compressive ghost imaging,”Appl. Phys. Lett.95, 131110 (2009), i.e.,
O(x,y)=(bi−b)Ii(x,y)(2)
where⋅=1/M Σ⋅ denotes an ensemble average over all the measurements M received by the bucket detector.

An alternative algorithm that uses a reference detector signal has also been employed for an improvement in the image reconstruction, as described in F. Ferri, et al., “Differential Ghost Imaging,”Phys. Rev. Lett.104, 253603 (2010), i.e.,
OCGI(x,y)=(bi−b/rri)(Ii(x,y)−I(x,y))(3)
where riis the reference detector measurement.

For analysis using compressive sensing image reconstruction (CSI), TVAL3, a total variation (TV) minimization solver is often used because of its speed and robustness in the presence of noise, though any suitable minimization solver can be used as appropriate. See C. Li, et al., “An efficient augmented Lagrangian method with applications to total variation minimization,”Comput. Optim. Appl.56, 507-530 (2013).

Such CSI reconstruction is performed by solving for the object image using the model

minOCSI∑iDi⁢OCSI+μ2⁢AOCSI-b22(4)
where DiOCSIis the discrete gradient, or total variation, of the reconstructed image OCSIat pixel i, A is the measurement matrix (speckle realizations), and b is the compressed signal, or bucket, value. The parameter μ is the penalty parameter that is adjusted to compensate for the noise in the bucket values and the sparsity level of the reconstructed image OCSI.

A comparison of image reconstruction using CGI vs. CSI is shown by the images inFIGS.3A-3F, whereFIGS.3A-3Care images reconstructed using CGI with 1000, 2000, and 3000 speckle patterns, respectively, andFIGS.3D-3Fwere reconstructed from the same number of speckle patterns using CSI. Both techniques are able to resolve the NRL target, but the CSI reconstruction using TVAL3 shows an improvement in contrast while preserving the edges of the image. It is worth noting that the image can be adequately reconstructed with CSI using fewer speckle realizations, providing the potential for faster data acquisition at the expense of computation time.

Measurements with Fog

The images inFIGS.4A-4Cshow the results of CGI image reconstruction of the target object (FIG.4A) obtained by means of Equation (3) using 4000 speckle patterns.FIG.4Bshows a reconstruction of the image in a fog-free environment; although the target is not completely resolved, the NRL letters are distinguishable. However, as can be seen fromFIG.4C, reconstruction of the image through fog using the same approach fails, with the image being totally blurred. This demonstrates that conventional reconstruction techniques fail in presence of fog and that the modified new approach in accordance with the present invention therefore is needed.

A potential source of the failed reconstruction is the additional temporal fluctuations in the bucket signal introduced by the fog that are not associated with the changing SLM patterns. The time-varying signal for CW laser light transmitted through the fog chamber is shown as solid curve inFIG.5along with the bucket detector measurements used without fog, shown inFIG.5by the pattern of circles.

It can be seen that the transmitted laser intensity displays peak-to-peak variations of up to 30%. This does not present significant issues for single-shot or multi-shot focal plane array detectors. However, structured light imaging relies on small changes in the detected light for multiple single detector exposures. These desired fluctuations are comparatively small, on the order of 1%. The ability to reconstruct an image using structured light will be severely degraded by any noise added to the bucket detector measurements. It can be expected that the additional noise fluctuations due to fog shown inFIG.5will render reconstruction impossible with the usual techniques.

However, it is also evident fromFIG.5that the fog-induced power fluctuations occur on a significantly slower time scale than the variations associated with the changing speckle patterns. This is confirmed by examining the power spectrum of the signals as shown inFIG.6.

The bucket detector measurements were taken at 40 Hz, with a Nyquist frequency of 20 Hz. As can be seen inFIG.6, most of the fog noise appears below ˜6 Hz. It can be expected that high-pass filtering of the bucket signal through the fog will remove most of the noise and only some of the needed bucket fluctuations, enabling reconstruction of the structured light even in the presence of fog. A successful demonstration of this technique is given in the next section. A similar situation was described in Zhang et al., supra, where slow fluctuations in the background intensity were shown to degrade the image reconstruction. As mentioned earlier, a high-pass filter was used in Zhang et al. to retrieve the reconstructed image. However, as also mentioned earlier, the fluctuations in our systems arise from variation in the transmission of the signal through the fog, not as fluctuations in background light that are not associated with the signal.

Mitigation of Fog Fluctuations

As noted above, to eliminate fog induced fluctuations, the method of the present invention uses a digital high-pass filter on the bucket signals processed by the data processor/controller. The computational nature of both CGI and CSI allows the use of different filters on the same data set, since the bucket values for imaging with and without fog are stored in a data processor such as data processor/controller300together with the corresponding speckle patterns. We can compare the high-pass filtered bucket values with fog to those measured without fog to determine how well this technique works. Of course this approach cannot yield a reconstruction completely identical to the one without fog since some functional lower frequencies are also removed. To compare the effect of different filters we employed a Pearson correlation coefficient between curves G and H given by

r⁡(x,y)=∑(x-x_)⁢(y-y_)∑(x-x_)2⁢(y-y_)2(5)
wherexis the mean of x andyis the mean of y. See K. Pearson, “Mathematical contributions to the theory of evolution—iii. regression, heredity, and panmixia,”Philos. Transactions Royal Soc. London. Ser. A, containing papers a mathematical or physical character pp. 253-318 (1896); see also J. Taylor,Introduction to error analysis, the study of uncertainties in physical measurements(University Science Books, 1997), 2nd ed., pp. 216-217. Here, r(x,y)=1 for perfect correlation.

FIG.7shows the Pearson correlation coefficient for Heaviside, exponential and forward-backward Butterworth filters in frequency space as a function of the cut-off frequency.

The exponential filter, given by

filt⁡(f,fcut-off,n)=1-exp⁡(-(ffcut-off)n)
has its best result for n=2 and is only slightly better than the Heaviside filter at its peak.

FIGS.8A-8Fare images comparing the results of CGI (FIGS.8A-8C) and CSI

(FIGS.8D-8F) reconstructions of the target object using 4000 speckle patterns without fog (FIGS.8A/8D), with fog (FIGS.8B/8E), and using the Butterworth high-pass filtered bucket values with fog (FIGS.8C/8F. Unsurprisingly, both the CGI and CSI reconstructions fail for the case with the fog bucket values without the high-pass filter. Even though fog renders reconstruction impossible without filtering, applying the high-pass filtering to the bucket values results in a successful reconstruction for both CGI and CSI, as shown by the images inFIGS.8C and8F. The contrast and SNR of the image reconstructed using CSI is significantly enhanced compared to the CGI method.

Advantages and New Features

The new feature of this invention is the combination of a fast single-pixel detector, the use of short pulses for time gating, a reference detector for normalizing fluctuations of the illumination intensity and a high-pass digital filter to suppress temporal fluctuations in the signal intensity due to variations in fog density. Individually, many of these techniques are known in other disciplines. For example, time gating, either by gating the detector or pulsing the source, is a known technique for suppressing return signals from near structures in range detection. Similarly, CGI and CSI with single-pixel bucket detectors have been described earlier. However, as demonstrated in our experiments (FIG.4), application of these techniques as described in the literature without the addition of the high-pass digital filter was totally ineffective in obtaining images through fog.

Similarly, the use of a digital high-pass filter for suppressing temporal variations of background environmental illumination was described in Zhang et al., “Digital filtering ghost imaging to remove light disturbances,”Appl. Opt.60, 809 (2021). However, Zhang does not teach the use of a normalizing reference detector to suppress variations in the laser illumination signal during the signal acquisition time in combination with the digital filter as is used in the present invention. Nor do they speak to the advantage of the combination for suppression of temporal variations impressed on the signal itself due to fluctuations of the properties of the propagation medium.

It is the combination of these elements that enables the use of single-pixel detectors to obtain images through obscuring media such as fog that may involve naturally occurring temporal fluctuations. The single-pixel detectors are considerably less expensive than two dimensional imaging detectors. In addition, sensitive two dimensional cameras are not available at all wavelengths, especially infrared wavelengths that may be advantageous for imaging through fog.

Thus, the present invention, which uses time gating and high-pass filtering of the values of signals received by a bucket detector can enable single-pixel structured image reconstruction in a foggy environment. While the method of the present invention has been demonstrated in a laboratory environment, it is expected that fluctuations due to fog in the field environment will occur on an even longer time scale, which should allow even better image reconstruction using the method of the present invention since fewer useful bucket values will be filtered out. In addition, use of an orthogonal basis for projection, such as Hadamard patterns, can also help improve the efficacy and convergence of this technique for both CGI and CSI.

Alternatives

Although particular embodiments, aspects, and features have been described and illustrated, one skilled in the art would readily appreciate that the invention described herein is not limited to only those embodiments, aspects, and features but also contemplates any and all modifications and alternative embodiments that are within the spirit and scope of the underlying invention described and claimed herein.

For example, in cases where imaging is to be done through water, the initial illumination beam can have wavelengths of 430 nm-550 nm because water has a transmission window in the blue-green spectral region. Laser repetition rates can vary from 100 Hz to 300 kHz and the cut-off frequency of the high pass filter can vary from 40 Hz to 500 Hz depending on the dynamics of the scattering medium. For a medium that fluctuates faster the repetition rate of the laser and cut-off frequency will be higher.

In some alternative embodiments, different high-pass filters such as Heaviside, Exponential or Butterworth filters or different illumination patterns, e. g., Hadamard patterns, can be used.

For example, in some alternative embodiments, a digital micro-mirror device (DMD) can be used as an SLM in place of a phase SLM as described above, where the DMD SLM can be implemented as part of illuminator100as shown inFIG.9or as part of the receiver200, as shown inFIG.10.

When arranged as part of the illuminator as shown inFIG.9, DMD SLM103ais used in conjunction with an imaging lens103b. The DMD SLM103aimposes pseudo random intensity patterns, e. g., a series of Hadamard patterns, on the illumination beam. The illumination beam with the imposed patterns are then passed through imaging lens103band onto the target object601in a manner similar to that described above with respect toFIG.1. In addition, as described above with respect toFIG.1, part of the illumination beam with the imposed intensity patterns is also directed into fast storage scope203via reference detector105for use by data controller/processor300in constructing the image of the target. The light from the illumination beam is reflected back from the target, collected by collecting optics201, detected by bucket detector202, and processed by data processor/controller300in a manner as described above with respect toFIG.1.

When the DMD SLM is arranged as part of the receiver (FIG.10) the illumination pulse102from laser101is transmitted directly to the target, either directly or with a mirror106that replaces SLM103, to produce a uniform illumination pattern. In this embodiment, receiver200also contains imaging optics204that form an image of the target600at DMD SLM205. Light from the illumination beam is reflected from target601and is received by imaging optics204and then directed into DMD SLM205which imposes an intensity pattern on the beam. The beam with the imposed pattern is then collected by collecting optics201, such as a lens as described above, using optional mirror206if necessary to direct the beam into the optics system, and is then detected by bucket detector202and processed by data processor/controller300in a manner as described above with respect toFIG.1.

In some embodiments, one or more of the reference detector or the bucket detector can be time-gated electronically. In other embodiments, multiple detectors for one or more of these elements can be used.

The present application contemplates these and any and all other modifications within the spirit and scope of the underlying invention described and claimed herein, and all such modifications and alternative embodiments are deemed to be within the scope and spirit of the present disclosure.