Single-shot compressed optical-streaking ultra-high-speed photography method and system

A system and a method for single-shot compressed optical-streaking ultra-high-speed imaging, the system comprising a spatial encoding module spatially encoding the transient event with a binary pseudo-random pattern into spatially encoded frames; a galvanometer scanner temporally shearing the spatially encoded frames; and a CMOS camera receiving the temporally sheared spatially encoded frames, during one exposure time of the camera, for reconstructing the transient event. The method comprises spatial encoding a transient event; temporal shearing resulting spatially encoded frames of the event, spatio-temporal integration, and reconstruction.

FIELD OF THE INVENTION

The present invention relates to imaging methods. More specifically, the present invention is concerned with a single-shot compressed optical-streaking ultra-high-speed imaging method and system.

BACKGROUND OF THE INVENTION

Single-shot ultra-high-speed imaging methods can be generally categorized into active-detection and passive-detection methods. The active-detection methods use specially designed pulse trains to probe 2D transient events, such as (x,y) frames that vary in time, and include frequency-dividing imaging and time-stretching imaging. Such methods are not suitable for imaging self-luminescent and color-selective events. By contrast, the passive-detection methods use receive-only ultra-high-speed detectors, such as rotatory-mirror-based cameras, beam-splitting-based framing cameras, in-situ storage image sensor CCD (charge-coupled device) cameras, and global shutter stacked CMOS (complementary metal oxide semiconductor) cameras for example, to record photons scattered and emitted from transient scenes. Such cameras either have a bulky and complicated structure or have a limited sequence depth, defined as the number of frames in one acquisition, and pixel count, defined as the number of pixels per frame.

To circumvent these drawbacks, computational imaging methods, combining physical data acquisition and numerical image reconstruction, were increasingly featured in recent years. In particular, the implementation of compressed sensing (CS) for spatial and/or temporal multiplexing has allowed overcoming the speed limit with a substantial improvement in the sequence depth and pixel count. Representative methods in computational imaging methods include programmable pixel compressive camera (P2C2), coded aperture compressive temporal imaging (CACTI), and multiple-aperture (MA)-CS CMOS camera. However, despite reaching over one megapixel per frame, the imaging speeds of P2C2 and CACTI, inherently limited by the refreshing rate of spatial light modulation and the moving speed of a piezoelectric stage, are limited at several thousand frames per second (fps), typically to kfps. MA-CS CMOS, despite ultra-high-speed imaging speeds, has a pixel count limited to 64×108 with a sequence depth of 32. Thus, existing computational imaging methods still fail to simultaneously combine high frame rates, sequence depth, and pixel count for ultra-high-speed imaging.

There is still a need in the art for a ultra-high-speed imaging method and system.

SUMMARY OF THE INVENTION

More specifically, in accordance with the present invention, there is provided a system for single-shot compressed optical-streaking ultra-high-speed imaging, comprising a spatial encoding module; a galvanometer scanner; and a CMOS camera, wherein the spatial encoding module is configured for spatially encoding the transient event with a binary pseudo-random pattern, yielding spatially encoded frames, the galvanometer scanner temporally shearing the spatially encoded frames of the transient event, and the CMOS camera receiving the temporally sheared spatially encoded frames, in one exposure of the camera, for reconstructing the transient event.

There is further provided a method for single-shot compressed optical-streaking ultra-high-speed imaging, comprising spatial encoding a transient event; temporal shearing resulting spatially encoded frames of the event, spatio-temporal integration, and reconstruction.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In a nutshell, a method according to an aspect of the present disclosure combines compressed sensing with optical streak imaging. The method comprises spatially encoding each temporal frame of a scene by compressed sensing using a spatial encoding module, thereby labeling the capture time of each frame. Then the method comprises temporal shearing in the temporal domain, using a temporal encoding module, thereby creating an optical streak image, capturing this streak image with an array detector in a single shot, and obtaining the temporal properties of light from this streak image. The mixture of 2D space and time data in the streak image can be processed to separate the data using reconstruction on the basis of the unique labels attached to each temporal frame.

A system10according to an embodiment of aspect of the present disclosure is illustrated inFIG.1A.

The system10comprises a spatial encoding module12. The spatial encoding module12is a spatial light modulator such as digital micromirror device (DMD), AJD-4500, Ajile Light Industries for example, on which a binary pseudo-random pattern is loaded with an encoding pixel size of 32.4×32.4 μm2. Alternatively, the spatial encoding module12may be a printed physical mask with an encoding pattern for example. The spatial encoding module12has a fixed ±12° flipping angle and about 1 Mega pixel count.

A transient scene is first imaged into the spatial encoding module12, where it is spatially encoded by the binary pseudo-random pattern. Resulting spatially encoded frames (c) are then relayed by a 4fsystem onto a CMOS camera14for detection. The CMOS camera14may be a cell phone, a CCD or a CMOS GS3-U3-23S6M-C, FLIR for example, with a frame rate per second in a range between 1 and 160, for example between 15 and 25, and a Mega pixel count.

A galvanometer scanner16, placed at the Fourier plane of the 4fsystem, temporally shears (so) the spatially encoded frames linearly to different spatial locations along the x axis of the camera14according to their time of arrival. The galvanometer scanner16may be a GS, 6220H, Cambridge Technology, for example. The galvanometer scanner16is selected with a rotation frequency per second in a range between 1 and 160, for example between 15 and 25, and a small angle step response, typically 200 μs.

The image is optically relayed from the transient scene to the CMOS camera14, by an optical relay module. Four achromatic lens and a mirror are shown; Lenses1and4are 75-mm focal length achromatic lenses with 1 inch diameter and Lenses2and3are 100-mm focal length achromatic lenses with 2 inch diameter, such as Thorlabs AC508-100, AC508-075, and the Mirror may be Thorlabs PF10-03-P01 for example. Alternatively, a camera lens with selected focal length and diameter may be used.

As a result of the spatial encoding of the frames by the binary pseudo-random pattern loaded on the DMD12, the N frames taken in a single exposure during the exposure time t of the camera14as allowed by rotation of the scanner16are ordered. The synchronization between the rotation of the galvanometer scanner16and the exposure of the camera14is controlled by a sinusoidal signal (tg) and a rectangular signal (te) generated by a function generator (not shown) as shown inFIG.1B. The function generator may be DG1022z, RIGOL TECHNOLOGIES, INC for example.

Finally, via spatiotemporal integration (T), the camera14compressively records the spatially encoded and temporally sheared scene as a 2D streak image E with a single exposure.

The operation of the system can be described by the following relation:
E=TSoCI(x,y,t),  (1)
where I(x,y,t) is the light intensity of the transient event, c represents spatial encoding by the DMD12, sorepresents linearly temporal shearing by the scanner16with the subscript “o” standing for “optical”, and T represents spatiotemporal integration by the camera14.

With prior knowledge or assumptions about the signal, including for example parameters of the encoding pattern, measured streak image, and physical forward operators such as spatial encoding, temporal shearing, and integration, and with the spatiotemporal sparsity of the scene, the light intensity of the transient event I(x,y,t) can be recovered from the measurement of the 2D streak image E by solving the inverse problem using compressed sensing reconstruction as follows:

I^=argminI⁢{E-TSo⁢CI22+λϕTV⁡(I)}(2)
where ∥⋅∥22represents the l2norm, λ is a weighting coefficient, and ΦTVis total variation (TV) regularizer. In experiments described hereinbelow, I(x,y,t) was recovered by using a compressed sensing-based algorithm developed upon a two-step iterative shrinkage/thresholding algorithm.

To obtain a linearly temporal shearing by the scanner16, the linear rotation of the galvanometer scanner16and the exposure of the camera14need be synchronized: a static target is placed at the object plane, in the plane of the transient scene inFIG.1A; and illuminated by a pulse laser to generate a transient scene. By tuning the initial phase of the sinusoidal function (tg), the exposure window of the camera14is adjusted in search of a peak or valley of the sinusoidal signal (tg), i.e. until local features of the static target are precisely matched in the streak image due to the symmetric back and forth scanning. Finally, 90° is added to the initial phase of the sinusoidal function (tg) to locate the linear slope region of the sinusoidal function (tg). The reconstructed movie has a frame rate of:

α=0.07 rad/V is a constant that links the voltage added onto the galvanometer scanner16, denoted as U, with the deflection angle in its linear rotation range. f4=75 mm is the focal length of Lens4, tRis the period of the sinusoidal voltage waveform added to the galvanometer scanner16, and d=5.86 μm is the pixel size of the CMOS camera14used in experiments. In addition, the pre-set exposure time teof the CMOS camera14determines the total length of the recording time window. If the entire streak is located within the CMOS camera14, the sequence depth can be calculated by Nt=rte. The number of pixels in the X axis of each frame, Nx, can be calculated by Nx≤Nc+1−Nt, where Ncis the number of pixels in each column of the CMOS camera14. The number of pixels Nyin the y axis of each frame is at most equal to the number of pixels Ntin each row of the CMOS camera14: Ny<Nt.

To characterize the spatial frequency responses of the system10, single laser pulses illuminating through a resolution target20were imaged (FIG.2A). A 532-nm continuous wave laser22controlled by an external trigger generated laser pulses with different temporal widths. Five different pulse widths, of 100, 300, 500, 700, and 900 μs, were used to provide decreased sparsity from 90% to 10% with a step of 20% in the temporal axis for a recording time window of 1 ms. The system10was used to capture these dynamic scenes at 60 kfps. The first panel inFIG.2Bshows illuminated bars corresponding to elements 4 to 6 in Group 2 and elements 1 to 6 in Group 3). Movies were reconstructed for each pulse width and datacubes representing the movies in a format of (x, y, t) were projected onto the x-y plane, as shown in the remaining panels inFIG.2B.

These results show that the spatial resolution of the system10depends on the sparsity of the transient scene. The contrast as well as the intensity the reconstructed image quality degrades with increasing laser pulse widths. To quantify the performance of the system by considering both effects, the normalized product of the contrast and the reconstructed intensity was used as the merit function (FIG.2C). For the 900-μs pulse illumination, Element 3 in Group 3 in the reconstruction has a normalized product below 0.25, which was used as the threshold to determine the resolvable feature, and the spatial resolution of the system was quantified to be 50 μm.

Thus, a method according to the present disclosure comprise multiplying a amplitude binary mask for each frame of the event, yielding encoded datacubes (x,y,t); shifting the different frames to different spatial positions as a function of their arrival time, yielding spatial-temporal shifting datacubes (x, y+t−1, t); integrating the datacubes as a 2D image (x, y+t−1); and retrieving a video from a measurement E of the 2D image, with:
E=TSoCI(x,y,t),  (1)
where I(x,y,t) is the light intensity of the transient event, C represents spatial encoding, Sorepresents linearly temporal shearing with the subscript “o” standing for “optical”, and T represents spatiotemporal integration; and

I^=argminI⁢{E-TSo⁢CI22+λϕTV⁡(I)}(2)
where ∥⋅∥22represents the l2norm, λ is a weighting coefficient, and ΦTVis total variation (TV) regularize.

To demonstrate the multi-scale ultra-high-speed imaging capability of the system, transmission of single laser pulses was captured through a mask. A beam splitter BS was used to divide the incident laser pulse into two components: the reflected component was recorded by a photodiode, generating time reference information (ground truth), and the transmitted component illuminated a transmissive mask with the letters USAF that modulated the spatial profiles of the laser pulses (FIG.3A), and was then recorded by the system10.

In a first experiment, a pulse train that contained four 300-μs pulses was generated. The imaging speed of the system10was set to 60 kfps. While the CMOS camera14, at its intrinsic imaging speed of 20 fps, provided a single image (see SI inFIG.38) without temporal information, the system10recorded the spatial profile of the mask and the intensity time course of the laser pulse in a movie with 240 frames/s. A representative frame (t=433 μs) is shown inFIG.3B.

FIG.3Cshows the normalized intensity of a selected cross section (dashed line inFIG.3BandFIG.3E), which demonstrates the well reconstructed spatial features with respect to the ground truth. The average intensity was also calculated in each frame. The time course shows a good agreement with the photodiode-recorded result (FIG.3D). Then the imaging speed was increased to 1.5 Mfps to record a single 10-μs laser pulse. The reconstructed movie is Movie3, and a representative frame (t=33 μs) is shown inFIG.3E. The comparison of the time courses of averaged intensity (FIG.3F) confirmed consistency between system and photodiode results under this imaging speed

To demonstrate the ability of the system to track fast moving objects, an animation of a fast-moving ball was imaged (FIG.4A). The animation comprised 40 patterns, which were loaded and played by a DMD30(such as D4100, Digital Light Innovations) at 20 kHz. A collimated laser beam32was imaged onto the digital micromirror device30at an angle of about 240 relative to the surface normal of the DMD30. The system10, positioned facing perpendicularly the surface of the DMD30, collected the light diffracted by the patterns at 140 kfps.

FIG.4Bshows a time-integrated image of the dynamic event acquired by the CMOS camera14of the system10at its intrinsic frame rate of 20 fps.

FIG.4Cshows a color-encoded image generated by superimposing ten representative time-lapse frames, with an interval of 215 μs, of the moving ball from the movies reconstructed by the system10. While the time-integrated image merely presents an overall trace, the time-lapse frames show the evolution of the spatial position and the shape of the moving ball, including the deformation of the ball from round to elliptical shape at turning points of its trajectory, at each time point.

To evaluate the accuracy of the reconstruction, the centroids of the bouncing ball were traced in each reconstructed frame (FIG.4E). The measurement errors were calculated by subtracting the measured position of centroids from the pre-set ones. Further, the root-mean-square errors (RMSEs) of reconstructed centroids along the x and y axes were calculated to be 22 μm and 9 μm, respectively. The anisotropy of the root-mean-square errors was attributed to the spatiotemporal mixing along the shearing direction.

The method and system were applied to wide-field phosphorescence lifetime imaging microscopy (PLIM). As illustrated inFIG.5A, a PLIM system100according to an embodiment of an aspect of the present disclosure comprises an excitation illumination unit110and a system10described hereinabove in relation toFIG.1Aas an imaging unit.

The excitation illumination unit110comprises a 980-nm continuous wavelength laser40, an optical chopper42, a tube lens44, an objective lens46, a dichroic mirror48, and an optical band-pass filter50.

The imaging speed of the imaging system10was 1 Mfps. Four up-conversion nanoparticles (UCNPs) with different core-shell structures were selected. All samples have a same core structure comprising NaGdF4: Er3+, Yb3+, Sample one having no shell, whereas Sample two to Sample four each have a shell of additional NaGdF4 around their core, with a shell of increasing thickness from Sample 2 to Sample 4. After pumped by 50-μs 980 nm laser pulse from the laser40, green (center wavelength at 545 nm), red (center wavelength at 660 nm) phosphorescence light emissions, and residual pump 980 nm light were detected by spectroscopy. To explore the green phosphorescence lifetime, the red emission light and the residual pump light were filtered out using filter50with center wavelength of 545 nm and spectral bandwidth of ±10 nm.

FIG.5Bshows a representative frame (at t=1 μs) of the movie that records 2-dimensional phosphorescence lifetime decay processing.FIG.5Cshows the exponential decay curves of the four samples, using point detection model where the imaging field of view of the PLIM system was reduced to a small area, with a diameter of about 100 micrometers. It can be seen that Sample one without a shell structure has the shortest lifetime (159 μm), whereas, Samples two to four have an increased lifetime (from 261 μs to 714 μs), as expected from the structure of the four UCNPs.

There is thus provided an imaging system comprising a DMD for spatially encoding each temporal frame of a scene by compressed sensing, a galvanometer scanner for temporal shearing, thereby creating an optical streak image, and a camera for capturing this linear image in a single shot. The mixture of 2D space and time data in the streak image is then processed to separate the data using reconstruction on the basis of the unique labels attached to each temporal frame by the DMD.

Based on optical streaking using a galvanometer scanner in a 4f imaging configuration, the present imaging system, using off-the-shelf camera, provides tunable imaging speeds of up to 1.5 Mfps, which is approximately three orders of magnitude higher than the state-of-art in imaging speed of compressed sensing-based temporal imaging using silicon sensors, a Megapixel-level spatial resolution, with a pixel count of 0.5 megapixels in each frame, and 500-frame sequence depth (i.e. the number of frames in the movie), and capable of single-shot 2-dimensional phosphorescence lifetime imaging.

The ultra-high-speed imaging capability of the system was demonstrated by capturing the transmission of single laser pulses through a mask and by tracing the shape and position of a fast-moving object in real time. There is thus provided a single-shot cost-efficiency ultra-high-speed universal imaging method and system.

The system may be integrated into a range of imaging instruments from microscopes to telescopes, to achieve a scalable spatial resolution by coupling with different front optics in these imaging instruments. Moreover, the system can be used with different cameras, such as CCD or CMOS cameras according to specific applications, allowing applying the method to a wide range of wavelengths and for acquiring various optical characteristics such as polarization. For instance, an electron-multiplying CCD camera may be combined with the system to enable high-sensitivity optical neuroimaging of action potential propagating at tens of meters per second under microscopic settings; by leveraging the imaging speed and spatial resolution of the system10, the method was applied to action potential propagating at tens of meters per second.

As another example, an infrared-camera-based may be integrated to enable wide-field temperature sensing in deep tissue using nanoparticles. In summary, by leveraging the advantages of off-the-shelf components including camera, galvo, DMD, and achromatic lenses, the present invention provides a system and a method for widespread applications in both fundamental and applied sciences

Featuring optical streaking using a galvanometer scanner in the 4f imaging system, the all-optical system uses an off-the-shelf CMOS camera with tunable imaging speeds of up to 1.5 Mfps, which is approximately three orders of magnitude higher than the state-of-art in imaging speed of compressed sensing-based temporal imaging using silicon sensors such as P2C2 and CACTI. In addition, the system can reach a sequence depth of up to 500 frames and a pixel count of 0.5 megapixels in each frame.

There is thus provided a single-shot compressed optical-streaking ultra-high-speed photography system and method, as a passive-detection computational imaging modality with a 2D imaging speed of up to 1.5 million frames per second (Mfps), a sequence depth of 500 frame, and an (x,y) pixel count of 1000×500 per frame, using standard imaging sensors typically limited to 100 frames per second.