Patent ID: 12254592

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present disclosure is further described below with reference to the specific embodiments and accompanying drawings.

Embodiment 1

In a dSIM reconstruction method in this embodiment of the present disclosure, an original image is a 3D original image. As shown inFIG.4, the method includes the following steps:

(1) Obtain an original image stack.

Illumination of a sample by structured light has three modulation directions: a first modulation direction, a second modulation direction, and a third modulation direction. Each modulation direction has five phases. One original image is acquired at each phase in each modulation direction such that 15 3D original images are obtained to form the original image stack at two modulation frequencies in space.

(2) Preprocess the original image stack.

(a) Two frequency shifts, namely, shifts of a first modulation frequency K1 and a second modulation frequency K2, are observed from the 3D original images inFIG.1AtoFIG.1F. The shifts of the two modulation frequencies are in a same direction as fringe modulation. For each 3D original image in the original image stack, extract the first modulation frequency K1, the second modulation frequency K2, and a zero frequency K0 of each pixel by using a wavelet packet frequency separation method.

(b) For each 3D original image, generate a first extracted image by combining the first modulation frequency K1 with the zero frequency K0 to obtain a first extracted image stack, and generate a second extracted image by combining the second modulation frequency K2 with the zero frequency K0 to obtain a second extracted image stack. Two extracted images are generated from one 3D original image. An extracted image stack including the first extracted image stack and the second extracted image is formed. A number of images in the extracted image stack is doubled, namely, 30.

(c) Perform interpolation on each image in the first extracted image stack and the second extracted image stack through spatial frequency domain FFT zero padding to increase a sampling frequency to more than twice and obtain first and second interpolated extracted image stacks.

(d) Denoise extracted images in the first and second interpolated extracted image stacks through Butterworth low-pass filtering to obtain first and second denoised extracted image stacks.

(e) Perform deconvolution on each extracted image in the first and second denoised extracted image stacks through an RL algorithm to improve a relative strength of a high-frequency signal and obtain first and second preprocessed extracted image stacks to form a preprocessed image stack.

(3) Extract the first modulation frequency K1 and the second modulation frequency K2 from the preprocessed image stack.

(a) Extract the first modulation frequency K1 from a timing signal in each pixel of each image in the first preprocessed image stack and extract the second modulation frequency K2 from a timing signal in each pixel of each image in the second preprocessed image stack by using a wavelet packet filter, where a wavelet family is Fejer-Korovkin.

(b) Extract, by using a sigmoid low-pass filter, complex modulation signals from time evolution of the first modulation frequency K1 and the second modulation frequency K2 extracted from each pixel, to convert an incoherent signal into a coherent signal.

(c) Perform FFT interpolation on a complex amplitude signal in the extracted complex modulation signals to increase the sampling frequency to more than twice and obtain a complex modulation image stack.

(4) Perform autocorrelation calculation on each pixel.

This is a first core step of dSIM. For the complex modulation signals extracted from the complex modulation image stack, an autocorrelation accumulation amount is calculated at each pixel. One autocorrelation image is generated for M phases of the first modulation frequency K1 in each modulation direction to obtain N autocorrelation images. The autocorrelation is followed by a real number. Phase information is canceled. One autocorrelation image is generated for the second modulation frequency K2 in each modulation direction to obtain N autocorrelation images. In this way, six autocorrelation images are generated from each 3D original image, as shown inFIG.1AtoFIG.1F.FIG.1GtoFIG.1LandFIG.1MtoFIG.1Rare respectively spatial domain and frequency domain images of processing results of the three modulation directions of the first modulation frequency K1 and the second modulation frequency K2. It can be learned that the second modulation frequency K2 has a larger frequency domain range. However, due to limitation of an OTF, a modulation signal strength of the second modulation frequency K2 that is higher is far less than that of the first modulation frequency K1. Consequently, an intensity of the autocorrelation image of the second modulation frequency K2 is lower than that of the first modulation frequency K1.

(5) Post-process the autocorrelation images.

This is a second core step of dSIM. Each autocorrelation image has unidirectional modulation, thereby providing a super-resolution image with unidirectional modulation. When the results of the three directions are added, a resolution isotropically increases. However, direct summation of all directions may greatly reduce the resolution. To maintain a high resolution in a single direction, deconvolution is performed on the result of each direction. RL deconvolution is performed on the autocorrelation images. RL deconvolution is used to enhance high-frequency components within a cut-off frequency. Then, a square root of each pixel value in deconvoluted autocorrelation images is calculated to obtain dSIM intermediate processing results. Calculating the square root can improve linearity of the results, but reduces the resolution of a dSIM image.

(6) Perform dSIM image fusion.

The dSIM intermediate processing results at the first modulation frequency in different modulation directions are added and a result is shown inFIG.2A. The dSIM intermediate processing results at the second modulation frequency in different modulation directions are added and a result is shown inFIG.2B. InFIG.2EandFIG.2F, cut-off frequencies of summation results of the first and second modulation frequencies K1 and K2 are almost isotropic in comparison with the unidirectional processing result inFIG.1A-R. Finally, a final dSIM result, namely, a summation result ofFIG.2AandFIG.2B, is obtained by adding all directional results of the first and second modulation frequencies K1 and K2, as shown inFIG.2C.

InFIG.2G, unlike a frequency domain of the conventional SIM, the frequency domain of the dSIM is smoother, in which there is hardly a peak caused by the frequency shift. Therefore, the dSIM hardly generates a cellular artifact in the conventional SIM. For convenience of comparison, the frequency domain of the conventional SIM is interpolated to make the frequency domain range and pixel size consistent with those of the dSIM. The white dotted box in the figure is the frequency range of the SIM reconstruction result. It can be learned fromFIG.2GandFIG.2H, the cut-off frequency of the dSIM is slightly higher than a frequency boundary of the SIM.

InFIG.3A-F, more details of experimental imaging of actin through the SIM and dSIM are provided and the results are compared in terms of defocus background cancellation and Moire artifacts. It can be learned from comparison betweenFIG.3CandFIG.3Dthat in a conventional SIM algorithm, as shown inFIG.3D, obvious defocus artifacts appear, which are represented as strong stripe artifacts in three directions as shown by arrows. This is because the conventional SIM algorithm cannot eliminate impact of a defocus signal, and the defocus is shifted to a high frequency during the frequency shift, resulting in the stripe defocus artifacts. In the processing result of the dSIM method, there is no defocus artifact, and microfilament protein on a focal plane can be clearly seen. It can be learned fromFIG.3Fthat due to a change in parameters such as a local modulation depth, ripple artifacts appear around normal microfilament protein, as shown by the arrows. The artifacts affect judgment of a real structure of the sample. InFIG.3E, there is no obvious Moire artifact. This indicates that the dSIM method is insensitive to the change in the parameters such as the modulation depth, and is not prone to artifacts caused by a parameter estimation error of the conventional algorithm.

Embodiment 2

In a dSIM reconstruction method in this embodiment, an original image is a 2D original image. The method includes the following steps:

(1) Obtain an original image stack.

Illumination of a sample by structured light has three modulation directions. Each modulation direction has three phases. One original image is acquired at each phase in each modulation direction such that nine 2D original images are obtained to form the original image stack.

(2) Preprocess the original image stack.

(a) Perform interpolation on each 2D original image in the original image stack through spatial frequency domain FFT zero padding to increase a sampling frequency to more than twice and obtain an interpolated extracted image stack.

(b) Denoise extracted images in the interpolated extracted image stack to obtain a denoised extracted image stack.

(c) Perform deconvolution on each extracted image in the denoised extracted image stack through an RL algorithm to improve a relative strength of a high-frequency signal and obtain a preprocessed image stack.

(3) Extract a modulation frequency K from the preprocessed image stack.

(a) Extract the modulation frequency K from a timing signal in each pixel of each image in the preprocessed image stack by using a wavelet packet filter.

(b) Extract, by using a sigmoid low-pass filter, complex modulation signals from time evolution of the modulation frequency K extracted from each pixel, to convert an incoherent signal into a coherent signal.

(c) Perform FFT interpolation on a complex amplitude signal in the extracted complex modulation signals to increase the sampling frequency to more than twice and obtain a complex modulation image stack.

(4) Perform autocorrelation calculation on each pixel.

For the complex modulation signals extracted from the complex modulation image stack, calculate an autocorrelation accumulation amount at each pixel, generate one autocorrelation image for three phases in each modulation direction to obtain three autocorrelation images, and generate a super-resolution image by using a correlation between signals at different spatial positions.

(5) Post-process the autocorrelation images.

Perform RL deconvolution on the autocorrelation images, and calculate a square root of each pixel value in deconvoluted autocorrelation images to obtain dSIM intermediate processing results. Calculating the square root can improve linearity of the result, but reduces the resolution of a dSIM image.

(6) Perform dSIM image fusion.

Add the dSIM intermediate processing results at the modulation frequency in different modulation directions to generate a final dSIM image.

Finally, it should be noted that disclosure of the embodiments is intended to help further understand the present disclosure. Those skilled in the art can understand that various substitutions and modifications may be made without departing from the spirit and scope of the present disclosure and the appended claims. Therefore, the present disclosure should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present disclosure is subject to the scope defined by the claims.

CITED REFERENCE

[1] Muller, M., et al., Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nature Communications, 2016.7:p. 10980.