Patent Description:
The present invention relates to a system for illuminating patterns on a sample, especially relating to a microscope-based system for illuminating varying patterns through a large number of fields of view consecutively at a high speed.

There are needs in illuminating patterns on samples (e.g. biological samples) at specific locations. Processes such as photobleaching of molecules at certain subcellular areas, photoactivation of fluorophores at a confined location, optogenetics, light-triggered release of reactive oxygen species within a designated organelle, or photoinduced labeling of proteins in a defined structure feature of a cell all require pattern illumination. For certain applications, the pattern of the abovementioned processes may need to be determined by a microscopic image. Some applications further need to process sufficient samples, adding the high-content requirement to repeat the processes in multiple regions. Systems capable of performing such automated image-based localized photo-triggered processes are rare.

One example of processing proteins, lipids, or nucleic acids is to label them for isolation and identification. The labeled proteins, lipids, or nucleic acids can be isolated and identified using other systems such as a mass spectrometer or a sequencer. STOMP (spatially targeted optical microproteomics) technique proposed by Kevin C Hadley et al. in <NUM> is a technique that is operated manually using a commercially available two-photon system, lacking the main elements to reach the high-content capability of this invention. The laser capture microdissection (LCM) system widely used to isolate a part of tissues or cell cultures using laser cutting does not have axial precision that this invention can achieve in addition to the lack of high-content capability. Patent document <CIT> discloses a fluorescence detecting apparatus including: a light source for illuminating the surface of detection of a sample with excitation light; a CCD area image sensor for detecting fluorescence emitted from the sample; and a micromirror array device for irradiating excitation light in a <NUM>-dimensional illumination pattern generated by an information processor based on the detected result from the CCD area image sensor. Patent document <CIT> discloses a method of fluorescence imaging including: illuminating a sample to excite its fluorescence and acquiring an image thereof; based on fluorescence spectral and spatial information from the sample's fluorescence image, segmenting the image into regions of similar spectral properties; for each image segment, exciting the fluorescence of the corresponding sample region, and detecting the corresponding fluorescence; based on modelling, determining expected fluorescence parameters from the fluorescence signals detected for each region; scanning the sample and determining final fluorescence parameters based on said expected fluorescence parameters. Patent document <CIT> discloses a microscope system including: a light source configured to emit first excitation light for causing a fluorescent substance of a biological sample stained with the fluorescent substance to emit light; an objective lens configured to condense the first excitation light to the biological sample; a scanning mechanism configured to change an orientation of the first excitation light from the light source such that the first excitation light condensed by the objective lens scans the biological sample; a photodetector configured to input a first fluorescence that is generated from the biological sample by the first excitation light condensed to the biological sample, and to convert the first fluorescence into an electrical signal; and a macro imaging unit configured to apply second excitation light to the biological sample, and to capture an image of a second fluorescence by macro imaging, the second fluorescence being generated from the biological sample. <CIT> discloses an optical arrangement for deflecting a light beam comprising: a first rotary drive defining a first axis of rotation, a second rotary drive defining a second axis of rotation, wherein the axes are mutually perpendicular, a first mirror rotated by the first rotary device, a second mirror rotated by the second rotary device, a third mirror arranged in an angular position with respect to the first mirror, wherein the first and third mirror rotate jointly about the first axis, a pivot point is defined on the second mirror, wherein the light beam pivots about the pivot which lies on the second of rotation of the second mirror, and the first, second and third mirrors are arranged, that a light beam running between the second and the third mirrors and the second axis of rotation are always substantially in one plane and the plane is perpendicular to the first axis of rotation.

In view of the foregoing objectives, the invention provides image-guided systems to enable illuminating varying patterns on the sample. In other words, such systems and methods have abilities to process a high content of proteins, lipids, nucleic acids, or biochemical species for regulation, conversion, isolation, or identification in an area of interest based on user-defined microscopic image features, widely useful for cell or tissue sample experiments. With a unique integration of optical, photochemical, image processing, and mechatronic designs, we are able to achieve image-guided illumination at a speed of <NUM> milliseconds per field of view, not accomplished by existing techniques such as STOMP or LCM. This speed is necessary to collect enough biomolecular samples in a reasonable duration. For example, we are able to collect enough protein samples for proteomic analysis with <NUM> hours of illumination. A distinct design strategy differentiates this invention from any existing techniques.

The system may comprise a microscope, an imaging light source, a digital camera, a first processing module, a second processing module such as field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC), an illumination light source, a shutter, a pattern illumination device such as a pair of galvanometer scanning mirrors, a digital micromirror device (DMD), or a spatial light modulator (SLM), a microscope stage, and an autofocus device. A processing module is programmed to grab a camera image and to process the image on board in real-time based on user-defined criteria to determine the locations of the sample to illuminate. It is then programmed to control a shutter and scanning mirrors to direct an illumination light source to these locations one point at a time. The integrated memory unit (e.g. DRAM), when available, provides the data storage space essential for fast processing and transfer of image data to enable a rapid process. One processing module (e.g. a computer) controls an imaging light source, an autofocus device, and a microscope stage for imaging, focus maintenance, and changes of fields of view, respectively. Imaging, image processing, illumination, and stage movement are coordinated by the program to achieve rapid high content image-guided illumination. A femtosecond laser may be used as the illumination light source to generate a two-photon effect for high axial illumination precision. The image processing criteria may be the morphology, intensity, contrast, or specific features of a microscope image. Image processing is done with image processing techniques such as thresholding, erosion, filtering, or artificial intelligence trained semantic segmentation methods. The speed and the high content nature of the device may enable collection of a large amount of location-specific samples for photo-induced molecular tagging, photoconversion, or studies of proteomics, transcriptomics, and metabolomics.

To achieve the above objective, the present disclosure provides a microscope-based system for image-guided microscopic illumination as outlined in independent claim <NUM>.

To use the system of this disclosure, a cell or tissue sample can be prepared with photosensitizers and chemical agents in the media. At one field of view, a microscopic image is taken. An image processing program is then applied to the captured image to determine where the sample (such as proteins, lipids, nucleic acids, or other biochemical species) would be illuminated (e.g. photo-activated or processed by photochemical reaction using a two-photon illumination light source). The computer then transfers the coordinates of points of interest to scanners for localized illumination. For example, when the scenario is that where a photochemical reaction is required, the photosensitizers which are already applied in the area of interest may be excited by the excitation energy provided by the illumination light and trigger the chemical agents to process proteins, lipids, nucleic acids, or biochemical species in the illumination area. The microscope stage is then controlled to move to the next field of view again and again to repeat this image-guided photoconversion process until enough samples are processed.

The high-content capability of the system may be achieved and facilitated by optimal choices of scanners, shuttering devices, imaging approach, and optimal designs of real-time image processing and control of a pattern illumination device, a microscopic stage and shuttering devices. Chips such as programmable or customized electronics using FPGA or ASIC may be used for system optimization. This integration of software, firmware, hardware, and optics enables the high-content capability that differentiates this invention from other related techniques.

Accordingly, the present disclosure provides a microscope-based system for image-guided microscopic illumination. The microscope-based system utilizes two independent processing modules (i.e., the first processing module and the second processing module) which simultaneously controls the imaging assembly for taking at least one image of a sample and the illuminating assembly for illuminating the sample. Moreover, the second processing module is in communication with the first processing module and receives the coordination of the targeted points on the sample (i.e. the "interested region" in the image of the sample and processed by the first processing module) so as to rapidly control the illuminating assembly to illuminate targeted points on the sample. Hence, the image-guided systems of this disclosure enable a high-content process to illuminate varying patterns through a large number of fields of view consecutively.

The embodiments will become more fully understood from the detailed description and accompanying drawings, which are given for illustration only, and thus are not limitative of the present invention, and wherein:.

The embodiments of the invention will be apparent from the following detailed description, which proceeds with reference to the accompanying drawings, wherein the same references relate to the same elements.

It is to be noted that all directional indications (such as up, down, left, right, front, rear and the like) in the embodiments of the present disclosure are only used for explaining the relative positional relationship, circumstances during its operation, and the like, between the various components in a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

Unless the context dictates the contrary, all ranges set forth herein should be interpreted as being inclusive of their endpoints, and open-ended ranges should be interpreted to include only commercially practical values. Similarly, all lists of values should be considered as inclusive of intermediate values unless the context indicates the contrary.

The recitation of ranges of values herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range. Unless otherwise indicated herein, each individual value with a range is incorporated into the specification as if it were individually recited herein. The use of any and all examples, or exemplary language (e.g. "such as") provided with respect to certain embodiments herein is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.

The system and method provide by the various embodiments of this present disclosure may relates to for processing, for example but not limited thereto, a high content of proteins, lipids, nucleic acids, or biochemical species comprising an imaging light source, a photosensitizing light source, a pattern illumination device such as a set of dual-axis high-speed galvanometric scanning mirrors, a microscope body, a focusing device, a high-precision microscope stage, a high-sensitivity digital camera, a control workstation (or a personal computer), a processing module such as a FPGA chip, and application software for camera control, image processing, stage control, and optical path control. It is therefore an object to process the proteins, lipids, nucleic acids, or biochemical species in an area of interest specified by fluorescent signals or structural signature of cell images. Additional object is to collect a large amount of proteins, lipids, or nucleic acids through high content labeling and purification in order to identify biomolecules of interest in the area of interest by a mass spectrometer or a nucleic acid sequencer, followed by proteomic, metabolomic, or transcriptomic analyses.

The systems according to some embodiments of the invention take fluorescent staining or brightfield images first. Image processing is then performed automatically on the images by a connected computer using image processing techniques such as thresholding, erosion, filtering, or artificial intelligence trained semantic segmentation methods to determine the points or areas to be processed based on the criteria set by the operating individual. A high-speed scanning system is used for pattern illumination to shine the photosensitizing light on the points or areas to induce processing of proteins, lipids, nucleic acids, or biochemical species in the illumination regions. Alternatively, DMD or SLM may be used for pattern illumination. Photo-induced processing is achieved by including photosensitizer such as riboflavin, Rose Bengal or photosensitized protein (such as miniSOG and Killer Red, etc.) and chemical reagents such as phenol, aryl azide, benzophenone, Ru(bpy)<NUM><NUM>+, or their derivatives for labeling purpose. The labeling groups can be conjugated with tagging reagents like biotin that is used for protein or nucleic acid pulldown. Photosensitizer, labeling reagent, and tagging reagent can be separate molecules, or can be one molecule with all three functions. Spatially controlled illumination can induce covalent binding of the labeling reagents onto amino acids, lipids, nucleic acids, or biochemical species in the specific region. As examples, streptavidin is used to isolate biotinylated proteins and then the labeled proteins are purified to be analyzed by a mass spectrometer. RNA may be isolated by associated protein pulldown and then analyzed by RNAseq or RTPCR. Because enough RNAs or proteins are needed to have low background results, efficient high-content labeling is thus a major requirement for this system.

Certain exemplary embodiments according to the present disclosure are described as below.

This disclosure provides an embodiment which is also a microscope-based system for image-guided microscopic illumination. Please refer to <FIG> and <FIG>. The microscope-based system of this embodiment comprises a microscope <NUM>, an imaging assembly <NUM>, an illuminating assembly <NUM>, and a processing module 13a. The microscope <NUM> comprises an objective <NUM> and a stage <NUM>. The stage <NUM> is configured to be loaded with a sample S. The imaging assembly <NUM> may comprise a (controllable) camera <NUM>, an imaging light source <NUM>, a focusing device <NUM>, and a first shutter <NUM>. Please further refer to both <FIG> and <FIG>, the illuminating assembly <NUM> may comprise an illumination light source <NUM> and a patter illumination device <NUM>. The patter illumination device <NUM> may include a second shutter <NUM>, a lens module <NUM> (such as the relay lens 113a and 113b, a quarter wave plate 113c), at least a pair of scanning mirrors <NUM> and a scan lens <NUM>. Alternatively, DMD or SLM can be used as the pattern illumination device <NUM>.

In this embodiment, the processing module 13a is coupled to the microscope <NUM>, the imaging assembly <NUM>, and the illuminating assembly <NUM>. The processing module 13a can be a computer, a workstation, or a CPU of a computer, which is capable of executing a program designed for operating this system.

The processing module 13a controls the imaging assembly <NUM> such that the camera <NUM> acquires at least one image of the sample S of a first field of view, and the image or images are transmitted to the processing module 13a and processed by the processing module 13a automatically in real-time based on a predefined criterion, so as to determine an interested region in the image S and so as to obtain a coordinate information regarding to the interested region. Later, the processing module 13a may control the pattern illumination device <NUM> of the illuminating assembly <NUM> to illuminate the interested region of the sample S according to the received coordinate information regarding to the interested region. Also, after the interested region is fully illuminated, the processing module 13a controls the stage <NUM> of the microscope <NUM> to move to a second field of view which is subsequent to the first field of view.

In this embodiment, the imaging light source <NUM> provides an imaging light through an imaging light path to illuminate the sample S during imaging the sample. The first shutter <NUM>, along the imaging light path, is disposed between the image light source <NUM> and the microscope <NUM>. The controllable camera <NUM> is disposed on the microscope <NUM> or on the imaging light path.

Also, the illuminating light source <NUM> provides an illuminating light through an illuminating light path to illuminate the sample S. The pattern illumination device <NUM>, along the illuminating light path, is disposed between the illumination light source <NUM> and the microscope <NUM>.

Please refer to <FIG>, which is a flow chart of the image-guided method carried out by the system according to this embodiment of the present disclosure. It is a more detailed example of the image-guided method described herein, but the present invention is not limited thereby. The process of the method depicted in <FIG> comprises the following steps S11' to S23'.

In brief, in the step S <NUM>', the operator moves the stage <NUM> of the microscope <NUM> to the starting position. In step S12', the processing module 13a opens the first shutter <NUM>. In step S13', the processing module 13a triggers the camera <NUM> to take an image of the sample S. In step S14', the processing module 13a then closes the first shutter <NUM>. In step S <NUM>', the camera <NUM> transfers the image data to the processing module 13a. In step S16', the processing module 13a performs image processing and obtains an XY-coordinate array targeted points to be illuminated (i.e. the "interested region" of the sample S). In step S <NUM>', the processing module 13a transfers the coordinate array to the signal converter (such as DAC) <NUM> to convert to analog voltages. In step S18', analog voltages are transferred to XY scanning mirrors <NUM> to direct the illumination light to one targeted point. In the step S19', the processing module 13a turns on the second shutter <NUM>. In step S20', the processing module 13a turns off the second shutter <NUM>. In the step S21', the system may check whether or not all targeted points are illuminated. In other words, if the targeted points are not fully illuminated yet, the procedure will go back to the step S18' and the processing module 13a in this step will control the XY scanning mirrors <NUM> to direct to the next point, and the second shutter on/off (i.e. steps S19' and S20'). Once all targeted points in the XY-coordinate array are illuminated, the procedure may then go to the next step. In the step S22', the processing module 13a controls movement of the stage <NUM> of the microscope <NUM> to the next field of view. In the step S23', the system will check whether or not all fields of view are processed. If all the fields of view of the sample S are processed, the whole procedure may come to an end. If not, the procedure will then go back to the step S12' to start another round of imaging, image processing, illumination, and stage movement after processes in each FOV. In other words, the same cycle of imaging, image processing, illumination, and stage movement is performed one FOV at a time until enough sample areas are illuminated.

In this embodiment, the image processing is done with real-time image processing techniques such as thresholding, erosion, filtering, or artificial intelligence trained semantic segmentation methods.

Because composition, variation or connection relationship to other elements of each detail elements of the microscope-based method can refer to the previous embodiments, they are not repeated here.

This disclosure provides another embodiment which is also a microscope-based system for image-guided microscopic illumination. This system includes an additional processing module to improve illumination performance and will be describe in detail. Please refer to <FIG> and <FIG>. <FIG> represents a schematic diagram of an imaging-guided system according to one embodiment of the present disclosure, and <FIG> depicts the optical path of the image-guided system of <FIG>.

As shown in <FIG> and <FIG>, the microscope-based system <NUM> for image-guided microscopic illumination comprises a microscope <NUM>, an illuminating assembly <NUM>, an imaging assembly <NUM>, a first processing module <NUM> and a second processing module <NUM>. The microscope-based system <NUM> is designed to take a microscope image or images of a sample and use this image or these images to determine and shine an illumination pattern on the sample, finishing all steps for one image rapidly (e.g. within <NUM>), and within a short time (e.g. <NUM> hours) for the entire illumination process for a proteomic study.

The microscope <NUM> comprises a stage <NUM>, an objective <NUM> and a subjective <NUM>. The stage is configured to be loaded with a sample S. The stage <NUM> of the microscope <NUM> can be a high-precision microscope stage.

The imaging assembly <NUM> may comprise a camera <NUM>, an imaging light source <NUM>, a focusing device <NUM>, and a first shutter <NUM>. The camera <NUM> is mounted on the microscope <NUM>. In detail, the camera <NUM> is coupled to the microscope <NUM> through the subjective <NUM> of the microscope <NUM>. The focusing device is coupled to the camera <NUM> and controlled to facilitate an autofocusing process during imaging of the sample S. The imaging light source <NUM>, which provides an imaging light (as shown in the shaded area in <FIG> from imaging assembly <NUM> to the objective <NUM>) through an imaging light path (as shown with the route indicated by the open arrows in the shaded area depicting the imaging light in <FIG>) to illuminate the sample S. The first shutter <NUM>, along the imaging light path, is disposed between the image light source <NUM> and the microscope <NUM>. The imaging light source <NUM> can be a tungsten-halogen lamp, an arc lamp, a metal halide lamp, a LED light, a laser, or multiple of them. The shuttering time of the first shutter may vary with the type of the imaging light source <NUM>. Using an LED light source as an example, the shuttering time of the first shutter <NUM> is <NUM> microseconds.

If one would like to perform two color imaging, the shutter of the first color light is turned off and the shutter of the second color light is turned on by the first processing module <NUM>. This may take another <NUM> microseconds. The camera <NUM> then takes another image with an exposure time of another <NUM> millisecond. The first processing module <NUM> then turns off the shutter of the second color light.

In this embodiment, please further refer to both <FIG> and <FIG>, the illuminating assembly <NUM> comprises an illuminating light source <NUM>, and a pattern illumination device <NUM> including a second shutter <NUM>, a lens module <NUM> (such as the relay lens 113a and 113b, a quarter wave plate 113c), at least a pair of scanning mirrors <NUM> and a scan lens <NUM>. Alternatively, DMD or SLM can be used as the pattern illumination device <NUM>. The illuminating light source <NUM> provides an illuminating light (as shown in the open arrows from the illuminating assembly <NUM> to the objective <NUM> in <FIG>) through an illuminating light path to illuminate the sample S. The second shutter <NUM>, along the illuminating light path, is disposed between the illuminating light source <NUM> and the microscope <NUM>. The pair of scanning mirrors <NUM>, along the illuminating light path, is disposed between the second shutter <NUM> and the microscope <NUM>. The camera <NUM> may be a high-end scientific camera such as an sCMOS or an EMCCD camera with a high quantum efficiency, so that a short exposure time is possible. To get enough photons for image processing, the exposure time is, for example, <NUM> milliseconds.

The first processing module <NUM> is coupled to the microscope <NUM> and the imaging assembly <NUM>. In detail, the first processing module <NUM> is coupled and therefore controls the camera <NUM>, the imaging light source <NUM>, the first shutter, the focusing device <NUM>, and the stage <NUM> of the microscope <NUM>, for imaging, focus maintenance, and changes of fields of view. The first processing module <NUM> can be a computer, a workstation, or a CPU of a computer, which is capable of executing a program designed for operating this system. The first processing module <NUM> then triggers the camera <NUM> to take the image of the sample S of a certain field of view (FOV). In addition, the camera <NUM> can be connected to the first processing module <NUM> through an USB port or a Camera Link thereon. The controlling and the image-processing procedures of this system will be discussed more detailed in the following paragraphs.

In this embodiment, the second processing module <NUM> is coupled to the illuminating assembly <NUM> and the first processing module <NUM>. In detail, the second processing module <NUM> is coupled to and therefore controls the pattern illumination device <NUM>, including the second shutter <NUM>, and the pair of scanning mirrors, for illuminating the targeted points in the interested region determined by the first processing module <NUM>. The second processing module may be a FPGA, an ASIC board, another CPU, or another computer. The controlling and the image-processing procedures of this system will be discussed more detailed in the following paragraphs.

In brief, the microscope-based system <NUM> is operated as below. The first processing module <NUM> controls the imaging assembly <NUM> such that the camera <NUM> acquires at least one image of the sample S of a first field of view. The image or images are then transmitted to the first processing module <NUM> and processed by the first processing module <NUM> automatically in real-time based on a predefined criterion, so as to determine an interested region in the image and so as to obtain a coordinate information regarding to the interested region. The image processing algorithm is developed independently beforehand using image processing techniques such as thresholding, erosion, filtering, or artificial intelligence trained semantic segmentation methods. Later, the coordinate information regarding to the interested region is transmitted to the second processing module <NUM>. The second processing module <NUM> controls the illuminating assembly <NUM> to illuminate the interested region (or, namely, irradiating those targeted points in the interested region) of the sample S according to the received coordinate information regarding to the interested region. In addition, after the interested region is fully illuminated (or all the targeted points in the interested region are irradiated), the first processing module <NUM> controls the stage <NUM> of the microscope <NUM> to move to the next (i.e. the second) field of view which is subsequent to the first field of view. After moving to the subsequent field of view, the method further repeats imaging-image processing-illumination steps, until interested regions of all designated fields of view are illuminated.

Moreover, this disclosure also provides another embodiment of a system which is configured to carry out microscope-based method for image-guided microscopic illumination. The microscope-based method uses the microscope-based system described above and comprises the following steps (a) to (e): (a) triggering the camera <NUM> of the imaging assembly <NUM> by the first processing module <NUM> to acquire at least one image of the sample S of a first field of view, and the sample S is loaded on the stage <NUM> of the microscope <NUM>; (b) automatically transmitting the image or images of the sample S to the first processing module <NUM>; (c) based on a predefined criterion, performing image processing of the sample S automatically in real-time by the first processing module <NUM> to determine an interested region in the image and obtain a coordinate information regarding to the interested region; (d) automatically transmitting the coordinate information regarding to the interested region to the second processing module <NUM>; (e) controlling an illumination assembly <NUM> by the second processing module <NUM> according to the received coordinate information to illuminate the interested region in the sample S. Besides, in this embodiment, after the interested region is fully illuminated, the method may further comprise a step of: controlling the stage <NUM> of the microscope <NUM> by the first processing module <NUM> to move to the next (i.e. the second) field of view which is subsequent to the first field of view.

The microscope-based system <NUM> used herein are substantially the same as that described above, and the details of the composition and variations of the compositing elements are omitted here.

Please refer to <FIG>, which is a flow chart of the image-guided method according to this embodiment of the present disclosure. It is a more detailed example of the image-guided method described herein, but the present invention is not limited thereby. The process of the method depicted in <FIG> comprises the following steps S11 to S23.

In step S11: The operator moves the stage <NUM> of the microscope <NUM> to the starting position.

In step S12: the first processing module <NUM> opens the first shutter <NUM>.

In step S13: the first processing module <NUM> triggers the camera <NUM> to take an image of the sample S.

In step S14: the first processing module <NUM> then closes the first shutter <NUM>.

In step S15: the camera <NUM> transfers the image data to the first processing module <NUM>. After the camera <NUM> takes the image of the sample S, it now has the data for the image, for example, a 2048x2048-pixel image with <NUM> bits for each pixel, equivalent to ~<NUM> Megabytes (MB). This data can then be transferred to the first processing module <NUM>. It may take less than <NUM> to complete the data transfer.

In step S16: the first processing module <NUM> performs image processing and obtains an XY-coordinate array targeted points to be illuminated (i.e. the "interested region" of the sample S). In other words, after receiving the image data transferred from the camera <NUM>, the CPU of the first processing module <NUM> performs image processing to determine the interested regions (or targeted points) of the sample S to be illuminated (or, namely, excited, irradiated, or photo-activated) by the illuminating light source <NUM>. The interested regions or the targeted points can be defined by the user. They can be the locations of cell nuclei, nucleoli, mitochondria, or any cell organelles or sub-organelles. They can be the locations of a protein of interest, such as myosin V, E-cadherin, p53, or any. They can be a morphological signature, such as primary cilia that are longer than <NUM> or microtubules close to the aster. They can also be regions at a specific time point, such as dividing nuclei. They can also be a feature defined by two color imaging, such as the colocation sites of proteins A and B, or actin filaments close to the centrosome.

Now, there are a lot of image processing operations that are readily available for the first processing module <NUM>. One can use existing operations or combinations of operations to achieve various image processing demands described above. For example, thresholding, erosion, dilation, edge detection, filtering, segmentation, or transformation can be used to define the regions of interest. Artificial intelligence trained semantic segmentation methods can also be used. The operations can be affected by the conditions of images, so the criteria may be tuned case by case. Basically all operations are matrix-based linear algebra operations. Because the different complexity levels of different image processing goals, the time required for this step is varying. A simple thresholding operation takes <NUM> milliseconds for an image, whereas a complex combination of operations can take <NUM> milliseconds for an image. A series of operations require the space of multiple copies of matrices, so an integrated DRAM on an FPGA board as the second processing module <NUM> may be needed.

In addition, the illumination light source <NUM> is different from the imaging light source <NUM> for sample imaging. The illumination light source <NUM> here is used only to illuminate the interested regions determined by image processing performed in the step S16. There are many potential usages for illuminating selected areas based on the image of the sample S. For example, one can perform photo-induced molecular tagging, photoconversion, or studies of proteomics, transcriptomics, and metabolomics for molecules with the introduction of photochemical reactions in these interested regions. As described below, the illumination of the interested regions is achieved by point scanning. That is, the illumination light source <NUM> may be a laser, and the point scanning is achieved by scanning mirrors <NUM> such as galvanometer mirrors. That is, it is similar to a confocal microscope setup. If one would like to have well-controlled axial illumination, one may use two-photon microscopy. In this case, one can use a femtosecond laser as the illumination light source <NUM>.

Moreover, as shown in <FIG>, <FIG> and <FIG>, the light path of the illumination starts from the illumination light source <NUM>. The second shutter <NUM> is needed for this illumination light source <NUM>. To reach a high switching speed for the point illumination, a mechanical shutter may not be fast enough. One may use an acousto-optic modulator (AOM) or an electro optic modulator (EOM) to achieve the high speed. For example, an AOM can reach <NUM>-nanosecond rise/fall time, enough for the method and system in this embodiment. After the second shutter <NUM>, the beam size may be adjusted by a pair of relay lenses 113a and 113b. After the relay lenses 113a and 113b, the quarter wave plate 113c may facilitate to create circular polarization. The light then reaches the pairs of scanning mirrors (i.e. XY-scanning mirrors) <NUM> to direct the illumination light to the desired point one at a time. The light then passes a scan lens <NUM> and a tube lens (included in a microscope, not shown here) and the objective <NUM> of the microscope <NUM> to illuminate the targeted point of the sample S. An objective <NUM> with a high numerical aperture (NA) may be needed to have enough light intensity for photochemical reactions or photoconversion.

The output of image processing after the step S16 is an XY-coordinate array (2D array) of targeted points to be illuminated covering the interested regions defined by the user. The number of illumination points per field of view (FOV) can vary depending the user's criteria. In some circumstances, <NUM> points per FOV are possible. In this case, the output contains <NUM> pairs of floating point numbers. <FIG> are two images produced by the image processing of one example of the present disclosure to define the regions of stress granules.

Please now return to the procedures of this method. In step S17: the second processing module <NUM> transfers the coordinate array to a signal converter (such as a digital-analog convertor, DAC) <NUM> to convert to analog voltages.

In step S18: Analog voltages are transferred to XY scanning mirrors <NUM> to direct the illumination light to one targeted point.

In the step S17 and S18, for each pair of the XY coordinate, the corresponding angles of the two XY scanning mirrors <NUM> and the corresponding voltages to drive them to the position are calculated. Illumination of these <NUM> targeted points is performed one point at a time. The illumination order of these points may be optimized through trajectory planning to minimize the total time needed to visit all points. For example, trajectory planning can be performed by using the solution to the traveling salesman problem (for example, see The Traveling Salesman Problem: A Guided Tour of Combinatorial Optimization by Lawler et al). The <NUM> pairs of floating point numbers of voltages are then transferred through the second processing module <NUM> to the signal converter <NUM>, which converts the digital numbers into analog signals. The analog signals are then transferred to the XY scanning mirrors <NUM> and the XY scanning mirrors <NUM> are driven to point to the first targeted illumination point. The response time of a galvo can reach <NUM> microseconds.

In step S19: the second processing module <NUM> turns on the second shutter <NUM>.

In step S20: the second processing module <NUM> turns off the second shutter <NUM>.

In the step S19, the second processing module <NUM> controls the voltage of the second shutter <NUM> to turn it on. A specific voltage may be determined to reduce the illumination light source power to a desired level for photochemical reactions or photoconversion. If the power reduction cannot be reached by the second shutter, one may add a polarization beam splitter and a half wave plate to reduce the power. The illumination duration depends on the need of photochemical reactions or photoconversion. For example, <NUM> microseconds may be needed for a specific reaction. After this duration, in the step S20, the second processing module <NUM> turns the second shutter <NUM> off.

In the step S21, the system may check whether or not all targeted points are illuminated. In other words, if the targeted points are not fully illuminated yet, the procedure will go back to the step S18 and the second processing module <NUM> in this step will control the XY scanning mirrors <NUM> to direct to the next point, and the second shutter on/off (i.e. steps S19 and S20). Once all targeted points (e.g. those <NUM> targeted points) in the XY-coordinate array are illuminated, the procedure may then go to the next step. Together, these <NUM> points may take about <NUM> milliseconds.

In the step S22: the first processing module <NUM> controls movement of the stage <NUM> of the microscope <NUM> to the next field of view. In this step, the first processing module <NUM> then controls the movement of the stage <NUM> of the microscope <NUM> to the next (adjacent) FOV. The stage movement takes about <NUM> milliseconds.

In the step S23, the system will check whether or not all fields of view are processed. If all the fields of view of the sample S are processed, the whole procedure may come to an end. If not, the procedure will then go back to the step S12 to start another round of imaging, image processing, illumination, and stage movement after processes in each FOV. In other words, the same cycle of imaging, image processing, illumination, and stage movement is performed one FOV at a time until enough sample areas are illuminated. That is, many FOVs are illuminated to perform enough photochemical reactions or photoconversion. In general, <NUM> milliseconds may be enough for each cycle.

Moreover, for a specific problem where <NUM> femtomolar (fmol) of a photoconverted molecule is needed, one equivalently needs <NUM>×<NUM><NUM> copies of a molecule. Assuming that for each illumination point, the illumination is enough to photo convert <NUM> copies of that molecule. This is possible for the light resolution (about <NUM> nanometers in diameter). Hence, one needs to illuminate (<NUM>×<NUM><NUM>)/<NUM>, or <NUM>×<NUM><NUM> points. In the case where each FOV has <NUM> illumination points, one will need to illuminate a total of <NUM>×<NUM><NUM> FOVs. If spending <NUM> milliseconds for each FOV, one will need a total of <NUM>×<NUM><NUM> seconds, or <NUM> hours to perform enough reactions. This total time is comparable to a 3D printing process. Without optimizing the time of each step, the total time may become impractical for real problems.

Also, this disclosure also provides another embodiment which is another microscope-based system for image-guided microscopic illumination. The microscope-based system for image-guided microscopic illumination is substantially the same as that is described above. Please refer to <FIG> and <FIG>. In this embodiment, the microscope-based system <NUM> comprises a microscope <NUM>, an illuminating assembly <NUM>, an imaging assembly <NUM>, a first processing module <NUM> and a second processing module <NUM>. The microscope <NUM> comprises a stage <NUM>, an objective <NUM> and a subjective <NUM>, and the stage <NUM> is configured to be loaded with a sample S. Please further refer to both <FIG> and <FIG>, the illuminating assembly <NUM> comprises an illuminating light source <NUM>, and a pattern illumination device <NUM> including a second shutter <NUM>, at least one relay lens (such as the relay lens 113a and 113b), a quarter wave plate 113c, at least a pair of scanning mirrors <NUM> and a scan lens <NUM>. Alternatively, DMD or SLM can also be used as the pattern illumination device <NUM>. The imaging assembly <NUM> may comprise a camera <NUM>, an imaging light source <NUM>, a focusing device <NUM>, and a first shutter <NUM>. The camera <NUM> is mounted on the microscope <NUM>.

The major difference between the systems described in the previous embodiment and here is that the first processing module <NUM> here is coupled to the stage <NUM> of the microscope <NUM> and the imaging light source <NUM> and the first shutter <NUM> of the imaging assembly <NUM>. However, the second processing module <NUM> here comprises a memory unit <NUM> and is coupled to the camera <NUM>, the illuminating assembly <NUM>, and the first processing module <NUM>. In other words, in this embodiment, the camera <NUM> is controlled by the second processing module <NUM> instead of the first processing module (i.e. the computer) <NUM>. The camera <NUM> can be connected to the second processing module <NUM> through a Camera Link if a high speed of image data transfer and processing is required. The memory unit <NUM> can be a random access memory (RAM), flash ROM, or a hard drive, and the random access memory may be a dynamic random access memory (DRAM), a static random access Memory (SRAM), or a zero-capacitor random access memory (Z-RAM).

Hence, in the system <NUM> embodied here, it is operated as follows. The first processing module <NUM> controls the imaging assembly <NUM> and the second processing module controls <NUM> the camera <NUM> such that the camera <NUM> acquires at least one image of the sample S of a first field of view. The image or images are then automatically transmitted to the memory unit <NUM> to the second processing module <NUM>. Image processing is then performed by the second processing module <NUM> automatically in real-time based on a predefined criterion, so as to determine an interested region in the image and so as to obtain a coordinate information regarding to the interested region. Later, the second processing module <NUM> controls the illuminating assembly <NUM> to illuminate the interested region of the sample S according to the received coordinate information regarding to the interested region.

Because composition, variation or connection relationship to other elements of each detail elements of the microscope-based system <NUM> can refer to the previous embodiments, they are not repeated here.

Also, this disclosure also provides still another embodiment which is another microscope-based method for image-guided microscopic illumination. The microscope-based method for image-guided microscopic illumination is substantially the same as that is described above. Please also refer to <FIG>, <FIG> and <FIG>, the microscope-based method for image-guided microscopic illumination comprises the following steps through (a) to (d): (a) controlling the imaging assembly <NUM> by the first processing module <NUM> and triggering the camera <NUM> of the imaging assembly <NUM> by the second processing module <NUM> to acquire at least one image of the sample S of a first field of view, and the sample S is loaded on the stage <NUM> of the microscope <NUM>; (b) automatically transmitting the image or images of the sample S to the memory unit <NUM> of the second processing module <NUM>; (c) based on a predefined criterion, performing image processing of the sample S automatically in real-time by the second processing module <NUM> to determine an interested region in the image and to obtain a coordinate information regarding to the interested region; and (d) controlling the illuminating assembly <NUM> by the second processing module <NUM> to illuminate the interested region in the sample S according to the received coordinate information.

Please refer to <FIG>, which is a flow chart of the image-guided method according to this embodiment of the present disclosure. It is a more detailed example of the image-guided method described herein, but the present invention is not limited thereby. The process of the method depicted in <FIG> comprises the following steps S11' to S23'.

In brief, in the step S <NUM>', the operator moves the stage <NUM> of the microscope <NUM> to the starting position. In step S12', the first processing module <NUM> opens the first shutter <NUM>. In step S <NUM>', the second processing module <NUM> triggers the camera <NUM> to take an image of the sample S. In step S14', the first processing module <NUM> then closes the first shutter <NUM>. In step S15', the camera <NUM> transfers the image data to the memory unit (e.g. DRAM) <NUM> of the second processing module <NUM>. In step S16', the second processing module <NUM> and the memory unit (e.g. DRAM) <NUM> perform image processing and obtaining an XY-coordinate array targeted points to be illuminated (i.e. the "interested region" of the sample S). In step S17', the second processing module <NUM> transfers the coordinate array to the signal converter (such as DAC) <NUM> to convert to analog voltages. In step S18', analog voltages are transferred to XY scanning mirrors <NUM> to direct the illumination light to one targeted point. In the step S19', the second processing module <NUM> turns on the second shutter <NUM>. In step S20', the second processing module <NUM> turns off the second shutter <NUM>. In the step S21', the system may check whether or not all targeted points are illuminated. In other words, if the targeted points are not fully illuminated yet, the procedure will go back to the step S18' and the second processing module <NUM> in this step will control the XY scanning mirrors <NUM> to direct to the next point, and the second shutter on/off (i.e. steps S19' and S20'). Once all targeted points in the XY-coordinate array are illuminated, the procedure may then go to the next step. In the step S22', the first processing module <NUM> controls movement of the stage <NUM> of the microscope <NUM> to the next field of view. In the step S23', the system will check whether or not all fields of view are processed. If all the fields of view of the sample S are processed, the whole procedure may come to an end. If not, the procedure will then go back to the step S12' to start another round of imaging, image processing, illumination, and stage movement after processes in each FOV. In other words, the same cycle of imaging, image processing, illumination, and stage movement is performed one FOV at a time until enough sample areas are illuminated.

The following examples of the invention will describe the practical application.

Practical Example <NUM>. The system can be used to reveal the proteome of cell nuclei. First, cell nuclei are imaged by staining with an anti-nuclear pore complex proteins antibody. After imaging, image processing analysis is performed to generate a series of point coordinates to be scanned by the photosensitizing light source. And then, the galvanometric scanners are controlled to scan these point coordinates. This illumination leads to free radical release of a photosensitizer, and thus results in biotinylation of amino acids in the scanned region. A large number of fields are imaged and illuminated to have enough biotinylated samples. The biotinylated amino acids are then purified and mass spectrometry is performed to identify the proteome of cell nuclei. We were able to perform the proteomic analysis of biotinylated samples using this invention and identified nuclear proteins by this high-content photo-induced biotinylation as shown in Table <NUM> as below.

Conditions of the experiment in Table <NUM>: fixation: methanol fixation <NUM> hrs; lysis buffer: breast cancer lysis buffer: <NUM> Tris-HCl pH=<NUM>, <NUM> DTT, <NUM>% SDS, 1x EDTA-free PIC; binding buffer: diluted lysis buffer with <NUM>% SDS, <NUM> DTT <NUM> NaCl; beads: Pierce high capacity streptavidin agarose beads; elution: 1x sample buffer.

Practical Example <NUM>. The system can be used to identify proteins of stress granules in cells. First, stress granules are imaged by staining with a stress granule marker such as G3BP1. After imaging, image processing analysis is performed to generate a series of point coordinates to be scanned by the photosensitizing light source. We were able to develop an image processing algorithm that can detect locations of stress granules precisely for further proteomic studies as shown in <FIG>.

Practical Example <NUM>. The system can be used to distinguish the difference of proteins in cancer cells that possess cancer stem cell markers (e.g. CD44) and those that do not. To distinguish the differences of cancer cells and cancer stem cells, the protocol is same as above embodiment <NUM>, except that the stem cell marker is used for imaging.

Practical Example <NUM>. The system can be applied to live cells and model animals (in vivo). Live cell imaging or in vivo imaging instead of antibody staining is performed and photosensitizing illumination is performed based on images from live cells or animals.

The major advantage of this invention is the ability to perform high-content high-precision photo-induced processing guided by cell images. Thus, one can specifically process proteins, lipids, nucleic acids, or biochemical species in an area of interest based on user-defined subcellular features. The labeled proteins, lipids, or nucleic acids can also be purified and identified using mass spectrometry when proteomic analysis is of interest. The high-content and high-precision capability of this invention have not be achieved by prior similar techniques, so that it is currently the only available system to reach the speed and precision requirements needed for collecting enough biomolecular samples guided by imaging for proteomic studies or biomedical studies.

In one embodiment, this system can be used to perform photoablation or photobleaching. Photosensitizers can be used to generate reactive oxygen species (ROS). For example, Rose Bengal can be used to generate singlet oxygen, a kind of ROS. Singlet oxygen is highly reactive that can damage biomolecules in vivo to reach photobleaching or photoablation. This system can reach subcellular photoablation that cannot be reached before in numerous cells simultaneously during culture or even tissues and can be also applied to phototherarpy.

In one embodiment, this system can be used to perform photoregulation. For example, a method called directed bioorthogonal ligand tethering (BOLT) has been used to achieve selective inhibition of protein functions, such as to control phosphorylation in vivo by modified kinase. The system and method provided by the aforementioned embodiments can be used to reach subcellular photoregulation with a large number of cells.

Some embodiments of the invention, a kind of system that complete the process of photosensitizing and processing biomolecules automatically, have the following advantages and effects.

First, some embodiments not covered by the claims propose a photo-labeling method that are superior to the chemical labeling method APEX because the system and/or the process illustrated by some embodiments of the invention can more precisely specify the labeling features instead of labeling all with only the affinity criterion. One can designate the target and use this system to distinguish and label different group of proteins in the same area. Thus, it has an advantage in finding the proteomics based on morphology or other features that APEX cannot achieve.

Second, the system illustrated by some embodiments of the invention is superior to STOMP because this invention implements the high-content capability that STOMP cannot achieve due to its limited labeling speed. The speed of STOMP will be too slow for low abundant proteins, while this invention can extract enough samples for proteins that have low copy numbers in each cell within a short period of time such as several hours. The system illustrated by some embodiments of the invention has the speed and specificity advantages due to integrated image processing, stage scanning, and galvanometric scanner control.

Third, some embodiments not covered by the claims have preferred reagents that can label many groups of proteins. Compare with chemical-labeling method such as miniSOG that only can label several groups of proteins; the system and the method illustrated by some embodiments of the invention can labels all the proteins in a specific area. Furthermore, they can be designed by software to label the area with a complicated shape.

In summary, the present disclosure provides a microscope-based system for image-guided microscopic illumination. The microscope-based system utilizes two independent processing modules (i.e., the first processing module and the second processing module) which simultaneously controls the imaging assembly for taking an image of a sample and the illuminating assembly for illuminating the sample. Moreover, the second processing module is in communication with the first processing module and receives the coordination of the targeted points on the sample (i.e. the "interested region" in the image of the sample and processed by the first processing module) so as to rapidly control the illuminating assembly to illuminate targeted points on the sample. Hence, the image-guided systems of this disclosure enable a high-content process to illuminate varying patterns through a large number of fields of view consecutively.

Claim 1:
A microscope-based system (<NUM>) for image-guided microscopic illumination of varying patterns on a sample through a plurality of fields of view of the sample, comprising:
a microscope (<NUM>) comprising an objective (<NUM>) and a stage (<NUM>);
an imaging assembly (<NUM>) comprising a controllable camera (<NUM>) and an imaging light source (<NUM>);
an illuminating assembly (<NUM>) comprising a pattern illumination device (<NUM>) and an illuminating light source (<NUM>) which provides an illuminating light through an illuminating light path to illuminate the sample (S); and
a processing module (13a) which is coupled to the microscope (<NUM>), the imaging assembly (<NUM>), and the illuminating assembly (<NUM>),
wherein the processing module (13a) is configured to perform the following cyclical procedure:
- control the imaging assembly (<NUM>) to illuminate the sample (S) with the imaging light source (<NUM>) such that the camera acquires at least one image of a field of view of the sample (S),
- process the at least one image automatically in real-time based on a predefined criterion so as to determine targeted points in the at least one image and so as to obtain a coordinate information of the targeted points,
- control the pattern illumination device (<NUM>) and the illuminating light source (<NUM>) of the illuminating assembly (<NUM>) to illuminate the field of view with a pattern corresponding only to the targeted points of the sample (S) according to the coordinate information of the targeted points, and
- after all the targeted points having been illuminated by the illuminating assembly (<NUM>), move the stage (<NUM>) to a subsequent field of view.