Patent Description:
This patent document relates to systems, devices and techniques for particle sorting in fluid, including flow cytometry devices and techniques and applications in chemical or biological testing and diagnostic measurements.

Document <CIT> discloses a system and method of cytometry which include presenting a single sperm cell to at least one laser source configured to deliver light to the sperm cell in order to induce bond vibrations in the sperm cell DNA, and detecting the signature of the bond vibrations. The bond vibration signature is used to calculate a DNA content carried by the sperm cell which is used to identify the sperm cell as carrying an X-chromosome or Y-chromosome. Another system and method may include flowing cells past at least one QCL source one-by-one using a fluid handling system, delivering QCL light to a single cell to induce resonant mid-IR absorption by one or more analytes of the cell, and detecting, using a mid-infrared detection facility, the transmitted mid-infrared wavelength light, wherein the transmitted mid-infrared wavelength light is used to identify a cell characteristic. Document <CIT> discloses a method of cryopreserving sperm cells including the steps of: adjusting the concentration of the sperm cells in a solution; adding a cryoprotectant and a protein source to the sperm cells in a solution; loading the sperm cells into a container; cooling the sperm cells to a holding temperature; maintaining the sperm cells at the holding temperature for a period; cooling the sperm cells to a temperature approaching a critical temperature zone, wherein the critical temperature zone is a zone at which ice crystal formation and changes in osmotic pressure would damage the sperm cells; cooling the sperm cells through the critical temperature zone at a cooling rate that is faster than the cooling rate used to cool the sperm cells from the holding temperature to the temperature approaching the critical temperature zone; and immersing the sperm cells in liquid nitrogen. Document <CIT> discloses a system for sortingtarget particles from a flow of particles. The system has a microscope, a lightsource, a CCD camera, microfluidic chip device withmicrofluidic channels, a detection apparatus for detecting thetarget particles with predefined specific features, a response generating apparatus for generating a signal in response to the detection of tharget particles, and an optical tweezing system for controlling movement of optical traps, the optical tweezing system is operably linked to the response signal. Document <CIT> discloses a system and method for the label-free analysis of cells including a purification device configured to receive a heterogeneous population of cells, the purification device temporarily trapping therein asubpopulation of cells from the heterogeneous population of cells and a cell analysis device positioned downstream of the purification device and configured to measure one or more cellular parameters including cell count, measured cell size, and/or cell morphology. In an alternative embodiment, thesubpopulation of cells is analyzed while they are trapped within the purification device. Document <CIT> discloses a method of detecting one or more diseased blood cells in a blood sample which includes introducing a blood sample into at least one inlet of anicrofluidic device comprising one or more linear channels wherein each channel has a length and a cross-section of a height and a width defining an aspect ratio adapted to isolate diseased blood cells along at least one portion of the cross-section of the channel based on reduced deformability of diseased blood cells as compared to non-diseased blood cells, wherein diseased blood cells flow along a first portion of the channel to a first outlet and non-diseased blood cells flow along a second portion of the channel to a second outlet. The one or more channels can be adapted to isolate cells along portions of the cross-section of the channel based on cell size. In some embodiments, the one or more channels can be spiral channels. Document <CIT> discloses a sampleanalyzer comprising a flow cell for forming a sample flow, a light source for irradiating the sample flow in the flow cell with near ultraviolet flight, and an imaging section for taking an image of a particle contained in the sample flow irradiated with the near ultraviolet light by the lightsource. A method for analyzing a sample is also disclosed. Document <CIT> discloses simultaneously collecting multimodal/multispectral images of a population of cells. Photometric and morphometric features identifiable in the images are used to separate the population of cells into subpopulations. Where the population of cells includes diseased cells and healthy cells, the images can be separated into a healthy subpopulation, and a diseased subpopulation. Where the population of cells does not include diseased cells, ratios of different cell types in patients not having a disease condition can be compared to the corresponding ratios in patients having the disease condition, enabling the disease condition to be detected. For example, blood cells can be separated into different types based on their images, and an increase in the number of lymphocytes, a phenomenon associated with chronic lymphocytic leukemia, can readily be detected. Document <CIT> discloses a method for verification of sorting of particles including receiving a first detection signal that is associated with optical characteristics of a particle in a first channel. A sorting channel of second channels is determined based on the first detection signal, thereby determining the sorting of the particle into the sorting channel based on the optical characteristics of the particle. A sorting signal for sorting the particle from the first channel into the sorting channel is transmitted. A second detection signal is received that is associated with the presence of the particle in the sorting channel. The sorting of the particle from the first channel into the sorting channel is verified based on the second detection signal. Document <CIT> discloses a device for sorting biological cells immersed in a flowing medium. The device comprises a holographic imaging unit comprising holographic imaging elements, a fluid handling unit comprising a plurality of microfluidic channels for conducting flowing medium along a corresponding holographic imaging element and comprising a microfluidic switch arranged downstream of an imaging region in the microfluidic channel for controllably directing each biological cell in the flowing medium into a selected one of a plurality of outlets. The device also comprises a processing unit adapted for real-time characterization of the holographic diffraction image obtained for each of the biological cells thereby taking into account a predetermined biological cell type signature. Document <CIT> describes an imaging cytometer. Light from a light source is focused by a focusing element to focused illumination spots at a sensing location, illuminating a cell as the cell traverses the sensing location. A collection lens collects light emanating from the cell and refocuses the collected light onto an array light sensor. The focusing element includes an array of microlenses having spherical surfaces. The system includes a processing unit that constructs a digital image of the cell based on signals produced by the array light sensor indicating the intensity and distribution of light falling on the array light sensor. The system characterizes cells using light emanating from the cells by fluorescence.

Flow cytometry is a technique to detect and analyze particles, such as living cells, as they flow through a fluid. For example, a flow cytometer device can be used to characterize physical and biochemical properties of cells and/or biochemical molecules or molecule clusters based on their optical, electrical, acoustic, and/or magnetic responses as they are interrogated by in a serial manner. Typically, flow cytometry use an external light source to interrogate the particles, from which optical signals are detected caused by one or more interactions between the input light and the particles, such as forward scattering, side scattering, and fluorescence. Properties measured by flow cytometry include a particle's relative size, granularity, and/or fluorescence intensity.

Cell sorting, including cell sorting at the single-cell level, has become an important feature in the field of flow cytometry as researchers and clinicians become more interested in studying and purifying certain cells, e.g., such as stem cells, circulating tumor cells, and rare bacteria species. Cell sorting can be achieved by various techniques.

Flow cytometry devices and systems can be implemented based on microfluidic technologies for research assays and diagnostics as well as for clinical applications. A microfluidic device is an instrument that can control the behavior of very small amounts of fluid (e.g., such as nL, pL, and fL) through channels with dimensions in relatively small dimensions, e.g., the sub-millimeter range. Microfluidic devices can be implemented to obtain a variety of analytical measurements including molecular diffusion values, chemical binding coefficients, pH values, fluid viscosity, molecular reaction kinetics, etc. Microfluidic devices can be built on microchips to detect, separate and analyze biological samples, which can also be referred to as a lab-on-a-chip. For example, a microfluidic device may use biological fluids or solutions containing cells or cell parts to diagnose diseases. Inside microfluidic channels of, for example, a microfluidic flow cytometer, particles including cells, beads, and macromolecules can be interrogated according to their optical, electrical, acoustic, and/or magnetic responses using flow cytometry techniques.

The technology disclosed in this patent document can be implemented to provide methods, devices and systems for producing images of particles in a flow system, and in specific configurations, the disclosed technology can be used for imaging particles in real time and subsequently sorting particles, including cells, based on the spatial information from the image. The disclosed techniques can be applied for producing cell images and sorting cells in flow cytometers. In applications, the disclosed technology can be used to detect and sort cells based on the fluorescent and/or scattering intensity by taking into account the spatial information such as the spatial distribution of fluorescence.

In implementations, for example, the disclosed systems possess the high throughput of flow cytometers and high spatial resolution of imaging cytometers, in which the cell images are produced at a fast enough rate to accommodate real-time cell sorting in a flow system based on physical and/or physiological properties of the cell, e.g., as opposed to just a detection event.

In some aspects, an image-based particle sorting system includes a particle flow device structured to include a substrate, a channel formed on the substrate operable to flow cells along a flow direction to a first region of the channel, and two or more output paths branching from the channel at a second region proximate to the first region in the channel; an imaging system interfaced with the particle flow device and operable to obtain image data associated with a cell when the cell is in the first region during flow through the channel; a data processing and control unit in communication with the imaging system, the data processing and control unit including a processor configured to process the image data obtained by the imaging system to determine one or more properties associated with the cell from the processed image data and to produce a control command based on a comparison of the determined one or more properties with a sorting criteria, in which the control command is produced during the cell flowing in the channel and is indicative of a sorting decision determined based on one or more cellular attributes ascertained from the image signal data that corresponds to the cell; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the cell into an output path of the two or more output paths based on to the control command, in which the system is operable to sort each of the cells during flow in the channel within a time frame of <NUM> or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

In some aspects, a method for image-based particle sorting includes obtaining image signal data of a cell flowing through a channel of a particle flow device; processing the image signal data to produce an image data set representative of an image of the cell; analyzing the produced image data set to identify one or more properties of the cell from the processed image data; evaluating the one or more identified properties of the cell with a sorting criteria to produce a control command to sort the cell based on one or more cellular attributes ascertained from the image signal data corresponding to the cell during cell flow in the particle flow device; and directing the cell into one of a plurality of output paths of the particle flow device based on to the control command.

In some aspects, an image-based particle sorting system includes a particle flow device structured to include a substrate, a channel formed on the substrate operable to flow particles along a flow direction to a first region of the channel, and two or more output paths branching from the channel at a second region proximate to the first region in the channel; an imaging system interfaced with the particle flow device and operable to obtain image data associated with a particle when the particle is in the first region during flow through the channel; a data processing and control unit in communication with the imaging system, the data processing and control unit including a processor configured to process the image data obtained by the imaging system to determine one or more properties associated with the particle from the processed image data and to produce a control command based on a comparison of the determined one or more properties with a sorting criteria; and an actuator operatively coupled to the particle flow device and in communication with the actuator, the actuator operable to direct the particle into an output path of the two or more output paths based on to the control command, in which the system is operable to sort each of the particles during flow in the channel within a time frame of <NUM> or less from a first time of image capture by the imaging system to a second time of particle direction by the actuator.

In some aspects, a method for image-based sorting of a particle includes obtaining image signal data of a particle flowing through a channel of a particle flow device; processing the image signal data to produce an image data set representative of an image of the particle; analyzing the produced image data set to identify one or more properties of the particle from the processed image data; producing a control command by evaluating the one or more identified properties with a sorting criteria; and directing the particle into one of a plurality of output paths of the particle flow device based on to the control command.

The above and other aspects of the disclosed technology and their implementations and applications are described in greater detail in the drawings, the description and the claims.

Some existing flow cytometer devices and systems detect and sort cells based on the fluorescence and/or scattering intensity without taking into account the spatial information such as the spatial distribution of fluorescence. There has been some advancements in the development of techniques to produce cell images for flow cytometers with high throughput and high spatial resolution. However, the cell image has not been produced in a fast enough rate to be useful for applications, in particular for cell sorting in a flow system, e.g., due to the required amount of computation to generate the cell image. As such, existing state of the art for cell sorting capabilities are "detection only" systems, and fail to "screen" the detected cells based on meaningful and nuanced criteria.

In applications, the disclosed technology can be implemented in specific ways in the form of methods, systems and devices for image-based cell sorting in flow cytometry using (a) real-time image acquisition of fast travelling cells by efficient data processing techniques utilizing mathematical algorithms implemented with FPGA and/or GPU and concurrent (b) "gating" techniques based on spatial characteristics of the particles as the sorting criteria from the real-time acquired images. Unlike traditional flow cytometers that use fluorescent intensities of chosen biomarkers as criteria for cell sorting, the methods, systems and devices in accordance with the disclosed technology allow for various user-defined gating criteria containing spatial features.

Examples of image-based gating criteria include cell contour, cell size, cell shape, size and shape of internal cell structures such as the cell nucleus, fluorescent patterns, fluorescent color distribution, etc. For example, users can draw the cells they wish to separate and the system will perform accordingly. With such unique capabilities, users such as researchers can track many important biological processes by localization of certain proteins within cytosolic, nuclear, or cell membrane domains and subdomains. Because every cell population has some degree of heterogeneity at a genomic (e.g., mutations, epigenetics) or environmental (e.g., asymmetric division, morphogen gradients) level, identification and extraction of single-cells according to their unique spatial features are envisioned to contribute significantly to the fields of immunology, tumor heterogeneity, stem cell differentiation, and analysis of neurons.

In some embodiments, an image-based particle sorting system includes a particle flow device, such as a flow cell or a microfluidic device, integrated with a particle sorting actuator; a high-speed and high-sensitivity optical imaging system; and a real-time cell image processing and sorting control electronic system. For example, an objective of the disclosed methods, systems and devices is to perform the entire process of (i) image capture of a particle (e.g., cell), (ii) image feature reconstruction from a time-domain signal, and (iii) making a particle sorting decision and sorting operation by the actuator within a latency of less than <NUM> to fulfill the needs for real-time particle sorting. In some implementations described herein, the total latency is less than <NUM> (e.g., <NUM>), in some implementations, the total latency is less than <NUM> (e.g., <NUM>), and in some implementations, the total latency is less than <NUM> (e.g., <NUM>). For implementations of cell sorting, for examples, the disclosed methods, systems and devices are able to image, analyze and sort cells by image features specific to life cycles, protein localization, gene localization, DNA damages, and other cellular properties, which can be connected to different diseases or pathogens.

<FIG> shows a diagram of an example embodiment of an image-based particle sorting system <NUM> in accordance with the present technology. The system <NUM> includes a particle flow device <NUM>, an imaging system <NUM> interfaced with the particle flow device <NUM>, a data processing and control unit <NUM> in communication with the imaging system <NUM>, and an actuator <NUM> in communication with the data processing and control unit <NUM> and operatively coupled to the particle flow device <NUM>. The particle flow device <NUM> is structured to include a channel <NUM> in which particles flow along a flow direction to an interrogation area <NUM> where image data are obtained by the imaging system <NUM> for each particle in the interrogation area <NUM>. The data processing and control unit <NUM> is configured to process the image data and determine one or more properties associated with the particle to produce a control command for sorting of the particle. The control command is provided to the actuator <NUM>, which is interfaced with the particle flow device <NUM> at a sorting area of the device <NUM>, such that the actuator operates to sort the particular particle into an output channel corresponding to the control command. The system <NUM> implements image-based sorting of the particles in real-time, in which a particle is imaged by the imaging system <NUM> in the interrogation area and sorted by the actuator <NUM> in the sorting area in real time and based on a determined property analyzed by the data processing and control unit <NUM>.

The system <NUM> is user-programmable to sort each particle based on user-defined criteria that can be associated with one or more of a plurality of properties exhibited by each individual particle analyzed in real time by the data processing and control unit <NUM>. Some example user-defined criteria include, but are not limited to, an amount and/or size of sub-features of or on the individual particle (e.g., sub-particles attached to living cells, including particles engulfed by cells or attached to cells); morphology of the individual particle; and/or size of the individual particle. In this manner, the system <NUM> is able to evaluate and sort particles by properties, such as properties of living cells, including sorting by cellular physiological functionalities (e.g., particle or substance uptake by a cell, or particle engulfment by a cell), by cell damage, by localization of proteins, or by other cellular properties.

<FIG> shows a block diagram of an example embodiment of the data processing and control unit <NUM>. In various implementations, the data processing and control unit <NUM> is embodied on one or more personal computing devices, e.g., including a desktop or laptop computer, one or more computing devices in a computer system or communication network accessible via the Internet (referred to as "the cloud") including servers and/or databases in the cloud, and/or one or more mobile computing devices, such as a smartphone, tablet, or wearable computer device including a smartwatch or smartglasses. The data processing and control unit <NUM> includes a processor <NUM> to process data, and memory <NUM> in communication with the processor <NUM> to store and/or buffer data. For example, the processor <NUM> can include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the process <NUM> can include a field-programmable gate-array (FPGA) or a graphics processing unit (GPU). For example, the memory <NUM> can include and store processor-executable code, which when executed by the processor <NUM>, configures the data processing and control unit <NUM> to perform various operations, e.g., such as receiving information, commands, and/or data, processing information and data, such as from the imaging system <NUM>, and transmitting or providing processed information/data to another device, such as the actuator <NUM>. To support various functions of the data processing and control unit <NUM>, the memory <NUM> can store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor <NUM>. For example, various types of Random Access Memory (RAM) devices, Read Only Memory (ROM) devices, Flash Memory devices, and other suitable storage media can be used to implement storage functions of the memory <NUM>. In some implementations, the data processing and control unit <NUM> includes an input/output (I/O) unit <NUM> to interface the processor <NUM> and/or memory <NUM> to other modules, units or devices. In some embodiments, such as for mobile computing devices, the data processing and control unit <NUM> includes a wireless communications unit, e.g., such as a transmitter (Tx) or a transmitter/receiver (Tx/Rx) unit. For example, in such embodiments, the I/O unit <NUM> can interface the processor <NUM> and memory <NUM> with the wireless communications unit, e.g., to utilize various types of wireless interfaces compatible with typical data communication standards, which can be used in communications of the data processing and control unit <NUM> with other devices, e.g., such as between the one or more computers in the cloud and the user device. The data communication standards include, but are not limited to, Bluetooth, Bluetooth low energy (BLE), Zigbee, IEEE <NUM>, Wireless Local Area Network (WLAN), Wireless Personal Area Network (WPAN), Wireless Wide Area Network (WWAN), WiMAX, IEEE <NUM> (Worldwide Interoperability for Microwave Access (WiMAX)), <NUM>/<NUM>/LTE cellular communication methods, and parallel interfaces. In some implementations, the data processing and control unit <NUM> can interface with other devices using a wired connection via the I/O unit <NUM>. The data processing and control unit <NUM> can also interface with other external interfaces, sources of data storage, and/or visual or audio display devices, etc. to retrieve and transfer data and information that can be processed by the processor <NUM>, stored in the memory <NUM>, or exhibited on an output unit of a display device or an external device.

<FIG> show diagrams of an image-based cell sorting microfluidic system <NUM> in accordance with some embodiments of the image-based particle sorting system <NUM>. The system <NUM> includes a microfluidic device <NUM> to flow particles through an optical interrogation channel for sorting, an imaging system <NUM> to obtain image data of the particles in an illumination area of the interrogation channel, a data processing and control system <NUM> to process the obtained image data in real time and determine a sorting command, and an actuator <NUM> to gate the particles in the microfluidic device <NUM> based on the determined sorting command.

As shown in <FIG>, the microfluidic device <NUM> structured to include a substrate 213having a passage forming a microfluidic sample channel <NUM>, and microfluidic sheath channels <NUM> that converge upon the sample channel <NUM>. In implementations, for example, the sample channel is configured to carry particles (e.g., cells) suspended in a fluid that flows in a flow direction, and the sheath channels <NUM> are configured to provide sheath flow of fluid to hydrodynamically focus the suspended particles in the fluid prior to flowing through an illumination area <NUM> of the microfluidic device <NUM>. In some embodiments, for example, the substrate 213can be formed in a bulk material, e.g., such as polydimethylsiloxane (PDMS), that is bonded to a base substrate, e.g., a glass base substrate or base substrate of other material.

The imaging system <NUM> of the system <NUM> includes a light source <NUM>, e.g., a laser, to provide an input or probe light at the illumination area <NUM> of the microfluidic device <NUM>, and an optical imager <NUM> to obtain images of the illuminated particles in the illumination area <NUM>. The example optical imager <NUM>, as shown in <FIG>, includes an objective lens <NUM> (e.g., of a microscope or other optical imaging device) optically coupled to a spatial filter (SF) <NUM>, an emission filter (EF) <NUM>, and a photomultiplier tube (PMT) <NUM>. In some implementations, for example, the imaging system <NUM> includes one or more light guide elements <NUM> to direct the input light at the illumination area <NUM> of the microfluidic device <NUM>. In the example shown in <FIG>, the light guide element <NUM> includes a dichroic mirror arranged with the light source <NUM> and the optical imager <NUM> to direct the input light at the illumination area <NUM>.

In some implementations of the imaging system <NUM>, the light source <NUM> (e.g., the laser) is configured to produce a fluorescent excitation signal that is incident upon the illumination area <NUM> to cause a fluorescent emission by the particles. The optical imager <NUM> captures the optical output fluorescent emission signal such that an image of the particle can be generated.

The data processing and control system <NUM> of the system <NUM> is configured in communication with the optical imager <NUM>, e.g., via the PMT, to rapidly process the imaged particles and produce a sorting control based on the processed image of each particle imaged in real-time. In some implementations of the data processing and control unit <NUM>, a FPGA processing unit is configured to rapidly process the image signal data received by the optical imager <NUM>. An example of such implementations can include a Virtex-II (xc2v3000) FPGA platform in conjunction with a compiler Xilinx <NUM>, which can be provided via a chassis Crio-<NUM> from National Instrument to execute algorithms in accordance with data processing methods of the present technology.

The actuator <NUM> of the system <NUM> is configured in communication with the real-time data processing and control system <NUM> to gate the particle flowing in a gate area <NUM> of the sample channel <NUM> into two or more output channels <NUM> of the microfluidic device. In some embodiments, for example, the distance between the illumination area <NUM> and the gate area <NUM> can be in a range of <NUM> to <NUM>. In implementations, the actuator <NUM> receives the sorting command from the data processing and control system <NUM> in real time, such that the imaging system <NUM> and data processing and control system <NUM> operate to capture and process the image of each particle while flowing through the illumination area <NUM> so that the actuator <NUM> receives and executes the sorting command to gate each particle accordingly. For example, in some implementations, the actuator <NUM> includes a piezoelectric actuator coupled to the substrate <NUM> to produce deflection that causes the particle to move in a particle direction in the gate area <NUM> that directs the particle along a trajectory to one of the two or more of output channels <NUM>.

In implementations of the system <NUM>, for example, the suspended single cells are hydrodynamically focused in the microfluidic channel by sheath flow, ensuring that the cells travel in the center of the fluidic channel at a uniform velocity. A fluorescence emission is detected by the PMT <NUM> in a wide-field fluorescence microscope configuration, such as in the example shown in <FIG>. In this example, to accommodate the geometry of the microfluidic device, the laser beam is introduced to the optical interrogation by <NUM>-degree reflection by a miniature dichroic mirror (DM) <NUM> positioned in front of a <NUM>× objective lens <NUM> (e.g., with NA=<NUM>, working distance=<NUM>).

<FIG> shows an example embodiment of the spatial filter (SF) <NUM> that is inserted in the detection path right at the image plane of the optical imager <NUM> of the imaging system <NUM>. The spatial filter design includes a pattern having a plurality of slits positioned apart. In some embodiments, for example, the spatial filter <NUM> includes a pattern of openings having uniform dimensions, in which the pattern of openings encodes a waveform on the received light by the optical imager. In the example shown in <FIG>, the pattern includes ten <NUM> by <NUM> slits positioned apart in the way of one is immediately after another. In some embodiments, for example, the spatial filter <NUM> includes a pattern of openings having a varying longitudinal and transverse dimensions with respect to the flow direction across the microfluidic channel, such that a waveform is encoded by the optical imager to allow optical detection of a position of a particle in at least two dimensions in the illumination area <NUM> of the microfluidic channel <NUM>. An example of two-dimensional spatially-varying spatial filter is provided in <CIT> entitled "OPTICAL SPACE-TIME CODING TECHNIQUE IN MICROFLUIDIC DEVICES".

Referring back to <FIG>, although the imaging system <NUM> shows only one PMT for detection of fluorescent signal, it is understood that more PMTs can be added to the optical imager <NUM>, and, if necessary, more excitation laser beams added to the light source <NUM>, to produce multi-color fluorescent cell images.

The real-time data processing and control system <NUM> includes a control loop system that is implemented using a field-programmable gate-array (FPGA) to process the captured images and produce the corresponding sorting commands for each particle. The data processing and control system <NUM> includes image processing and image-based particle sorting algorithms executable by the FPGA to provide automated cell image generation and accurate sorting by the system <NUM>. For example, once a sorting decision is made by the FPGA algorithm, the example on-chip integrated piezoelectric lead-zirconate-titanate (PZT) actuator <NUM> is actuated to apply fluidic pressure at the nozzle structure at the sorting junction. In some implementations, the example PZT actuator <NUM> is configured to execute the fluid displacement operation in response to the sorting command in a time frame at or less than <NUM> per operation. For example, the fluid displacement by the PZT actuator in less than <NUM> exhibits single cell hydrodynamic manipulation capabilities with a high throughput.

Other examples of features of a particle flow device and/or an actuator that can be used in example embodiments of the devices, systems, and methods in accordance with the disclosed technology are provided in <CIT> entitled "FLUIDIC FLOW CYTOMETRY DEVICES AND PARTICLE SENSING BASED ON SIGNAL ENCODING". Other examples of features of an optical imaging system that can be used in example embodiments of the devices, systems, and methods in accordance are provided in <CIT> which is a U. National Stage Application filing under <NUM> U. § <NUM> based on <CIT> entitled "IMAGING FLOW CYTOMETRY USING SPATIAL-TEMPORAL TRANSFORMATION" and published as <CIT>.

<FIG> show example results of a PMT signal and the fluorescence cell image constructed from the PMT signal, processed by the example data processing and control system <NUM>, using an example algorithm described in <FIG> and executed by the example FPGA. <FIG> shows an example time-domain PMT output signal of fluorescent light from a A549 cell stained with CellTrace CFSE. <FIG> shows the corresponding processed (e.g., resized) fluorescence image depicting the real size of the cell. The numbered regions segmented by dashed lines in the figures demonstrate the correspondence between the time-domain signal and the resulting image. Size is labeled in <FIG>.

The spatial resolution of the restored image in x- (transverse) direction depends on the number of the slits on the spatial filter, and in y- (cell-travelling) direction depends on the sampling rate and cell flow rate. In the original image restored by the imaging flow cytometer (shown in <FIG>), the effective pixel size is <NUM> in x-direction and about <NUM> in y-direction. The recovered image represents a <NUM> by <NUM> area in the object plane in the microfluidic channel.

<FIG> shows a diagram of an example embodiment of a method <NUM> for image-based sorting of particles. Implementations of the method <NUM> can be performed by the various embodiments of the image-based particle sorting system <NUM> in accordance with the present technology, such as the system <NUM> and the system <NUM>.

The method <NUM> includes a process <NUM> to capture image data, by the imaging system <NUM>, of a particle flowing through a channel in the particle flow device <NUM>, e.g., at the interrogation area <NUM>. For example, the process <NUM> can include continuously capturing images at a predetermined rate or varying rates, which may be based on the particle flow speed in the channel. In some implementations, the process <NUM> includes receiving, at a controller of the imaging system <NUM>, an image capture command from the data processing and control unit <NUM> to affect the image capture protocol to obtain the image data. For example, the data processing and control unit <NUM> can change one or more parameters of the image capture protocol executed by the imaging system <NUM> in real-time implementations of the system.

The image capture rate can be associated with the data volume of single particles. For example, the system can determine the image capture rate and/or other parameters of image capture protocol based at least in part on (a) the particle flow speed of the particle flow device <NUM> and (b) the electronic sampling rate, e.g., depending on what resolution is desired, which can be used to determine the data volume of single particles. The image capture rate can be associated with the data recording/computing capabilities of the data processing and control unit <NUM> and/or a controller (e.g., processor) of the imaging system <NUM>. For example, higher speed analog-to-digital conversion (ADC) and larger memory can increase the image capture rate. The image capture protocol including image capture and processing parameters can be selected based at least in part on that the design of the spatial filter, e.g., as in some implementations the optical output from the spatial filter can affect the processing of the algorithm complexity. Notably, optical factors, such as magnification, optical filters, imaging mode, etc. typically have little, if any, influence on the image capture speed, but are significant in obtaining the input data that is processed to produce the overall results, e.g., determination of the particle properties that are evaluated for determining a sorting decision or other analyses.

The method <NUM> includes a process <NUM> to receive the image data, at the data processing and control unit <NUM> and from the imaging system <NUM>, in which the received data is associated with a particle imaged by the imaging system <NUM> flowing in the channel of the particle flow device <NUM>. For example, in some implementations of the process <NUM>, the data processing and control unit <NUM> receives time domain signal data, e.g., optical intensity, from one or more PMTs of the imaging system <NUM> for each particle imaged in the illumination area on the particle flow device <NUM>. The method <NUM> includes a process <NUM> to process the data (e.g., image signal data) to produce an image data set representative of an image of the particle imaged by the imaging system <NUM>. For example, in some implementations of the process <NUM>, the received image signal data is pre-processed by filtering the data, reconstructing an image based on the filtered data using an image reconstruction algorithm in accordance with the present technology, and/or resizing the reconstructed image, e.g., in which the reconstructed image is converted to binary image data. The method <NUM> includes a process <NUM> to analyze the produced image data set to identify one or more features of the imaged particle based on predetermined criteria and to determine a sorting command based on the one or more identified features. In some implementations of the method, the method <NUM> includes a process <NUM> to first process the received image signal data to detect a presence of the particle in the illumination area, and then process the data in accordance with the process <NUM> to produce the image data set that is subsequently analyzed in accordance with the process <NUM>. The method <NUM> includes a process <NUM> to provide the sorting command to the actuator <NUM>. The method <NUM> includes a process <NUM> to execute, by the actuator <NUM>, the sorting command to direct the particle flowing in sorting area of the channel of the particle flow device <NUM> to the corresponding output channel of the device <NUM>. For example, in implementations of the method <NUM>, the imaging system <NUM> and the data processing and control system <NUM> implement the processes <NUM> and <NUM>, <NUM>, <NUM>, and <NUM>, respectively, in real time, such that the actuator <NUM> receives and executes the sorting command to direct the particles accordingly, e.g., gate each particle to the appropriate output channel of the device <NUM>, within a diminutive time period from the image capture (process <NUM>) to the actuation of particle gating (process <NUM>). For example, in some implementations of the process <NUM>, the actuator <NUM> includes a piezoelectric actuator coupled to the flow device <NUM> to produce hydrodynamic deflection in the fluid that causes the particle to move in a particle direction along a desired trajectory to enter the desired output channel.

In some embodiments, the data processing and control unit <NUM> includes software modules corresponding to one or any combination of the processes <NUM>, <NUM>, <NUM> and <NUM> stored in the memory <NUM> and executable by the processor <NUM> to implement the processes <NUM>, <NUM>, <NUM> and/or <NUM>. <FIG> shows a diagram of example data processing modules of the systems in accordance with the present technology. For example, the data processing and control unit <NUM> can include a Particle Detection module <NUM>, a Process Image module <NUM>, and a Make Sorting Decision module <NUM>. As shown by the example diagram of <FIG>, once a particle (e.g., cell) is detected by implementing the algorithms executable by the Particle Detection module <NUM>, the data processing and control unit <NUM> proceeds to implement the Process Image module <NUM> if a particle (e.g., cell) is detected, otherwise, it returns to keep recording images, e.g., via PMT readout. By extracting parameters from cell images and comparing those values to pre-defining sorting criteria, e.g., performed by the Process Image module <NUM>, the Make Sorting Decision module <NUM> is implemented to determine whether to trigger the actuator <NUM> or not. In some embodiments, the data processing and control unit <NUM> includes a Record Image module <NUM> to control, at least partially, the imaging system <NUM> to implement the process <NUM>, e.g., which can include initiating and/or adapting the image capture protocol or settings of the imaging system. For example, in some implementations such as with the imager <NUM>, when the Record Image module <NUM> is operable to control the recording of the PMT readout in a certain length.

In some implementations, the parameters associated with particle properties (e.g., cell properties) are extracted based on the type of parameter or parameters to be extracted. For example, for different morphology parameters, the process to extract the parameters associated with cell morphological properties can include at least some of the following techniques. The process can include analyzing the image area, for example, including determining the number of pixels with value "<NUM>" in the binary image. The number of pixels is image area. The process can include analyzing the perimeter, for example, including determining the number of pixels on the image contour, e.g., as the image contour is detected. The process can include analyzing the diameter in the x direction, for example, including (e.g., in a binary image) determining the number of "<NUM>" pixels in each row that represents the x direction. For example, the largest count can be determined as the diameter in x direction. The process can include analyzing the diameter in the y direction, for example, including (e.g., in a binary image), determining the number of "<NUM>" pixels in each column that represents the y direction. For example, the largest count can be determined as the diameter in y direction.

<FIG> shows a diagram of an example implementation of the process <NUM>, e.g., implementation of the Particle Detection module <NUM>, to detect cells for image-based cell sorting. In some implementations, for example, the Particle Detection module <NUM> searches for a section of the time-domain PMT signal that has integrated fluorescence intensity, referred to herein as "brightness", that is larger than a preset threshold. For example, in some implementations, the cell-travelling speed in the flow cell or microfluidic device is <NUM>/s, and a <NUM> sampling rate is used, such that the particle sorting and image processing algorithm integrates every consecutive <NUM> data points to obtain the value of brightness. Every time when the brightness is larger than a first preset threshold, e.g., threshold1, this means a cell is entering the optical system's field of view in the image plane. The algorithm determines the time derivative of brightness, which is then compared to a second preset threshold, e.g., threshold2. If the time derivative is smaller than threshold2, the algorithm considers that the cell is well within the field of view, and the example process <NUM> continues to determine an identified feature or features of the detected cell, e.g., implementation of the Process Image module <NUM> is consequently initiated.

<FIG> shows a brightness-time plot depicting an example cell detection process implementation based on the processing of a time-domain PMT signal associated with a single cell traveling through the illumination area of the system. As shown in the plot, a cell is detected to enter the imaging area (e.g., interrogation area <NUM>) based on a threshold (e.g., brightness ≥ <NUM>) just after the <NUM> time marker, and the system captures image signal data until the cell is detected to depart the imaging area based on the threshold (e.g., brightness ≤ <NUM>) after the <NUM> time marker.

<FIG> shows a diagram of an example implementation of the process <NUM>, e.g., implementation of the Process Image module <NUM>, to determine an identified feature or features of the detected cell for image-based cell sorting. In the example shown in <FIG>, the process <NUM> is implemented after some aspects of the process <NUM>, e.g., particularly after analysis of image signal data to determine if a cell is detected or not. For example, in some implementations as shown in <FIG>, after the partial analysis of the received image signal data to determine the presence of a cell, the Process Image module <NUM> pre-processes the image signal data (e.g., time-domain PMT signal) including filtering or other signal processing techniques. For example, in some implementations, the received image signal data is low-pass filtered to eliminate high frequency noise. In some examples, a <NUM>th order hamming window for low-pass filtering is used, which is an example of a particular low pass filter applicable. The Process Image module <NUM> can implement other features of the process <NUM> including image reconstruction techniques (e.g., based on an example algorithm described herein, including Equation (<NUM>)) to turn the time-domain signal into an image that represents <NUM>-dimensional spatial distribution of the cells' fluorescence. After that, the reconstructed cell image is resized for fidelity purpose and converted into binary image. For binary images, for example, an optional open filter technique is applied to remove spurious noise, and the Process Image module <NUM> detects one or more cell features (e.g., the cell wall or membrane detection) after the open filter is applied. In some implementations, the open filter can be applied after cell feature(s) have been detected. The Process Image module <NUM> extracts parameters based on the detected cell features that describes aspects of the cell, such as cell morphology, which can be compared to the values from user's preset sorting criteria for subsequent determination of the sorting command (e.g., gating command).

Example implementations of the method <NUM> using the system <NUM> are described below for an example study demonstrating image-based cell sorting. In the example study, the imaging system <NUM> included a <NUM> mW <NUM>-nm laser (e.g., iBeam-SMART, Toptica) that has an oval beam shape with Gaussian energy distribution, which is collimated, focused, and then expanded to illuminates an area of <NUM> (x-direction) by <NUM> (y-direction) to form the illumination area <NUM> on the microfluidic device <NUM>. The fluorescence passing the miniature dichroic mirror with <NUM> cutoff wavelength (e.g., ThorLabs) and the scattering light were collected through a <NUM>×, <NUM>. 55NA objective lens (e.g., Mituyoyo). The light intensity signal in each channel was acquired by a PMT (e.g., H9307-<NUM>, Hamamatsu).

Image reconstruction techniques, implemented by the Process Image module <NUM>, for example, were used for spatial-to-temporal transformation of image data that can be mathematically formulated in the following: <MAT> where S(t) is the measured PMT signal, Cell is the two-dimensional cell fluorescence or scattering intensity profile, F(x, y) is the characteristic function of the spatial filter, I(x, y) is the intensity profile of laser illumination, y is along the cell-travelling direction and x is along the transverse direction, and M is the magnification factor of the optical system pertaining to the flow cytometer. As the cell travels in the microfluidic channel at a speed v, the image projected onto the spatial filter, e.g., the example SF <NUM> shown in <FIG>, travels at an effective speed of Mv. In the simplest case to explain the principle and solving for Cell in Equation (<NUM>), one can choose F(x, y) to be a series of small slits (e.g., <NUM> by <NUM> rectangular slits) represented approximately in Equation (<NUM>) and I(x, y) to be a constant from a laser beam of uniform intensity (i.e., top-hat beam profile): <MAT> where x = <NUM>, <NUM>,. , N is the number of row in the spatial filter, L is the distance between two slits that transmits fluorescence. As a result, for example, the cell image can be constructed from the following relation: <MAT>.

An example image reconstruction technique that can be implemented by the Process Image module <NUM>, for example, includes determining two variables: the number of sampling points in each peak, and the starting point of the time-domain image signal data (e.g., PMT signal) for the cell flowing. For example, since cells are not travelling at perfectly uniform speed, number of sampling points in each peak slightly changes; and since cell travelling speed as well as cell position in the image area slightly change (e.g., <NUM> by <NUM> image area set based on the example spatial filter), the starting point of the time-domain PMT readout also changes. In the example image reconstruction technique, m is referred to starting point of PMT readout, n is referred to number of points in each peak. Based on cell speed variation, for example, n ranges from <NUM> to <NUM>. Accordingly, m ranges from <NUM> to <NUM>-10n. The example image reconstruction technique sweeps m and n to assure the best combination to reconstruct cell image. Summation of intensities at starting point of each peak is calculated for every combination in the sweeping, for example. The combination with smallest summation is the right answer for image reconstruction. After m and n are calculated, they can be used to reconstruct both bright field and fluorescence images since signal recorded by both PMTs are synchronized. The example image reconstruction technique calculates values based on Equation (<NUM>), <MAT>.

<FIG> shows an example data plot depicting the results of an implementation of the image reconstruction technique, e.g., which shows example results of the search for the best combination of starting point and number of sampling points for each peak. The black "*" symbols shown in the data plot represent starting points of each peak found by the reconstruction algorithm.

Generally, with the spatial filter described above (e.g., spatial filter <NUM> of <FIG>) inserted at the image plane, fluorescence from different parts of the cell will pass different slits at different times. As a result, the waveform of the fluorescent signal from the PMT includes a sequence of patterns separated in time domain, and each section of the signal in the time domain corresponds to the fluorescent signal generated by each particular regime of the cell. After the light intensity profile over each slit is received, the cell image of the entire cell can be constructed by splicing all the profile together. In the example embodiment of the system <NUM> shown in <FIG>, the spatial filter contains ten <NUM> by <NUM> rectangular slits positioned in sequence. With a <NUM>× objective lens (M = <NUM>), for example, the filter design allows construction of the fluorescent or scattering image of a travelling cell, e.g., no larger than <NUM> by <NUM>, using the algorithm including the Equation (<NUM>), which requires a minimum amount of computations and is suitable for high-throughput, real-time image-based cell classification and sorting. For example, using a <NUM>×/<NUM>. 55NA objective lens, <NUM> sampling rate for acquiring PMT signal, and <NUM>/s cell-travelling speed that is given by <NUM>µL/min sample flow rate and <NUM>µL/min sheath flow rate, the effective size of the pixel in y-direction is <MAT> which is smaller than the Rayleigh Criterion, thus resulting in a diffraction-limited resolution in y-direction. Here R is the sampling rate of PMT readout in this calculation. It is noted that other implementations parameters can be used, such as: cell-travelling at speed around <NUM>/s (e.g., given by the flow rate); image area set to be <NUM> by <NUM> (e.g., PMT readout based on fluorescence image of <NUM> peaks); sampling rate of <NUM> kSamples/s, such that each peak includes <NUM> sampling points.

For the example study, the design of spatial filter was drawn in AutoCAD and printed to a transparency mask at <NUM>,<NUM> dots per inch (dpi). A layer of negative photoresist (e.g., NR9-1500PY, Futurrex, Inc. ) was spun at <NUM>,<NUM> rotations per minute (rpm) on a <NUM>-inch glass wafer. The wafer was heated on a hot plate at <NUM> for <NUM> minutes then exposed to UV light (e.g., EVG620NT, EV Group) through the transparency mask. Post UV exposure, the wafer was baked at <NUM> for another <NUM> minutes before development in RD6 (e.g., Futurrex, Inc. ) for <NUM> seconds. A film of <NUM> thick aluminum was sputtered onto the glass wafer. After metal lift-off, the patterns of the spatial filter were formed and the glass wafer was diced into <NUM> by <NUM> pieces. To help hold the spatial filter in the flow cytometer system, the spatial filter having ten <NUM> by <NUM> slits was mounted to a sample holder fabricated by 3D printing method.

For the example study, HEK293T human embryonic kidney cell samples were transfected with pEGFP-GR plasmids (e.g., Addgene). After continuous culturing for <NUM> days, <NUM> dexamethasone (e.g., Sigma-Aldrich) was added to the culture media. After incubation for <NUM> minutes, the HEK293T cells were harvested, fixed by <NUM> % paraformaldehyde, washed and resuspended in 1X phosphate buffered saline (PBS). Before every imaging experiment, the suspension was diluted in PBS to a concentration of <NUM> cells/µL.

The example embodiment of the image-based cell sorting system <NUM> used in the example study included a spatial mask including <NUM> slits and utilized a cell flow speed of around <NUM>/s. The image area was set to be <NUM> by <NUM>. In this design, the PMT signal from a cell's fluorescence appears to be <NUM> separated peaks, and number of sampling points of each peak is around <NUM>. However, the speed of different cells is not perfectly uniform, so the number of sampling points for each cell slightly changes, and the starting point of each peak also needs to be determined. As such, the data processing algorithms were configured to account for such varying cell speeds, i.e., to search for the start point and number of sampling points for each peak within a certain range, so that the cell image can be successfully reconstructed. Based on the variations in cell flowing speed, the number of sampling points of each peak typically ranged from <NUM> to <NUM>. So the total number of points in the cell image is from <NUM> to <NUM>. For each cell, once the Particle Detection module <NUM> determined that there comes a cell, the PMT signal in a length of <NUM> sampling points was recorded. For example, to assure the best combination of staring point and number of sampling points for each peak, the algorithm sweeped the number of sampling points from <NUM> to <NUM> and start points of each peaks accordingly. For each combination in the sweeping, the sum of intensities at all starting points was calculated, and the combination with smallest summation was chosen as correct answer for image reconstruction.

shows a data plot depicting the fluorescence intensity signal for a cell based on an example image reconstruction. The example result are indicative of searching for the best combination of staring point and number of sampling points for each peak.

<FIG> shows an example of a reconstructed image of a cell, e.g., in which the criteria was used the determine boundaries of a cell using fluorescent area to produce the image of the cell. For the open filter, which is an erosion followed by a dilation, a <NUM> by <NUM> neighborhood was used. In the step of cell wall or membrane detection, all the pixels in a binary cell image are scanned. For the pixels that have non-zero intensity, the example algorithm checked all nine pixels in its <NUM> by <NUM> neighborhood. By counting its neighboring pixels that have non-zero intensities, for example, if the counted number is larger than <NUM> and smaller than <NUM>, this pixel was determined as a pixel on cell wall or membrane.

The example technique was implemented in at least in-part using an FPGA. Table <NUM> shows the round-up approximate latency results for each step in the example implementation of the data processing algorithm, in which the total latency is well within <NUM>. The example data processing algorithm is flexible to be implemented on multiple platforms, including in parallel. For example, utilizing the parallel processing power of graphics processing unit (GPU), the algorithm can also be implemented in GPU, by, for example, either the CUDA architecture by Nvidia or the OpenCL by AMD. The image soring algorithm has a much shorter runtime combining the parallel processing power of GPU, so the sorting throughput is further improved. Although the formulation of producing cell images is described in the form of example methods in this patent document, it is worth noticing that the algorithm can somewhat differ when using different spatial filter designs than the previously described <NUM>-slit spatial mask. The overall working procedure of the system, however, remains the same.

Table <NUM> shows example data depicting the performance of modules in the example FPGA design, e.g., in which latencies is represented in time (ms).

Example sorting results from the example study are described. For example, to demonstrate the feasibility of the example real-time image-based cell sorter system, sorting tests on a mixture of cells possessing fluorescent cytoplasm and cells possessing fluorescent nucleus were performed. The normal human embryonic kidney cells, HEK293T cells, after transfection with pEGFP-GR plasmids, are expressing GFP that can be excited by <NUM> laser and has an emission peak at <NUM> in their cytoplasm. After <NUM> dexamethasone treatment for <NUM> hour, the fluorescence translocates to the cell nucleus from the cytoplasm region.

<FIG> show example fluorescence microscope images of transfected but not drug treated cells (<FIG>) and transfected and drug treated cells (<FIG>). The images of <FIG> represent an area of <NUM> by <NUM>. As shown by the example representative microscopic images of the transfected HEK293T cells without drug treatment (<FIG>), and those of treated cells (<FIG>), the drug treated cell image has smaller fluorescence area, even though the magnitude of the fluorescence intensity of both treated and untreated cells has a quite wide range.

As shown in the images of <FIG>, the left column of both rows shows the fluorescence image; the middle column shows the bright field image; and the right column shows the overlay images. Row <NUM> shows the sample cell with fluorescence distributed in cytoplasm (e.g., untreated with dexamethasone). Row <NUM> shows the sample cell with fluorescence distributed in nucleus (treated with dexamethasone). The cells are pEGFP-GR plasmids transfected HEK293T human embroynic kidney cells.

<FIG> show fluorescence cell images taken by the image-based cell sorter system of transfected but not drug treated cells (<FIG>) and transfected and drug treated cells (<FIG>). The example images represent an area of <NUM> by <NUM>.

<FIG> shows a histogram of the example calculated fluorescence area of all events and of the sorted cells from the example study.

In the example study, the sorting criteria was preset to be <NUM><NUM>, so the drug treated cells are sorted by the example system. In this example study, due to the variation in cell size, nucleus size, and drug uptake by cells, it is possible that not all drug treated cells were sorted, but, the sorted cells all have fluorescence only in nucleus, so the purity is secured.

As demonstrated in the example implementation of an example embodiment of the system <NUM>, the disclosed systems and techniques provide a high-throughput flow cytometer with cell sorting capabilities based on single cells' fluorescent and light scattering images. Realizing the cell image capture, process and sorting actuation in FPGA and/or GPU, the example results show that the afore-described design provides an overall processing latency at millisecond level.

<FIG> shows a diagram of an image-based cell sorting microfluidic system <NUM> in accordance with some embodiments of the image-based particle sorting system <NUM>. The system <NUM> includes a microfluidic device <NUM> to flow particles through an optical interrogation channel for sorting, an imaging system <NUM> to obtain image data of the particles in an illumination area of the interrogation channel, a data processing and control system <NUM> to process the obtained image data in real time and determine a sorting command, and an actuator <NUM> to gate the particles in the microfluidic device <NUM> based on the determined sorting command. In some implementations of the example system <NUM>, the microfluidic device <NUM> and the actuator <NUM> can include the microfluidic device <NUM> and the actuator <NUM>, respectively.

The imaging system <NUM> of the system <NUM> includes a light source <NUM>, e.g., a laser, to provide an input or probe light at the illumination area of the microfluidic device <NUM>, and an optical imager <NUM> to obtain images of the illuminated particles in the illumination area <NUM>. The example optical imager <NUM>, as shown in <FIG>, includes an objective lens <NUM> (e.g., of a microscope or other optical imaging device) optically coupled to a spatial filter (SF) <NUM>, emission filters (EF) 1125A and 1125B, and photomultiplier tubes (PMT) <NUM>. In the example implementation shown in <FIG>, the imaging system <NUM> includes dichroic mirrors (DM) 1129A and 1129B, in which DM 1129A is arranged with the light source to direct the input light at the illumination area on the microfluidic device <NUM> and DM 1129B is arranged with the optical imager <NUM> in the optical path to direct a portion of the optical output signal to the PMT 1126B via the EF 1125B, while the undirected portion of the optical output signal proceeds to PMT 1126A via the EF 1125A. In some implementations, the light source <NUM> (e.g., the laser) is configured to produce a fluorescent excitation signal that is incident upon the illumination area to cause a fluorescent emission by the particles. The optical imager <NUM> captures the optical output fluorescent emission signal at the PMTs 1126A and 1126B, such that an image of the particle can be generated.

In example implementations of the system <NUM>, suspended single cells are hydrodynamically focused in the sorting channel of the microfluidic device <NUM> by sheath flow, ensuring that the cells travel in the center of the fluidic channel at a uniform velocity. For example, both fluorescence emission and bright field signal can be detected by the multiple photomultiplier tubes, e.g., PMTs 1126A and 1126B. To accommodate the geometry of the microfluidic device, a <NUM> laser beam from the laser <NUM> is introduced to the optical interrogation by <NUM>-degree reflection by a miniature dichroic mirror 1129A positioned in front of a <NUM>× objective lens (e.g., NA=<NUM>, working distance=<NUM>). In some example embodiments, the system <NUM> includes an optical light source <NUM> (e.g., LED, such as the example <NUM> LED shown in <FIG>), which can be placed at the opposite side of the channel to generate bright field images, and in which the light can be focused at the laser illumination position. The spatially coded filter, e.g., SF <NUM>, is inserted at the image plane in the detection path. To route desired emission band to their respective PMTs, the dichroic mirror 1129B splits the fluorescent and bright field light collected by the objective lens based on spectrum. The spatial resolution of the reconstructed image is determined by the filter. Resolution in x- (transverse) direction depends on the number of slits on the filter, and in y- (cell-travelling) direction depends on the sampling rate and cell flow speed. In the example embodiment of the system <NUM> shown in <FIG>, the effective pixel size is <NUM> in x-direction and about <NUM> in y-direction.

<FIG> shows a flow diagram of an example image processing implementation of an example embodiment of the system <NUM>. The data processing and control unit <NUM> includes image processing modules to process fluorescence and bright field images captured by the imaging system <NUM>. An example image processing module, e.g., to implement aspects of the processes <NUM> and <NUM> of the method <NUM>, for example, includes pre-processing the received image signal data from the multiple PMTs 1126A and 1126B. In such implementations, the pre-processing techniques can be performed after a detection of a cell is determined based on analysis of the received image signal data. In the example shown in <FIG>, the PMT signals of fluorescence and bright field images are low-pass filtered to eliminate high frequency noise. In some implementations, for example, a 10th order hamming window is used for low-pass filtering. The example image processing module is configured to execute an image reconstruction algorithm to reconstruct both bright field and fluorescence image from time-domain PMT signal, e.g., into <NUM> dimensional images. Since bright field and fluorescence signals are generated by the same slits, they are synchronized by the data processing and control unit <NUM>. In some implementations, the image reconstruction algorithm is launched once for both bright field and fluorescence images. In some implementations, for example, reconstructed images are resized to <NUM>×<NUM> pixels. In some implementations, for example, grayscale images are converted to binary image based on intensity threshold. In some implementations, for example, binary images are filtered by open filter to eliminate spurious noise. In some implementations, for example, cell contour algorithm is launched to detect contour of both binary images. In some implementations, for example, morphology parameters are extracted based on processed images and sorting decision is made based on extracted parameters.

<FIG> shows a flow diagram depicting example data through the example image processing steps shown in <FIG>. In the diagram, the example image processing module produces a reconstructed and resized image of the detected individual cell including image contour features, which allow for cellular feature parameters such as cell morphology to be extracted from the produced image.

The algorithm is implemented on FPGA (National Instrument cRIO-<NUM>). Table <NUM> shows example round-up approximate latency results for each step implemented by the example image processing module of the data processing and control unit <NUM>. Image processing for bright field and fluorescence images is executed in parallel. As shown in the table, the total latency of image processing is around <NUM> for this example.

Table <NUM> shows example data depicting the performance of modules in the example FPGA design for example implementations of the system <NUM>, e.g., in which latencies is represented in time (ms).

By decreasing the time latency of image processing, for example, the sorting throughput can be improved. With more powerful FPGA that has more computational resource, for example, the image processing module can be further paralleled to improve time latency. The example image processing algorithm can also be implemented on a GPU alternatively or additionally (e.g., in parallel processing with FPGA), e.g., such as by either the CUDA architecture by Nvidia or the OpenCL by AMD as examples. Because the example image processing algorithm is data-parallel, by utilizing the parallel processing power of GPU, the processing speed can be much accelerated, so that a much higher throughput can be achieved.

In some implementations, for example, the data processing and control unit <NUM> can be configured to evaluate extracted morphology parameters, e.g., using Receiver Operating Characteristics (ROC). The top parameters are selected for real-time sorting.

In some implementations, a method for evaluating extracted morphology parameters using ROC technique includes the following. For example, the ROC technique can be applied to processed data from the image data obtained by flow cell samples through the system. For example, the cell images can be separated into two subsets, e.g., in some instances by manual identification. As an example, the two subsets can be translocated cells (e.g., a sorted group) and un-translocated cells (e.g., an unsorted group). For each cell, morphology parameters are extracted. The ROC technique can include generating a distribution for each parameter of both subsets. The technique includes generating the curve, which is the ROC curve for the distribution. The technique includes calculating the area under the curve (AUC), i.e., the integration of the curve. The technique includes evaluating how the parameters apply for the classification, e.g., the parameter with the larger AUC can be selected as the best suitable parameter for classification.

<FIG> shows an example distribution plot for subsets using an example ROC technique to evaluate extracted parameters such as cell morphology parameters. In the example shown, the ROC includes TP, FP, FN, TN (TP is true positive, FP is false positive, FN is false negative, TN is true negative). The example curve represents the ROC curve. The parameter with larger AUC is better for the classification.

After parameters are selected for sorting, Support Vector Machine (SVM) can be used to generate nonlinear hyperplane as gating criterion using the selected parameters.

As an example, after the images are separated into subsets, e.g., such as the two example subsets, the selected parameters are calculated for each image. Each image corresponds to a n dimensional vector (n is the number of selected parameters). The example SVM technique is implemented to generate the boundary in n dimensional space that separates the subsets. For example, in making sorting decision, the selected parameters for each cell can be calculated, e.g., so as to know the cell is on which side of the boundary (e.g., which subsets does the cell belong to). Then, the sorting decision can be made.

Example extracted morphology parameters are shown in Table <NUM>.

Example implementations of the method <NUM> using the system <NUM> are described below for example studies demonstrating image-based cell sorting, including sorting based on protein translocation, sorting based on cell life cycle, and sorting based on number of beads bonded on the cell membrane of cells.

One example study included the sorting of pEGFP-GR plasmids translocated HEK-297T Human embryonic kidney cells, e.g., an implementation of sorting based on protein translocation. The example study demonstrated the capability to identify and sort pEGFP-GR plasmids translocated HEK-297T Human embryonic kidney cells from translocated and un-translocated mixtures. In the study, HEK-293T Human embryonic kidney cells were transfected with GR-GFP and separated into <NUM> plates. One plate of cells was untreated so the fluorescence stays in cytoplasm. The other plate of cells was treated with drug so the fluorescence migrates from cytoplasm to nucleus. Both types of cells were mixed and flow through the system, in which only translocated cells were sorted and collected based on implementation of the example method.

In the example study, the recorded PMT signal was processed by the data processing and control unit, e.g., executing the algorithms implemented by Matlab code. Based on the PMT signal, both fluorescence and bright field images of each cell were reconstructed, and morphology parameters of each cell were extracted and recorded. The extracted morphology parameters were used for supervised machine learning to generate criteria for real-time image-based cell sorting.

In the example study, the reconstructed cell images were separated into two subsets by manual identification. One subset of cells was un-translocated cells with fluorescence in cytoplasm, the other subset of cells was translocated cells with fluorescence in nucleus.

<FIG> show cell images of un-translocated cells and translocated cells captured by the example system and reconstructed by the Matlab code. <FIG> shows images of un-translocated cells, and <FIG> shows images of translocated cells. In the images of <FIG>, the image on the left is the fluorescence image of the cell, the image in the middle is bright field image of the cell, and the image on the right is overlay image with detected bright field contour.

In the example study, morphology parameters were evaluated based on the two annotated subsets using Receiver Operating Characteristics (ROC). The top <NUM> parameters were selected for real-time sorting. For example, the top <NUM> parameters in this case were Fluorescence area/Bright field area (e.g., area ratio), Fluorescence perimeter/Bright field perimeter (e.g., perimeter ratio), and Fluorescence area.

In the example study, sorting criteria using the three selected parameters was employed for real-time sorting of the cells in the system. A three dimensional nonlinear hyperplane separating the two cell subsets based on the selected top three parameters was formed by Support Vector Machine (SVM).

<FIG> shows an example of a hyperplane exhibiting separation of the two cell sets (e.g., translocated cells and un-translocated cells) for implementations of the sorting criteria.

The two types of cells were mixed <NUM>:<NUM> and diluted to <NUM>/µL using phosphate buffered saline (PBS). The mixed sample were flowed through the example image-based sorting system and sorted based on the real-time module. The cells traveled at a speed of <NUM>/s that is given by <NUM>µL/min sample flow rate and <NUM>µL/min sheath flow rate. The example imager included a microscope with a CCD camera that was used to capture both fluorescence and bright field images of collected cells.

<FIG> show cell images of un-translocated cells and translocated cells captured by the example system, via the microscope. <FIG> shows the microscope images of un-translocated cells, and <FIG> shows the microscope images of translocated cells. In the images of <FIG>, the image on the left is the fluorescence image of the cell imaged by the microscope, the image in the middle is bright field image of the cell by the microscope, and the image on the right is overlay image.

One example study included the sorting of MDCK Madin-Darby Canine Kidney Epithelial cells at G2/M stage, e.g., an implementation of sorting based on cell life cycle. In the study, MDCK Madin-Darby Canine Kidney Epithelial cells were fixed and cell nucleus was stained with Propidium Iodide (PI). Fixed and stained MDCK cells were flowed through the example image-based sorting system, and only cells at G2/M phase were sorted and collected.

Similar to the example study for sorting based on protein translocation, the recorded PMT signals were processed to reconstruct cell images and extract morphology parameters. The reconstructed cell images were separated into two subsets by manual identification. One subset of cells was at G1 phase. At G1 phase, constituents of nucleus are confined in nucleus membrane. The other subset of cells was at G2/M phase. At G2/M phase, nucleus membrane breaks down and constituents of nucleus are distributed within the cell.

<FIG> show cell images of G1 phase cells and G2/M phase cells captured by the example system and reconstructed by the Matlab code. <FIG> shows images of cells at G1 phase, and <FIG> shows images of cells at G2/M phase. In the images of <FIG>, the image on the left is the fluorescence image of the cell, the image in the middle is bright field image of the cell, and the image on the right is overlay image with detected bright field contour.

Similar to the example study for sorting based on protein translocation, morphology parameters were evaluated using ROC, and a three dimensional nonlinear hyperplane was formed by SVM. The top <NUM> morphology parameters are Fluorescence area/Bright field area (e.g., area ratio), Fluorescence area and Fluorescence perimeter/Bright field perimeter (e.g., perimeter ratio).

<FIG> shows an example of a hyperplane exhibiting separation of the two cell sets (e.g., cells at G1 phase and cells at G2/M) for implementations of the sorting criteria.

MDCK cells were diluted to <NUM>/µL using PBS. The sample was flowed through the example system and sorted based on the real-time module. The cells traveled at speed <NUM>/s that is given by <NUM>µL/min sample flow rate and <NUM>µL/min sheath flow rate. The example imager included a microscope with a CCD camera that was used to capture both fluorescence and bright field images of collected cells.

<FIG> show cell images of G1 phase cells and G2/M cells captured by the example system, via the microscope. <FIG> shows the microscope images of cells at phase G1, and <FIG> shows the microscope images of cells at G2/M phase. In the images of <FIG>, the image on the left is the fluorescence image of the cell imaged by the microscope, the image in the middle is bright field image of the cell by the microscope, and the image on the right is overlay image.

One example study included the sorting of HEK-297T Human embryonic kidney cells based on number of beads bond with the cells, e.g., an implementation of sorting based on number of beads bonded on the cell membrane of cells. In the study, HEK-297T Human embryonic kidney cells were bonded with fluorescence beads and stained with CFSE kits. The fluorescence of cells was at <NUM> and fluorescence of beads is at <NUM>. Fluorescence signals at two wavelengths at routed by dichroic mirror and detected by two PMTs. Cells were sorted based on number of bonded beads. The image processing module for this application was modified accordingly to process the images.

<FIG> shows a flow diagram of an example image processing implementation for sorting based on number of beads bonded on the cell membrane of cells implemented by an example embodiment of the system <NUM>. The data processing and control unit <NUM> includes image processing modules to process fluorescence and bright field images captured by the imaging system <NUM>. An example image processing module, e.g., to implement aspects of the processes <NUM> and <NUM> of the method <NUM>, for example, includes pre-processing the received image signal data from the multiple PMTs 1126A and 1126B. As shown in the diagram, for example, the fluorescence signals at both wavelengths were low-pass filtered. The fluorescence signal at <NUM> was used for image reconstruction in this example study. The reconstructed beads image was resized to <NUM>×<NUM> pixels. The top-hat transform was implemented to remove image background. In this example implementation, a <NUM>×<NUM> pixels neighborhood was used for top-hat transform. In this example implementation, a grayscale image was converted to binary image. Morphology parameters were extracted. In this example, image area was chosen as sorting criterion as beads have relatively uniform size. In some implementations, for example, morphology parameters are extracted based on processed images and sorting decision is made based on extracted parameters.

Table <NUM> shows the total latency of real-time image processing is <NUM>. The example modules were implemented in using FPGA for beads counting.

<FIG> shows examples of grayscale cell images processed by the image processing module with different number of beads. As shown in <FIG>, the images in the left column are fluorescence images of beads, the images in the middle column are fluorescence image of the cells, and the images in the right column are overlay images.

<FIG> shows a histogram of beads image area for different number of beads.

Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing unit" or "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.

Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices.

It is intended that the specification, together with the drawings, be considered exemplary only, where exemplary means an example. Additionally, the use of "or" is intended to include "and/or", unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment.

Various embodiments described herein are described in the general context of methods or processes, which may be implemented in one embodiment by a computer program product, embodied in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers in networked environments. A computer-readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), Blu-ray Discs, etc. Therefore, the computer-readable media described in the present application include non-transitory storage media. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein.

For example, one aspect of the disclosed embodiments relates to a computer program product that is embodied on a non-transitory computer readable medium. The computer program product includes program code for carrying out any one or and/or all of the operations of the disclosed embodiments.

Claim 1:
A method for cell sorting, comprising:
obtaining (<NUM>, <NUM>) image signal data of a cell flowing through a channel of a particle flow device;
processing (<NUM>) the image signal data to produce an image data set representative of an image of the cell, wherein the processing the image signal data includes detecting the presence of the cell prior to producing the image data set representative of the image of the cell, wherein the detecting the presence of the cell includes:
calculating a brightness value associated with a magnitude of signal intensity of the image signal data;
comparing the brightness value with a first threshold;
determining a time derivative value of the brightness when the brightness value exceeds the first threshold;
evaluating that the time derivative value is smaller than a second threshold; and
when the time derivative value is smaller than the second threshold, continuing to process the image signal data;
analyzing (<NUM>) the produced image data set to identify one or more properties of the cell from the processed image data;
evaluating the one or more identified properties of the cell with a sorting criteria to produce a control command (<NUM>) to sort the cell based on one or more cellular attributes ascertained from the image signal data corresponding to the cell during cell flow in the particle flow device; and
directing (<NUM>) the cell into one of a plurality of output paths of the particle flow device based on the control command.