Patent ID: 12233406

Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale.

DETAILED DESCRIPTION

As used herein, “a sample” or “an injection sample” is a substance for injection into an adherent cell unless otherwise stated. The sample is usually prepared in liquid form for injection into the cell. The sample may be a liquid, an emulsion, or a mixture of liquid and minute solids.

Disclosed herein is a system for automatically providing microinjection of a sample to a plurality of adherent cells with provision of a high throughput in microinjection. Particularly, the high throughput is achievable by providing multiple micropipettes for microinjection instead of providing only a single micropipette such that the system only needs to move an adherent cell on the Petri dish to the nearest one of the multiple micropipettes for microinjection. A travel time is thus shortened, leading to an increase in the microinjection throughput.

The disclosed system is exemplarily illustrated with the aid ofFIG.1, which depicts, in accordance with an exemplary embodiment of the present invention, a schematic diagram of an automated microinjection system19for automatically injecting a plurality of adherent cells2with a sample.

In the automated microinjection system19shown inFIG.1, a specific case of using two micropipettes for microinjection is considered. Using two micropipettes in the system19has practical significance over using more than two micropipettes in that using two micropipettes increases a microinjection throughput over using a single micropipette but incurs less manufacturing cost than using more than two micropipettes. Furthermore, the algorithm in determining an optimized injection sequence for the plurality of adherent cells2is less computationally complex for the two-micropipette case than the case with more than two micropipettes. Despite the practical significance of the two-micropipette case, the present invention is not limited to the system configuration of using only two micropipettes; the disclosed system may use more than two micropipettes for microinjection during operation.

In operating the system19for microinjection, the plurality of adherent cells2is disposed on a Petri dish6. The Petri dish6may be a 35 mm glass-bottom Petri dish, where the glass bottom enables the plurality of adherent cells2to be observed (by a microscope) from the Petri-dish bottom.

A reference vertical direction900is defined as shown inFIG.1. Herein in the specification and appended claims, positional and directional words such as “above,” “below,” “higher,” “upper,” “lower,” “top,” “bottom” and “horizontal” are interpreted with reference to the reference vertical direction900.

The automated microinjection system19comprises a motorized stage8, a plurality of motorized micromanipulators5,7, and one or more computers40. The motorized stage8is used for two-dimensionally moving the Petri dish6. The plurality of motorized micromanipulators5,7is used for manipulating a plurality of micropipettes1,3. The plurality of micropipettes1,3is used for injecting the sample to the plurality of adherent cells2. An individual micromanipulator (viz., micromanipulator5or7) is configured to hold and manipulate one micropipette in the plurality of micropipettes1,3. The one or more computers40are used for controlling the system19.

Although it is sufficient for the individual micromanipulator (e.g., micromanipulator5) to move its micropipette (e.g., micropipette1) up and down for piercing a certain adherent cell while the motorized stage8positions this adherent cell right below the micropipette (e.g., micropipette1), it is preferable and often that the individual micromanipulator (e.g., micromanipulator5) is capable of moving its micropipette (e.g., micropipette1) three-dimensionally for offering operational convenience in microinjecting the aforesaid certain adherent cell. In practice, the speed of the individual micromanipulator to three-dimensionally move its micropipette is considerably lower than a running speed of the motorized stage8in moving the Petri dish6. For example, a maximum running speed of 50 mm/s is achievable by the motorized stage8whereas the individual micromanipulator may only have a maximum speed of 5 mm/s. Thus, minimizing a total travel time of positioning each cell in the plurality of adherent cell2to a corresponding micropipette for achieving the goal of increasing a resultant microinjection throughput is preferably carried out by optimizing an injection sequence for microinjecting the plurality of adherent cells2by minimizing a total travel time of the motorized stage8in visiting the plurality of adherent cells2.

In the system19, the one or more computers40are configured to control the motorized stage8to sequentially visit respective cells in the plurality of adherent cells2according to an injection sequence. Advantageously, the injection sequence is an optimized one selected by minimizing a total distance traveled by the motorized stage8to sequentially visit the respective cells such that each of the respective cells is visited once by one micropipette selected from the plurality of micropipettes1,3. That is, the optimized injection sequence is obtained by minimizing an objective function that is the total traveled distance of the motorized stage8in sequentially visiting all the respective cells in the plurality of adherent cells2. In practical operations of the system19, typically the respective cells are randomly distributed on the Petri dish6. Under this situation of random distribution, using the plurality of micropipettes1,3reduces the minimized total distance over using a single micropipette, thereby increasing a throughput of microinjection achievable by the system19.

To facilitate automated microinjection, preferably the system19is further equipped with a gas pressure provider13connectable to an individual micropipette for controllably forcing out the sample present in the individual micropipette. Additionally, the one or more computers40are further configured as follows. When a certain cell in the plurality of adherent cells2is visited by a certain micropipette manipulated by a corresponding micromanipulator, the one or more computers40control the corresponding micromanipulator to cause the aforesaid micropipette to pierce into the aforesaid cell, and control the gas pressure provider13to cause the aforesaid micropipette to inject a predetermined amount of the sample into the aforesaid cell.

In determining the optimized injection sequence, a first requisite is to obtain or determine locations of the respective cells in the plurality of adherent cells2. Determination of these locations is most conveniently accomplished by first acquiring a digital image of the plurality of adherent cells2, and then using an image processing technique to locate the respective cells. Usually, an optical microscope12is installed in the system19for viewing the Petri dish6that is disposed on the motorized stage8. Furthermore, a camera11is optically coupled to the optical microscope12for capturing an image (referred to as a cell image for the sake of convenience) of the plurality of adherent cells2as viewed through the optical microscope12.

With the presence of the camera11, the one or more computers40control the camera11to capture the cell image, and determine XY locations of the respective cells from the cell image. The one or more computers40also obtain XY locations of micropipette tips of the plurality of micropipettes1,3. The XY locations of micropipette tips may be obtained from the cell image, or independently from one or more other images taken by the camera11. According to the XY locations of micropipette tips of the plurality of micropipettes1,3and the XY locations of the respective cells, the one or more computers40determine the optimized injection sequence.

In certain embodiments, the one or more computers40are further configured to, in determining the XY locations of the respective cells, use a deep learning-based algorithm to perform image segmentation on the respective cells. “Deep learning” means to use an artificial neural network composed of hundreds, even thousands of layers, to automatically “learn” useful representations from raw data with multiple levels of abstraction. One such artificial neural network for medical image segmentation is a U-net framework. For details of the U-net framework, see O. RONNEBERGER, P. FISCHER and T. BROX, “U-Net: convolutional networks for biomedical image segmentation,” inMedical Image Computing and Computer-Assisted Intervention—MICCAI2015, ser. Lecture Notes in Computer Science, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Springer International Publishing, 2015, pp. 234-241.

Usually, the optical microscope12does not have a FOV sufficiently large enough to cover the entire Petri dish6. Hence, the plurality of adherent cells2is located within the FOV viewable by the optical microscope12. Other adherent cells outside the FOV are not deemed to be in the plurality of adherent cells2. In normal practice, the system19processes these other adherent cells after the system19completes microinjection of the plurality of adherent cells2. Preferably, the one or more computers40are further configured as follows. After microinjection of the plurality of adherent cells2is completed, the one or more computers40controls the motorized stage8to move such that the optical microscope12originally viewing a first segment (viz., a first region) of the Petri dish6switches to viewing a second segment (viz., a second region) thereof. The first segment contains the plurality of adherent cells2. The second segment contains a next plurality of adherent cells for microinjection after completion of microinjection of the plurality of adherent cells2.

The optical microscope12may be an ordinary one or may be a special-purpose microscope. In certain embodiments, the optical microscope12is a fluorescence microscope for detecting fluorescence emitted by cells as well as observing the cells, especially living cells.

In the art, an inverted microscope is popular for observing living cells under microinjection because a micropipette is located above a Petri dish. Preferably, the optical microscope12is an inverted microscope. In certain embodiments, the optical microscope12is an inverted fluorescence microscope. Nonetheless, the present invention is not limited to the case that the optical microscope12is an inverted microscope; the optical microscope12may be an upright microscope.

In certain embodiments, the system19further comprises a plurality of manually rotatable stages15,17for mounting the plurality of motorized micromanipulators. Each manually rotatable stage may be three-dimensionally rotatable for providing operational convenience in mounting a respective micromanipulator.

Generally, the individual micromanipulator is installed with a micropipette holder4for holding a respective micropipette. The micropipette holder4may be, for instance, a stainless-steel micropipette holder.

In certain embodiments, the motorized stage8includes a dish holder plate82for carrying the Petri dish6. The dish holder plate82is usually a flat plane for the Petri dish6to reside on.

The system19may be divided into a computer vision subsystem and a robotic control subsystem. The computer vision subsystem comprises the optical microscope12and the camera11. Most experimental operations in using the system19may be performed in the diascopic illumination mode under a light source10and a 40× objective lens9. The objective lens9is optically coupled to, or is part of, the optical microscope12. The robotic control subsystem comprises the motorized stage8and the plurality of motorized micromanipulators5,7fixed on the plurality of manually rotatable stages15,17. In normal operations, the micropipette holder4holding the micropipette1is attached to the X-axis of the micromanipulator5. Similarly, the micromanipulator7has the same arrangement. Injection pressure is provided by the gas pressure provider13for forcing out the sample from the micropipette1during microinjection of the plurality of adherent cells2. The plurality of adherent cells2is cultured and injected on the Petri dish6, which is placed onto the motorized stage8throughout the experiment. All the electromechanical components used by the system19are placed on an anti-vibration table14. The one or more computers40are used to control different elements of the system19for facilitating microinjection of the adherent cells2. The micropipettes1,3, usually made of glass, are replaceable. The micropipette1is detachably attachable to the micropipette holder4. Similarly, the micropipette3is detachably attachable to a corresponding micropipette holder installed in the motorized manipulator7.

FIG.2depicts a flowchart showing a procedure of automated microinjection for the automated microinjection system19. First, the system19is initialized (S1). Second, adherent cells cultured in the Petri dish6having a glass bottom are transferred from an incubator onto the motorized stage8and positioned over the optical microscope12(S2). Third, the micropipettes1,3filled with injection samples are installed onto the two micromanipulators5,7and lowered to a focus plane in a FOV of the optical microscope12(S3). Fourth, if the optical microscope12is equipped with the DIC mode, switch to the DIC mode for enhancing the contrast of adherent cells to help distinguishing different cells (S4). Fifth, the motorized stage8moves from segment to segment of the Petri dish6to collect images (S5). Each segment is a physical rectangular region of the dish bottom corresponding to a FOV of the optical microscope12as reflected in an individual image taken by the camera11. Sixth, cell images are analyzed for detection and segmentation (S6). Seventh, injection sequences in each segment are optimized (S7). Eighth, the motorized stage8is moved back to the first segment; the micropipettes1,3are lowered down to the dish bottom (S8). Ninth, each micropipette tip's template image is collected for automatically detecting tip-dish contact in the tenth step S10in the working flow (S9). Tenth, automatic micropipette-dish contact is performed to collect information of the penetration depth (S10). Eleventh, the moving plane of the dish holder plate82relative to each micromanipulator is calculated (S11). After completing the preparations, cell microinjection can start (S12). If every target cell in a segment has been injected (S13), then the motorized stage8moves to the next segment (S14) to continue microinjection. If all the detected cells are injected (S15), the system stops (S16).

FIG.3shows an example of an optimized injection path of one micropipette injecting over one hundred cells under 20× magnification, where the optimized injection path is shown on a microscope-taken image with respect to a pixel coordinate frame as a frame of reference. Optimization of the injection path can improve productivity, where the optimization is done by minimizing the path length. The injection should start from the cell nearest to the micropipette tip until the farthest one in the shortest path. This process is an asymmetric TSP (n+1 nodes), which can be transformed to a symmetric TSP (2n+2 nodes) for solving the asymmetric TSP. The position of the tip in the pixel coordinate frame is denoted by (u0, v0). The positions of n cells are denoted by (ui, vi)i=1, . . . , n. Let D(n+1)×(n+1)be the distance matrix of the sequence of nodes (cells) (ui, vi)i=0, . . . , n, where dij=√{square root over ((ui−uj)2+(vi−vj)2)}. The matrix D(n+1)×(n+1)is symmetric and non-negative. To ensure the optimized path to start from (A0, B0), one sets all the distances to node 0 to zero, such that D(n+1)×(n+1)becomes {tilde over (D)}(n+1)×(n+1). Let {tilde over (d)}maxbe {tilde over (d)}max=max({tilde over (d)}ij), and transform {tilde over (D)}(n+1)×(n+1)to D′(n+1)×(n+1)as follows:

dij′={0if⁢i=jd∼i⁢j+3⁢d∼max+εotherwise,(1)
where ε>0 is a small positive number. The asymmetric distance matrix D′(n+1)×(n+1)is then used to construct a symmetric distance matrixD(2n+2)×(2n+2)as

D¯=[∞(D′)TD′∞](2⁢n+2)×(2⁢n+2)(2)
where ∞ is replaced by a large positive matrix. The matrixDacquires the optimized injection path. The mapping between nth and (n−1)th cells is (un−un-1+u0, vn−vn-1+v0). In one experimental finding, the optimization was completed in about 0.1 s only.

FIG.4is an image showing a random cell distribution and two opposite micropipettes. Guiding both micropipettes to inject only nearby cells to save time is reasonable in an ideal situation. In operating the automated microinjection system19, however, it is more practical that the system19inserts the sample to one cell at a time instant by one micropipette to avoid optimization of the injection sequence to become an overly complex task that burdens the one or more computers40. That is, a cell is injected either by the left micropipette P1or by the right micropipette P2but not both. As a result, the optimized injection sequence is aligned with the minimized total distance traveled by the motorized stage8. As will be shown later, the optimized injection sequence can be obtained by solving an E-GTSP.

FIG.5shows the formulation of the optimization problem for the two micropipettes (P1and P2) and cells into an E-GTSP. Given a weighted complete directed or undirected graph G=(V, E) and a partition V0, V1, . . . , Vn(cluster) of the overall node-set V, a minimum cost cycle containing precisely one node from each cluster V0, V1, . . . , Vnis identified. The overall node-set V={node(i)|i=0, 1, . . . , n}∪{all corresponding virtual nodes} contains every node and the edge set E={eij} represents edges joining node(i) to node(j) with a cost cijin the graph G. Denote detected cells as C1, C2, . . . , Cn.

Theorem 1. The distance traveled by the motorized stage8for injection of a cell Cmby the micropipette P2is equivalent to the distance for injection of a virtual cell C′mwith a new coordinate (u′m, v′m)=(um+uP1−uP2, vm+vP1−vP2) by the micropipette P1.

Proof Since it is desired to arrange the overall injection sequence to minimize the distance traveled by the motorized stage8, it is required to know which cell(s) should be injected by which micropipette and when to switch to the other after injecting a cell by a preceding micropipette. The key to answering the two questions lies in analyzing the distance of the motorized stage8when switching of the micropipettes is performed for injecting cells that are far apart. Without loss of generality, suppose that the micropipette first injects a nearby cell called Cl, then the micropipette P2injects one of its nearby cells Cm, and then the micropipette P1injects its second nearby cell Cq. In such case, the motorized stage8first moves the cell Clto the reference position of the micropipette P1for penetration, then moves the cell Cmto the reference position of the micropipette P2, and then moves the cell Cqback to the reference position of the micropipette P1. The distance traveled by the motorized stage8, denoted as dtotal, is given by
dtotal=dCl→Ref. pos. P1+dCm→Ref. pos. P2+dCq→Ref. pos. P1(3)
where
dCl→Ref. pos. P1=√{square root over ((ul−uP1)2+(vl−VP1)2)},  (4)
dCm→Ref. pos. P2=[(ul+(um+uP1−iP2))2+(vl+(vm+vP1−vP2))2]1/2(5)
and
dCq→Ref. pos. P1=[((um+uP1−uP2)−uq)2+(vm−vP1−vP2)−vq)2]1/2.  (6)
It can be inferred from (5) and (6) that the injection of a cell Cmby the micropipette P2is equivalent to the injection of a virtual cell C′mwith a new coordinate (u′m, v′m)=(um+uP1−uP2, vm+vP1−vP2) by the micropipette P1, in terms of the distance traveled by the motorized stage8. As such, in a series of n cells, determining which cells to be injected by the micropipette P2can be reformulated to forcing the micropipette P1to choose either a real cell, for example, Cmor its virtual opponent C′m. ▪

As a consequence, denote the reference position of the micropipette P1as node(0), the position of the cell Cias node(2i−1), and the position of the corresponding virtual cell C′ias node(2i) for i=1, 2, . . . , n. The node(0) is in the cluster V0and the two nodes (2i−1) and (2i) are in the cluster Vi, as shown inFIG.5. The cost cijof an edge eijbetween node(i) and node(j) is calculated as the Euclidean distance dij=√{square root over ((ui−uj)2+(vi−vj)2)}, i,j=0, 1, . . . , n, which forms the undirected graph G=(V, E) with V={node (i)|i=0, 1, . . . , n} and E={edge eijwith a cost cij=dij}. In the E-GTSP, any qualified cycle in the graph G visits precisely one node from each cluster, which means either a real cell Cior its virtual opponent C′ishould be chosen, which further judges whether cell Ciwill be injected by micropipette P1or by P2.

Finding an optimized solution requires transforming the E-GTSP to a standard TSP. First, all costs inside each cluster are set to zero, i.e. c(2i−1)(2i)=d(2i−1)(2i)=d(2i)(2i−1)=0, to ensure that the optimized cycle enters a cluster Viat node(2i−1) (or node(2i)) and then exits from the other node, node(2i) (or node(2i−1)). Second, the costs from node(2i−1) to the other nodes outside the cluster Viand the costs from node(2i) to the other nodes outside the cluster Viare exchanged. More precisely,
c(2i−1)q=d(2i)q=√{square root over ((u2i−uq)2+(v2i−vq)2)},q∉Vi,i=1,2, . . . ,n,(7)
and
C(2i)q=d(2i−1)q=√{square root over ((u2i−1−uq)2+(v2i−1−vq)2)},q∉Vi,i=1,2, . . . ,n.(8)

Third, all costs from other nodes to node(0) are set to zero to ensure that the optimized tour starts from the reference position of the micropipette P1but not the return. After the above three steps are done, the E-GTSP is transformed into an asymmetric TSP with a distance matrix denoted by {tilde over (D)}(2n+1)×(2n+1). Let {tilde over (d)}maxbe given by {tilde over (d)}max=max({tilde over (d)}ij), and then transform the matrix {tilde over (D)}(2n+1)×(2n+1)to the matrix D′(2n+1)×(2n+1)as follows:

dij′={0if⁢i=jd∼i⁢j+3⁢d∼max+εotherwise,(9)
where ε≤0 is a small positive number.

The asymmetric distance matrix D′(2n+1)×(2n+1)is used to construct a symmetric distance matrixD(4n+2)×(4n+2)as

D¯=[∞(D′)TD′∞](4⁢n+2)×(4⁢n+2)(10)
where ∞ is replaced by a large positive matrix. The one or more computers40solve the matrixDand obtains the optimized injection sequence starting from the micropipette P1or the micropipette P2.

FIG.6shows an example of an optimized injection path of two micropipettes injecting 29 cells under 40× magnification, where the optimized injection path is shown on an image with respect to a pixel coordinate frame used as a frame of reference. Solid arrows represent consecutive injections by one micropipette while dashed arrows represent a switch to the other micropipette. The injection starts from the cell C8and then C23, C24and C22consecutively by using the micropipette P2. After injecting these nearby cells of P2, the system starts to inject cells C11and C9by using the micropipette P1. Then, the system injects cells C21, C12, C17, C26, C28, C29and C6by the micropipette P2. Soon, cells C27and C25will be injected by the micropipette P1while the cell C3will be injected by the micropipette P2. After that, the micropipette P1injects the cell C13while the micropipette P2injects the cell C16. Again, the micropipette P1injects the cell C20while the micropipette P2injects the cell C19. Finally, all the remaining cells C1, C14, C4, C7, C5, C15, C2, C18and C10are injected by the micropipette P1. In this case, the total distance traveled by the motorized stage is 777 μm. However, if the micropipette P1injects all the cells, the optimized injection sequence is C19, C16, C3, C8, C23, C24, C6, C29, C28, C26, C27, C25, C17, C12, C22, C21, C18, C10, C2, C15, C5, C7, C4, C14, C1, C20, C13, C9and C11, of which the minimum distance is 980.25 μm. If the micropipette P2injects all the cells, the optimized injection sequence is C11, C9, C13, C20, C1, C14, C4, C7, C5, C15, C2, C10, C18, C21, C22, C12, C17, C25, C27, C26, C28, C29, C6, C24, C23, C8, C3, C19and C16, of which the minimum distance is 1037.25 μm. Obviously, the collaborative operation of the two micropipettes results in a smaller distance than any single micropipette does. The optimization is critical to the overall injection performance.

Based on the foregoing discussion on setting up the E-GTSP for the two-micropipette case, extension to a general situation that the system19employs q micropipettes, q≥2, in the plurality of micropipettes for microinjection is detailed as follows.

The q micropipettes are denoted as P1, P2, . . . , Pq. The micropipette Pk, k∈{1, 2, . . . , q} has a coordinate (uPk, vPk) on the pixel coordinate frame. The coordinates for the q micropipettes are determined from the XY locations of the micropipette tips. Denote n as a number of cells in the plurality of adherent cells2. It follows the n adherent cells are denoted as Cm, m=1, 2, . . . , n. The cell Cmhas a coordinate (um, vm) determined from the XY locations of the respective cells.

The E-GTSP for the case of using q micropipettes is formulated by constructing an undirected graph G=(V, E) where V is the overall node-set and E is the edge set.

The overall node-set V is a set of nq+1 nodes and is denoted by V={node(l)|l=0, 1, . . . , nq}. In addition, V is partitioned into n non-overlapping clusters of nodes. The n clusters of nodes are denoted as V0, V1, . . . , Vn. The cluster V0has one node and is given by V0={node(0)} where node(0) represents P1, the reference micropipette. The cluster Vm, m∈{1, 2, . . . , n}, has q nodes and is given by Vm={node((m−1)q+k)|k=1, 2, . . . , q}, where node((m−1)q+1), node((m−1)q+2), . . . , node(mq) respectively represent Cm, C′m(2), C′m(3), . . . , C′m(q). In the last expression, Cmis the mth (real) cell in the plurality of adherent cells2, and C′m(k) is the kth virtual cell of the mth real cell. The injection of a cell Cmby the micropipette Pkis equivalent to the injection of a virtual cell C′m(k) with new coordinates (u′m(k), v′m(k))=(um+uP1−uPk, vm+vP1−vPk) by the micropipette P1, in terms of the distance traveled by the motorized stage8. Note that each of the aforementioned new coordinates is obtained by an application of Theorem 1.

The edge set E={eij|i, j=0, 1, . . . , nq} represents edges joining node(i) and node(j) with a cost cijin the graph G. Values of cijand cjiare same and are equal to the Euclidean distance dijbetween entities at node(i) and at node(j), where dij=(ui−uj)2+(vi−vj)2for i, j∈{1, 2, . . . , nq}.

With G constructed, one can make use of an algorithm known in the art for numerically solving the E-GTSP defined by G to identify an ordered sequence of nodes containing one node from each of V0, V1, . . . , Vnsuch that the ordered sequence of nodes forms a minimum cost cycle. Finally, the optimized injection sequence can be obtained from the ordered sequence of nodes.

Algorithms for solving E-GTSP can be found from, for example: K. HELSGAUN, “Solving the equality generalized traveling salesman problem using the LinKernighan-Helsgaun Algorithm,”Mathematical Programming Computation(2015) 7:269-287; and C.-M. PINTEA, P. C. POP and C. CHIRA, “The generalized traveling salesman problem solved with ant algorithms,”Complex Adaptive System Modeling(2017) 5:8. The two aforementioned disclosures are incorporated herein by reference. Alternatively, the E-GTSP can be solved by: transforming the E-GTSP to an asymmetric TSP; transforming the asymmetric TSP into a symmetric TSP; and numerically solving the symmetric TSP to yield the ordered sequence of nodes. This approach is detailed above for the specific case of q=2. Algorithms for transforming the E-GTSP to the symmetric TSP for a general value of q can be found in the art, e.g., from C. E. NOON and J. C. BEAN, “An Efficient Transformation of the Generalized Traveling Salesman Problem,”Information Systems and Operational Research, vol. 31, no. 1, February 1993, pp. 39-44, the disclosure of which is incorporated herein by reference.

FIG.7shows the movement and coordination of the micropipette1and the motorized stage8when injecting cells. The X-axis of the micromanipulator5is tilted to 45° relative to the horizontal level, causing the micropipette1to be tilted by 45°. Before microinjection starts, a reference position of the micropipette tip (A0, B0), (xr, yr, zr), (Xr, Yr) is recorded when the tip touches the dish bottom76. Here, (A0, B0) denotes the tip position in the pixel coordinate frame, (xr, yr, zr) is the reference position of the tip, and (Xr, Yr) is the reference position of the motorized stage8. During microinjection, the X-axis of the micromanipulator5is withdrawn back by 20 μm (xr−20, yr, zr) first, and the target cell2is moved to the reference position (Xr, Yr). Finally, the X-axis of the micromanipulator5pushes the micropipette1forward to the reference position (xr, yr, zr) to pierce the cell2.

FIG.8shows the generation of sidelong sliding when the micropipette tip touches the dish bottom76and continues to go down. A common approach to know the tip depth is to manually lower down the micropipette1until its tip touches the dish bottom76and is deformed to generate noticeable sidelong displacement. It is time-consuming and prone to errors even for an experienced operator, often causing micropipettes to break.

In the automated microinjection system19, this tip-depth determination operation is automated to improve accuracy and save time by matching a template image to source images recursively in four steps. Positions801to803as indicated inFIG.8show the generation of the sidelong sliding when the tip touches the dish bottom76and continues to go down.

FIG.9shows the geometry and illustration of template matching. The patch I(i+r, j+s) is moved across the source image I by an offset (r, s) and used to calculate the similarity to the template image T(i, j).

FIG.10shows how the home-made program matches the template image of the micropipette tip to every source image. First, the operator collects a template image T of the tip in focus. Second, a source image I is compared against the template image T to find a match. We use the OpenCV function matchTemplate to identify the matching area. The function matchTemplate selects a patch at every location in a source image and calculates the similarity between that patch and the template image, and then stores the similarity in the result matrix dE(r, s). The similarity is calculated by adding up squared differences in the pixel intensities of the template image T(i, j) and those of a patch in the source image I(i+r, j+s) at every location (r, s), as expressed in (11):

dE(r,s)=∑i=0CI∑j=0RI[I⁡(i+r,j+s)-T⁡(i,j)]2,(11)
where (r, s) is in the range [0, CI)×[0, RI). After obtaining the result matrix dE(r, s), the function minAlaxLoc searches the best similarity and finds its position as the location of the tip in the pixel coordinate frame. These matching locations are recorded temporarily, and every other location of the matching results (every frame 1 and frame 3) are compared to determine if the tip touches the bottom. Third, the micropipette is lowered down at a constant speed of 1 μm/s. Finally, the micropipette is retracted immediately as soon as the position difference from the second step exceeded a threshold; at the same time, positions of the tip and the stage were both determined.

One advantageous feature of the system19is that insertion depths of the micropipettes1,3are adaptively adjustable according to a moving plane of the dish holder plate82of the motorized stage8. The moving plane is used for characterizing unevenness between a focus plane of the optical microscope12and a moving trajectory of the motorized stage8. Ideally, the moving plane is a horizontal plane (with respect to the reference vertical direction900) such that no adjustment to the insertion depths of the micropipettes1,3is required. Practically, however, the moving plane is not perfectly horizontal so that adaptive adjustment of the insertion depths of the micropipettes1,3is advantageous. Such adaptive adjustment is realizable by automatically making contact between the individual micropipette and the dish bottom of the Petri dish6and by fitting the data into a virtual plane. Preferably, the one or more computers40are further configured to control the motorized stage8and the plurality of motorized micromanipulators5,7in a coordinated way that the individual micromanipulator goes down or up during movement of the motorized stage8to compensate for the unevenness between the focus plane of the optical microscope12and the moving trajectory of the motorized stage8. To achieve this purpose, it is required to determine the moving plane for characterizing the aforesaid unevenness.

FIG.11shows the determination of the moving plane of the dish holder plate82for adaptively adjusting the penetration depth of the micropipette. The moving plane has a reference numeral of56while the focus plane of the optical microscope12is referenced as55. We use the least-square method to identify the moving plane56based on stage positions (Xi, Yi)i=i, . . . mand tip depths (zi)i=1 . . . m. The formula of the moving plane56is z=PX+QY+R, where P, Q and R are unknown scalars to be determined. To avoid ill-conditioned numerical calculations, we calculated the averagesX=m−1Σi=1mXi,Y=m−1Σi=1mYiandz=m−1Σi=1mzifirst and substituted the calculated averages into the original plane formula to form a new plane formula {circumflex over (z)}=P{circumflex over (X)}+QŶ+{circumflex over (R)}, where {circumflex over (z)}=z−z, {circumflex over (X)}=X−X, Ŷ=Y−Yand {circumflex over (R)}=R+PX+QY−z. The best fit occurs only when the sum of the squared errors between the depths {circumflex over (z)}iand the plane values P{circumflex over (X)}i+QŶi+{circumflex over (R)} is minimized. The sum (the error function) is written as

e⁡(P,Q,Rˆ)=∑i=1m[(P⁢Xˆi+Q⁢Yˆi+Rˆ)-zˆi]2.(12)
Equation (12) is non-negative and reaches the minimum only when every partial derivative evaluated at Pbest, Qbestand {circumflex over (R)}bestequals zero, i.e. ∂e/∂P|P=Pbest=0, ∂e/∂Q|Q=Qbest=0 and ∂e/∂{circumflex over (R)}|{circumflex over (R)}={circumflex over (R)}best=0. These partial derivatives can be written as

0=∑i=1m[(Pb⁢e⁢s⁢t⁢Xˆi+Qb⁢e⁢s⁢t⁢Yˆi+Rˆb⁢e⁢s⁢t)-zˆi]⁢Xˆi,(13)0=∑i=1m[(Pb⁢e⁢s⁢t⁢Xˆi+Qb⁢e⁢s⁢t⁢Yˆi+Rˆb⁢e⁢s⁢t)-zˆi]⁢Yˆi⁢and(14)0=∑i=1m[(Pb⁢e⁢s⁢t⁢Xˆi+Qb⁢e⁢s⁢t⁢Yˆi+Rˆb⁢e⁢s⁢t)-zˆi].(15)
Rearrangement of (13) to (15) leads to the following equation:

[∑Xˆi2∑Xˆi⁢Yˆi∑Xˆi∑Xˆi⁢Yˆi∑Yˆi2∑Yˆi∑Xˆi∑Yˆim][Pb⁢e⁢s⁢tQb⁢e⁢s⁢tRˆb⁢e⁢s⁢t]=[∑zˆi⁢Xˆi∑zˆi⁢Yˆi∑zˆi].(16)
The matrix at the left-hand side of (16) is

[∑(Xi-X¯)2∑(Xi-X¯)⁢(Yi-Y¯)0∑(Xi-X¯)⁢(Yi-Y¯)∑(Yi-Y¯)2000m](17)
and the right-hand side of (16) is

[∑(zi-z¯)⁢(Xi-X¯)∑(zi-z¯)⁢(Yi-Y¯)0].(18)
Here we define α=Σ(Xi−X)2, γ=Σ(Yi−Y)2, β=Σ(Xi−X), (Yi−Y), ψ=Σ(zi−z)(Xi−X) and ζ=Σ(zi−z)(Yi−Y). Then (16) becomes

[αβ0βγ000m][Pb⁢e⁢s⁢tQb⁢e⁢s⁢tRˆb⁢e⁢s⁢t]=[ψζ0].(19)
Finally, we have
Pbest=(γψ−βζ)/(αγ−β2),  (20)
Qbest=(αζ−βψ)/(αγ−β2)  (21)
and
{circumflex over (R)}best=z−PbestX−QbestY.(22)
The formula for the moving plane56is then given by
z=Pbest(X−X)+Qbest(Y−Y)+z.(23)

Some experimental results, which were obtained by using the system19for microinjection, are listed as follows for demonstrating the effectiveness of the present invention.

FIG.12shows an example of an optimized injection path of one micropipette injecting over one hundred cells under 20× magnification.

FIG.13shows the cell detection results, in which detected cells are delineated by rectangles and marked with a confidence score. A deep learning algorithm is used to detect cells automatically. A total of 771 DIC images of Hep G2 cells from the microscope under 40× magnification is sampled. All images were resized to 1152×864 pixels and then annotated manually by experts. Conventionally, these data should be divided into three subsets: the training set to train different algorithms (models) to check the convergence; the validation set to evaluate the trained algorithms and tune the algorithms' hyperparameters (e.g., layers of an algorithm); and the test set to evaluate the generalization ability of a trained algorithm. We chose two algorithms in advance and split the overall image set into two subsets: the training set containing 617 randomly selected images; and the test set containing all the rest images. Data augmentation, such as flipping, random contrast distortion, and brightness distortion, was used to teach the algorithm the desired invariance and robustness properties when only 617 training samples were available.

For cell detection, the algorithm output predictions consist of bounding boxes (positions) of cells and confidence scores of the bounding boxes. For every valid prediction, the maximum IoU of the bounding box b and all ground-truth bounding boxes big, were calculated to determine the correspondence of a predicated bounding box b and a ground-truth bounding box bg:

I⁢o⁢U⁡(b,bg)=maxiarea⁢(b⋂big)area⁢(b⋃big),∼i=1,2,…,m.(24)
A prediction is considered a true positive only when its IoU is larger than a threshold α; otherwise, it is considered a false positive. The precision-recall curve was drawn by varying the threshold α. We calculated the AP at different IoU threshold α as the mean precision p at 11 recall (r) levels (0, 0.1, . . . , 1) by using the 11-point interpolation metric in:

A⁢P=11⁢1⁢∑i∈{0,0.1,…,1}max⁡(p⁡(r≥l)).(25)
The precision-recall curves of the cell detection algorithm are provided inFIG.15andFIG.16.

FIG.14shows the cell semantic segmentation results, in which segmented cells are marked white. For cell semantic segmentation, five MATLAB metrics were used to evaluate the results: global accuracy, mean accuracy, mean IoU, weighted IoU, and mean BF score. The global pixel accuracy is the ratio of correctly categorized pixels to the total number of pixels in all test images. The accuracy refers to the percentage of correctly labeled pixels for each category, that is, the cell or background in an image, while the mean accuracy is the average accuracy of all categories in all images. The IoU is the ratio of correctly labeled pixels to the total number of ground truth and predicted pixels in that category, while the mean IoU is the average IoU of all categories in all images. The weighted IoU is the average IoU of each category weighted by the number of pixels in that category. The BF score measures the alignment of the predicted boundary for each category with the ground truth, while the mean BF score is the average BF score of all categories in all images. Detailed evaluations of the cell segmentation algorithm are summarized in Table 1.

TABLE 1Evaluation of Cell Semantic Segmentation.GlobalMeanMeanWeightedMean BFAccuracyAccuracyIoUIoUScore95.331%83.338%72.455%88.219%68.438%

FIG.15shows the precision-recall curve generated at IoU threshold α=0.5 of the cell detection algorithm.

FIG.16shows the precision-recall curve generated at IoU threshold α=0.7 of the cell detection algorithm.

FIG.17shows the result of cells successfully injected with FITC. FITC, a typical fluorescent dye, was injected into MC3T3 fibroblast cells to evaluate the success and survival rates. FITC is impermeable to cell membranes and dissolves quickly in the medium, so only successfully injected cells were counted for evaluation purposes. The concentration of FITC was 0.5 mg/mL, the output pressure for injection was 1.0 psi, the insertion duration was 100 ms, and the cell density was retained at approximately 30 to 40 cells per FOV under 40× magnification. The time interval between the two injections was set to 0.8 s, and the interval between two segments (FOV) was set to 2 s. Under such circumstances, a one-hour experiment can usually contain about 110 segments and around 4,000 cells, i.e. (3, 600−2×100)/0.8=4225≈4000.

FIG.18shows the result of dead cells stained by PI. Cells were washed with fresh medium and returned to the incubator as soon as experiments were completed. The cells were stained with PI, a fluorescent cell viability indication dye, to examine the survival rate after 30 minutes. FITC emits green fluorescent light (525 nm) under the excitation of cyan laser (488 nm) while PI emits red fluorescent light (617 nm) under the excitation of green laser (535 nm) and only binds to the DNA of the dead cells. Viable and dead cells were counted by moving the dish one segment at a time and stitching captured images together. The success and survival rates are counted according to (26) and (27), respectively:

Success⁢rate=Number⁢of⁢green⁢fluorescent⁢cellsNumber⁢of⁢injections⁢and(26)Survival⁢rate=Number⁢of⁢red⁢fluorescent⁢cellsNumber⁢of⁢injections.(27)
Table 2 summarizes the experimental results. A total of 11,857 injections were made, among which 7,147 were successfully injected, and 5,861 cells were still alive after 30-minute re-incubation. The success rate and the survival rate were 60.3% and 82.0%, respectively.

TABLE 2Cell Microinjection Results.TrialInjectionInjectedSurvivedSuccessSurvivalnumberTimenumbercellsCellsraterate160 min3,7482,1871,59658.4%73.0%260 min4,0852,4452,31559.9%94.6%360 min4,0242,5151,95062.5%77.5%Total180 min11,8577,1475,86160.3%82.0%

Lastly, a summary of non-limiting advantages offered by the system19is provided as follows. First, the system19can automatically detect unstained cells by using the deep learning technology. Second, the system19can optimize the injection path of tens to hundreds of cell positions in a short time. Third, the system19can inject cells continuously for a long time (currently about an hour) with two micropipettes by using constant outflow-based injection and adjusting the penetration depth adaptively according to the moving plane56, whereas many existing microinjection systems cannot ensure long time working without changing micropipettes. Fourth, the system19can achieve automated high-throughput microinjection of adherent cells based on the first three advantages.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiment is therefore to be considered in all respects as illustrative and not restrictive. The scope of the invention is indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.