METHODS AND SYSTEMS FOR TRACKING BIOLOGICAL MATERIAL

The present disclosure relates to a method performed by one or more computers for tracking a biological material of a subject during an in-vitro fertilization process. The method includes receiving, from a camera, an image of a dish having a visual characteristic and a drop disposed on the dish, the dish holding the biological material at a drop location. The method then includes processing the image of the dish, using a drop identification model, to identify the drop according to the visual characteristic. Further, the method includes assigning an identifier to the drop associated with the drop location, and recording the identifier of the drop associated with the drop location.

TECHNICAL FIELD

The present disclosure relates to methods and systems for tracking a biological material, and more specifically, methods and systems for tracking biological material in an in-vitro fertilization process.

BACKGROUND

When conducting an in-vitro fertilization (IVF) cycle, standard practice is to create multiple embryos and transfer the embryo that has the best chance of developing into a healthy baby back into the uterus. Embryos that are aneuploid (i.e., having an uneven number of chromosomes) are less likely to make it to birth, so genomic testing of embryos, which can identify aneuploid embryos, has become a common practice.

To carry out genomic testing, cells are removed from the embryo and sent to a genomics lab for testing, and the embryo is vitrified (i.e., frozen in liquid nitrogen) while awaiting results. After receiving the results, the embryo associated with the biopsy will be thawed and transferred if viable or discarded if not viable.

It is essential that accurate records are kept so that the genomic tests results can be linked back to the embryo (now vitrified) that the biopsy sample came from. To maintain this link, the embryo is assigned an identity either prior to or at a biopsy stage. During biopsy, the biopsy sample also assumes this identity. The identity is typically a sequential number, which when linked with the patient's ID becomes a unique identifier.

The biopsy process happens in a drop of fluid on a dish. After the biopsy has happened, the embryo will often be moved into a drop on another dish, and from there through other drops (some on the same dish, some on others dishes) until the embryo is vitrified. A similar process happens with the biopsy sample. The dishes are labelled with the patient ID, and most drops are labelled with the identity number of the embryo. There may be multiple drops in a single dish.

To prevent mistakes, every time an embryo is moved between dishes, it is best practice for a second embryologist to be called over to witness the movement. The witness ensures that the two dishes have the same patient information, the correct embryo is being selected to be moved, and that it is placed in the correct drop in the receiving dish. Often, the embryologist and witness will each initial a paper record to show that this has happened.

SUMMARY

The present disclosure is directed to systems and methods for tracking a subject's biological material in a lab during an IVF process. The systems disclosed herein provide seamless and automated tracking that reduces instances of error and the need for a witness at each transfer step (i.e., moving a biological sample between different dishes or vessels or between different drops on the same dish).

In a first example aspect, a method performed by one or more computers for tracking a biological material of a subject during an in-vitro fertilization process may include receiving, from a camera, an image of a dish having a visual characteristic and a drop disposed on the dish, the dish holding the biological material at a drop location. The method may include processing the image of the dish, using a drop identification model, to identify the drop according to the visual characteristic. Further, the method may include assigning an identifier to the drop associated with the drop location, and recording the identifier of the drop associated with the drop location.

In a second example aspect, one or more non-transitory computer storage media may store instructions that when executed by one or more computers cause the one or more computers to perform operations for tracking a biological material in an in-vitro fertilization (IVF) process. The operations may include receiving an image of a dish, wherein the dish comprises a visual characteristic and one or more drops. The operations may further include processing the image of the dish, using a drop identification model, to identify a drop associated with a drop location according to the visual characteristic. The operations may further include assigning an identifier to the drop based on the visual characteristic and recording the identifier of the drop associated with the drop location.

In a third example aspect, a system for tracking a biological material in an in-vitro fertilization (IVF) process may include a microscope, a camera, one or more computers, and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices may store a database containing a plurality of visual characteristics, and instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for tracking a biological material in an in-vitro fertilization process. The operations may include receiving an image of a dish, wherein the dish comprises a visual characteristic and one or more drops. The operations may further include processing the image of the dish to identify a drop at a drop location according to the visual characteristic and/or the one or more drops. Further, the operations may further include assigning an identifier to the drop associated with the drop location and recording the identifier of the drop associated with the drop location.

In accordance with any one of the first, second, and third aspects, the method, system, and non-transitory computer storage media for tracking a biological material of a subject during an in-vitro fertilization process and may include any one of the following forms.

In one example, receiving may include receiving a partial or entire layout image of the dish using a microscope camera.

In another example, receiving may include receiving an entire layout of the dish using a wide-view camera.

In some examples, the method may include identifying a first status or condition of a pipette at the drop location.

In some examples, the pipette may receive the biological material at the drop location.

In some examples, the method may include recording in a memory the first status or condition of the pipette holding the biological material.

In other examples, identifying the first status or condition may include determining that the pipette enters a first drop holding the biological material.

In yet another example, the method may include identifying a second status or condition of the pipette holding the biological material at a second location.

In one form, the method may include analyzing the second status or condition of the pipette.

In another form, the method may include determining, before the biological material is delivered to the second location, that the second location for depositing the biological material correlates with a standard operating protocols stored in a database of the memory.

In some forms, the method may include signaling an error message after determining that the second location does not correlate with standard operating protocols.

In other forms, the method may include signaling a correct message after determining that the second location correlates with standard operating protocols.

In yet another form, the method may include recording a delivery status of the biological material from the pipette to the second location.

In some forms, the second location may be a tube having a unique identity.

In one aspect, the method may include recording a delivery status of the biological material from the pipette to the second location.

In some aspects, the second location may be a drop of washing solution.

In another aspect, the method may include recording a delivery status of the biological material from the pipette to the second location.

In some aspects, the second location may be a drop on a second dish.

In some aspects, the method may include identifying a third status or condition of the pipette holding the biological material at a third location.

In other aspects, the method may include assigning the biological material a unique identity.

In some aspects, the unique identity of the biological material may be maintained as the biological material moves.

In yet another aspect, identifying the biological material may include identifying that the biological material is an embryo associated with the drop location.

In one example, identifying the biological material may include identifying that the biological material is a biopsy of an embryo associated with the drop location.

In another example, the method may include processing the image of the dish, using a subject identification model, to classify a subject identification associated with the dish.

In some examples, the method may include recording in the memory the subject identification associated with the dish.

In some examples, the method may include processing the image of the dish having a drop pattern, using a drop pattern identification model, to classify a type of dish associated with the drop pattern.

In other examples, the method may include obtaining, from a database, a pattern of drops on the dish.

In some examples, the method may include processing a model input that comprises the pattern of drops on the dish using a machine learning model, having a set of machine learning model parameters, to generate a model output that characterizes a likelihood that the pattern of drops on the dish is associated with a type of dish.

In some examples, the method may include classifying, based on the model output of the machine learning model, whether the pattern of drops is associated with the type of dish.

In yet another example, the method may include training the machine learning model, by a machine learning training technique, to determine trained values of the set of machine learning model parameters.

In one form, training the machine learning model by the machine learning training technique may include obtaining a set of training examples.

In some forms, each training example may include (i) a training input comprising a pattern of drops on a dish, and (ii) a target output based on whether the pattern of drops designates the type of dish.

In some forms, training the machine learning model may include training the machine learning model on the set of training examples.

In another form, training the machine learning model on the set of training examples may include training the machine learning model to, for each training example, process the training input of the training example to generate a model output that matches the target output of the training example.

In some forms, the operations may include receiving an image of a pipette adjacent to or in the drop.

In some forms, the operations may include identifying a status or condition of the pipette as receiving a biological material associated with the drop.

In other forms, the operations may include receiving an image of a second dish having a visual characteristic and one or more drops.

In some forms, the operations may include identifying the second dish according to the visual characteristic.

In some forms, the operations may include processing the image of the second dish, using a drop identification model, to identify a drop associated with a drop location of the second dish according to the visual characteristic.

In some forms, the operations may include assigning an identifier to the drop based on the visual characteristic.

In some forms, the operations may include recording the identifier of the drop associated with the drop location of the second dish.

In yet another form, a database may include information related to a plurality of dish types and a plurality of drop patterns for each of the plurality of the types of dishes.

In some forms, the operations may include receiving an image of a dish having a drop pattern.

In some forms, the operations may include comparing the drop pattern to the plurality of drop patterns associated with the plurality of types of dishes stored in the database.

In some forms, the operations may include identifying a dish type of the dish according to the drop pattern.

In one aspect, the operations may include receiving an image of a pipette adjacent to or in a different drop at a drop location of a second dish.

In another aspect, the operations may include identifying a status or condition of the pipette before delivering the biological material associated with the drop location of the dish to the drop location of the second dish.

In some aspects, the operations may include receiving an image of the pipette adjacent to or in a second drop of the dish.

In other aspects, the operations may include identifying a status or condition of the pipette before delivering the biological material associated with the drop location to the second drop of the dish.

In yet another aspect, the operations may include receiving an image of the pipette adjacent to or in a tube.

In one example, the operations may include identifying a status or condition of the pipette before delivering the biological material associated with the drop location to the tube.

In another example, the operations may include, before delivering the biological material, determining that the status or condition of the pipette correlates with a correct drop location according to standard operating protocols stored in a database.

In some examples, the operations may include receiving an image of the pipette entering a drop located at the drop location.

In other examples, the operations may include identifying a status or condition of the pipette entering the drop as receiving the biological material associated with the drop location.

In yet another example, the operations may include receiving an image of the pipette entering a different drop located at a different drop location.

In one form, the operations may include identifying a status or condition of the pipette entering the different drop as delivering the biological material associated with the drop location to the second drop location.

In another form, the operations may include processing the image of the dish, using a dish identification model, to classify a dish orientation or dish type according to the visual characteristic.

In some forms, the operations may include assigning the biological material a unique identity.

In some forms, the unique identity of the biological material may be maintained as the biological material moves.

In some aspects, the camera may be a wide-view camera configured to image an entire layout of the dish.

In other aspects, the system may include a microscope camera.

In yet another aspect, the camera may be a microscope camera.

In one example, the system may include a wide-view camera configured to image an entire layout of the dish.

In another example, the database may contain information related to a plurality of dish types and a plurality of drop patterns for each of the plurality of the types of dishes.

In some examples, the operations may include receiving an image of a pipette adjacent to or in a drop of a second dish.

In other examples, the operations may include identifying a status or condition of the pipette before delivering the biological material associated with the drop location of the dish to the drop of the second dish.

In yet another example, the operations may include delivering a correct message after determining the status or condition of the pipette correlates with the correct drop location.

In one form, the operations may include delivering an error message after determining the status or condition of the pipette does not correlate with the correct drop location.

Systems and methods described in the present disclosure can include one or more of the following advantages.

In some examples, the system is compatible with multiple dish types (i.e., flat dishes or welled dishes) having various drop layouts, so every step of an IVF cycle can be recorded, thereby eliminating the need for a second witness. Additionally, the system is compatible with other labware used in the IVF cycle, such as, for example, PCR tubes, pipettes, vitrification devices, test tubes, and transfer catheters. By eliminating the need of a second witness, the tracking system and method disclosed herein can reduce costs associated with IVF, and streamline the IVF process.

In some examples, the system can be arranged to constantly witness the actions of a technician (e.g., embryologist), so drops cannot be moved without the system recording the movements. The system sees the embryo being moved, so the movement is truly witnessed.

In some examples, the system provides real-time feedback to the embryologist and thereby prevents errors from occurring. Specifically, the system can incorporate a clinic's standard operating protocols (SOPs), and the actions of the embryologist can be compared against the SOP. For example, while the embryologist is looking through the microscope, and it appears that an embryo will be placed in the wrong drop, the system will notify the embryologist before an incorrect transfer happens.

In some examples, the system improves the workplace environment. Specifically, the system reduces scanning equipment, RFID tag or barcode printers, etc., thereby by avoiding clutter. Additionally, by using one or more cameras with varying fields of view, the visibility of the workspace may improve. While using a microscopic view, the embryologist can work on any drop they can see, and while using a wide-view camera, the embryologist can find the next drop to work on seamlessly.

In some examples, the system may be incorporated easily into existing work spaces, and may be retrofitted to work with existing microscopes. Further, in some examples, the system can be used for tracking other biological materials in an IVF process or other process.

Definitions

As used herein, the terms “top,” “bottom,” “upper,” “lower,” “above,” and “below” are used to provide a relative relationship between structures. The use of these terms does not indicate or require that a particular structure must be located at a particular location in the apparatus.

Other aspects, features, and advantages of the present disclosure will be apparent from the following detailed description, figures, and claims.

DETAILED DESCRIPTION

A tracking assembly disclosed herein provides a seamless and automated chain of custody of biological material within an IVF lab that reduces instances of error and the need for an additional embryologist to witness each transfer step (i.e., moving a biological sample between different dishes or vessels or between different drops on the same dish) of an IVF cycle. InFIG.1A, an assembly10for tracking a subject's biological material in a lab during an IVF process includes an imaging system14including a camera22that is coupled to a microscope18and coupled to a computer24. The microscope18and camera22are arranged on a work surface26, and the computer24is disposed underneath or beside the work surface26. The camera22is coupled to the computer24, and may be coupled to a user interface30. The imaging system14is configured to monitor and record any transfer of biological material on the work surface26. Adjacent or near the assembly10is a cryopreservation device32that stores biological material. In the illustrated example, an RFID reader33is integrated with the work surface26to read RFID tags associated with various dishes that are placed on the RFID reader33. The RFID tags may be configured to identify the subject.

FIG.1Bshows a block diagram of the example imaging system14ofFIG.1A. The imaging system14is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. Generally, the imaging system14includes one or more processors and data storage or memory devices that define a detection model and stores instructions executed by the one or more processors. More specifically, the imaging system14uses image recognition software (e.g., computer vision, machine vision, etc.) and/or machine learning to image, process, and identify drops, visual characteristics, dishes, vessels, and pipettes brought under the microscope18. The imaging system14tracks and keeps records of biological material as an embryologist moves the biological material between different locations including different drops, dishes, vessels, and pipettes. In the illustrated example ofFIG.1B, the imaging system14includes the computer24, the camera22, a memory52, and a detection model56.

In the assembly10ofFIG.1A, the camera22has a wide field-of-view (FOV) and is coupled to a body34of the microscope18. The camera22continuously or periodically captures images of one or more dishes38,40under or near the microscope18on the work surface26, and delivers the images to the imaging system14. The wide FOV camera22is placed adjacent to the lens44of the microscope18to image an entire drop layout of one or more dishes38,40. As shown inFIG.1A, a first dish38is disposed directly under the lens44of the microscope18and placed over a light source48on the work surface26, and a second dish40is disposed adjacent to the first dish38. The camera22is positioned to view each drop layout on the dish38directly under the microscope18or both dishes38,40on the work surface26. In the illustrated example assembly, the camera22is a webcam, but may be any suitable camera.

Returning toFIG.1B, the system14can store images and image data generated by the camera22in the memory52. The memory52may be, for example, a physical data storage device or a logical data storage area of the computer24. Additionally, the system14stores data that defines the locations of each of the drops of each dish in the memory52.

The imaging system14is configured to receive an image of a dish38having a visual characteristic and one or more drops disposed on the dish38, and then is configured to process the image using a detection model56to both the dish and/or one or more drops according to one or more visual characteristics. The imaging system14assigns a unique identifier to the dish and/or to the identified drop, and records the identifier of the drop associated with the drop location of the drop. The imaging system14may process the other drops disposed on the dish38in the same way. The imaging system14may also record visual characteristics that can be used to identify that specific dish and distinguish it from other dishes with the same drop pattern. Additionally, the imaging system14is configured to process the image using the detection model56to identify the dish38(e.g., dish type, orientation of the dish, drop pattern) according to one or more visual characteristics. Further, the imaging system14can detect when a pipette enters or exits a drop of biological material disposed on the dish38. The imaging system14can track whether any biological material has been moved from the identified drop and where the biological material is moved to, keeping records of each transfer of material from one drop to another. The visual characteristic may be disposed on a dish, tube, pipette, or other vessel and may include one or more of a marking, drop pattern, drop size, drop shape, drop location, relative drop locations, barcode, tag placement, name, number, a combination of characters, or other identifier that identifies a drop type, subject, biological material, orientation, dish type, vessel type, pipette type, dish size, wells, molded in details such as well numbers or grid locations, drop type, or a combination thereof.

Specifically, the imaging system14is communicatively coupled to the camera22and receives the images from the camera22. The system14then processes images to identify and/or classify various characteristics of the drop, dish, and/or pipette. The one or more data storage devices (i.e., the memory52) of the imaging system14defines the detection model56and a database containing, for example, subject information, a plurality of visual characteristics, a plurality of types of biological material, standard operating protocols (SOPs) for the IVF process, a plurality of dish types, and a plurality of drop patterns for each of the dish types. After receiving the image from the camera22, for example, the imaging system14processes the image and compares the image with information stored in the database. The detection model56includes various models for analyzing a variety of parameters, such as, for example, drop identification model, a material identification model, a subject identification model, a dish identification model, a pipette identification model, a pipette-in-drop identification model, a PCR tube identification model, and a vitrification device identification model.

The computer24is communicatively coupled to the camera22by a wired and/or wireless connection, such as via Bluetooth™, or radio communication (e.g., Wi-Fi). The computer24is configured to deliver real-time feedback in the form or prompts and/or alerts to the embryologist at each stage of the IVF process. This real-time feedback is delivered through the user interface30and through audible feedback, which is coupled to the computer24.

Different IVF clinics have different protocols for the IVF procedure, but there are a lot of commonalities between them.FIG.2illustrates a flow chart of an IVF cycle100, and will be described with reference to the assembly10and imaging system14ofFIG.1A. Initially, the cycle100begins at a first stage I where sperm from a semen sample104from patient A fertilizes a single or multiple eggs108from patient B in a fertilization dish112. The fertilized eggs then become embryos116. At a second stage II, embryos116are transferred to a culture dish120where multiple embryos116reside in a single drop124. In the illustrated example, the culture dish120includes two drops124,126each containing multiple embryos116. However, the number of embryos may vary by subject. When the embryos116develop to a desired maturity, the embryos116are transferred from the culture dish120to a holding dish130at a third stage III, where each embryo116A,116B,116C is disposed in a drop134and is assigned an identity (e.g., numbers, letters, or combination thereof) in the dish130. In the illustrated example, first, second, and third embryos116A,116B,116C, are assigned numbers1,2,3, respectively. As these embryos are moved between drops and dishes, their new location will be recorded by the system14. Further, the system14can identify how many empty drops remain on the dish130by assuming that each drop that is not accessed by the pipette remains empty. The system14also records which drop(s) contain(s) an embryo (or biopsy at subsequent stages) and can alert an embryologist not to dispose an embryo into a drop already containing an embryo (or biopsy).

At the third stage III, the camera22attached to the microscope18ofFIG.1Acaptures images of the entire holding dish130. The holding dish130has a plurality of drops134(e.g., eight drops) arranged in a circle on the surface of the dish130, as shown inFIGS.3-5. At a 12 o'clock position (relative to the orientation ofFIG.3), the dish130includes a visual marking138or characteristic that can be identified by the imaging system14. The characteristic138in this example is a thick line extending radially outwardly from a first drop location1to an edge of the dish130. The imaging system14is configured to identify and track each drop on the dish130by assigning each drop with an identifier, such as, for example, a number (e.g., numbers1through8) relative to characteristic138. The system14tracks the location of the drops134as the dish130moves (rotationally or translationally) from an original position and within the FOV of the camera22. For example, the system14tracks a first drop location1from the twelve o'clock position illustrated inFIG.3to a two o'clock position inFIG.4. In some examples, the drops134are mapped to a known drop pattern or pattern layout stored in the memory52of the system14at the third stage III. For example, the system14can identify the circular layout or pattern of the drops134of the first dish38as being a holding dish130for mature embryos.

Turning back toFIG.2, a fourth stage IV of the IVF cycle100involves transferring an embryo (e.g., embryo116A) from the first drop location1of the holding dish130to a biopsy dish142(e.g., to remove a few cells from the embryo that will be used for genetic testing). Before the physical transfer of the embryo116A, the camera22captures an image of a pipette146adjacent to the first drop location1of the holding dish130, as shown inFIG.5. The system14then identifies a status or condition of the pipette146, using a pipette identification model that classifies correlations of a pipette location relative to a drop location (i.e., pipette is adjacent to the drop location) and/or a pipette-in-drop identification model that classifies correlations of the pipette contacting the drop (i.e., pipette receives embryo assigned to the drop location). If the pipette146was identified as being in contact with the drop, the system14then stores the status data and remembers the pipette146is holding the embryo116A that was formerly associated with the first drop location1of the holding dish130. After an embryologist brings the second dish40under the microscope18, the camera22captures a wide-view image (i.e., an image capturing an entire layout of the dish) of the second dish40, and the system14receives the image and identifies that a different dish is under the microscope18. The system14can identify that the dish40under the microscope18is a biopsy dish142by recognizing a single, centrally disposed drop134on the dish142, and/or a patient label associated with the embryo116A.

The status or condition of the pipette may be related to location of the pipette (e.g., adjacent to a drop, in contact with a drop, adjacent to a PCR tube), the contents of the pipette (e.g., delivering biological material, receiving biological material, not containing biological material, containing an embryo, containing a biopsy, containing multiple embryos, etc.). The status or condition of the pipette can also be assigned to the drop that is receiving the biological material, or the drop in which the pipette is aspirating the biological material. Additionally, the status or condition of the pipette or drop may be independently processed from providing real-time feedback to the embryologist.

Using a pipette-in-drop identification model, the system14can distinguish when a pipette146is contacting a drop134disposed on the dish38. Referring toFIG.5, first, the system14determines that the pipette146enters a layer of oil64on the dish38by identifying a first meniscus68adjacent to the pipette146. The imaging system14processes the presence of the first meniscus68created by the pipette146and layer of oil64to determine that the pipette146is about to retrieve or deliver a biological material to the drop134. The system14also determines that the pipette146enters the drop134by identifying a second meniscus72adjacent to the pipette146and closer to a distal end of the pipette146. The system14processes the presence of the second meniscus72created by the pipette146and the drop134. When the system14recognizes and identifies two menisci68,72, the system14determines that the pipette146has entered the drop134to deliver or retrieve a biological material. Accordingly, the system14can distinguish when a pipette146is merely adjacent to the drop134, using the pipette identification model describe above, or is in fluid communication with the drop134and receives or delivers the biological material using the pipette-in-drop identification model.

As the embryologist brings the pipette146near the second dish40, the system14analyzes the received images and determines whether the second dish40is an appropriate dish into which the embryologist can deposit the embryo116A. If the second dish40is the correct biopsy dish142assigned to the subject, the system14delivers a “correct” message, such as a visual, audible, and/or tactile indicator, for the embryologist to proceed with transferring the embryo116A that is held in the pipette146to a drop134on the second dish40. However, if an incorrect dish with an identical drop pattern is brought under the microscope18, for example, the system14can distinguish the first dish38from the second dish40by identifying a different visual characteristic (e.g., a marking associated with an embryo of the subject) of a plurality of characteristics that may be stored in the database. In this case, the system14would deliver an “incorrect” message or signal to the embryologist.

Specifically, the user interface30has a speaker36that is configured to play a sound to deliver a “correct” message and a different sound to deliver an “incorrect” message when prompted by the system14. For example, before the embryologist transfers the biological material to a different location, the system14signals to the user interface30to deliver either the “correct” or “incorrect” message via the speaker36by playing the sound corresponding to the embryologist's actions. For example, each of the “correct” and “incorrect” messages has a distinct sound audible by the embryologist to alert the embryologist that the move or transfer the embryologist is about to make is either correct or incorrect. Additionally, the user interface30is configured to temporarily flash a message or color on a screen37of the user interface30to deliver “correct” and “incorrect” messages when prompted by the system14. For example, before the embryologist transfers the biological material to a different location, the system14signals to the user interface30to display a first color or text on the screen37to deliver the “correct” message or display a second color or text on the screen37to deliver the “incorrect” message. In other examples, the assembly10may include a separate speaker and/or a separate light communicatively coupled to the system14to display or deliver “correct” and “incorrect” messages.

Afterwards, the biopsy dish142is taken away from the work surface26to take a biopsy from the embryo116A. After the biopsy process, the biopsy dish142returns to the work surface26with the drop134holding both the embryo116A and a biopsy of the embryo116A. The system14again receives and analyzes the images of the drop134holding both the embryo116A and the biopsy, and identifies that the biopsy dish142has both a biopsy and embryo116in the drop134.

At a fifth stage V shown inFIG.2, the embryo116A, after having the biopsy taken from it, is transferred back to the holding dish130. Again, before any physical transfer occurs, the system14identifies a status or condition of the pipette146at the biopsy dish142as the pipette is adjacent to the drop134using the pipette-in-drop identification model (i.e., pipette receiving embryo assigned to drop location) and stores the status data. After the embryologist brings the holding dish130back under the microscope18, the camera22captures a wide-view image of the holding dish130, and the system14receives the image and identifies that the holding dish130is once again under the microscope18.

As the embryologist brings the pipette146near the holding dish130, the system14analyzes the received images and determines whether the holding dish130is an appropriate dish for the embryologist to deposit the embryo116A. The system14also identifies whether the pipette146is adjacent to the correct drop location of the holding dish130using the pipette identification model. If the holding dish130is the correct holding dish130assigned to the subject and the pipette146is adjacent to the drop location1from which the embryo116A was originally drawn, the system14delivers (e.g., via the user interface30) a “correct” message, such as a visual, audible, and/or tactile indicator, for the embryologist to proceed with transferring the embryo116A held in the pipette146to the drop134at the first drop location1on the holding dish130. The embryologist can then return to the biopsy dish142—now containing a single drop134with only the biopsy of the embryo116A—to remove the biopsy from the biopsy dish142and transfer the biopsy to a wash dish150to, for example, prepare the biopsy for genetic testing, as described below.

If, on the other hand, the holding dish130is not the correct holding dish130assigned to the subject, or if the pipette146is adjacent to an incorrect drop location, the system14delivers (e.g., via the user interface30) an “error” message, such as a visual, audible, and/or tactile indicator, alerting the embryologist not to proceed with transferring the embryo116A held in the pipette146to the holding dish130and/or to the incorrect drop location on the holding dish130.

At a sixth stage VI, the biopsy from the biopsy dish142is transferred to a wash dish150having three separate washing drops134A,134B, and134C. Once again, before physically transferring the biological material between dishes, the system14first identifies a status or condition of the pipette146at the biopsy dish142as the pipette146is adjacent to the drop134using the pipette identification model and stores the status data. Once the pipette146is in the drop134, the system14, using the pipette-in-drop identification model, identifies and records the status or condition of the pipette146as receiving the biopsy. After receiving the biopsy in the pipette146, the embryologist replaces the biopsy dish142with the wash dish150under the microscope18. The camera22captures a wide-view image of the wash dish150, and the system14receives the image and identifies that the wash dish150is a different dish from the biopsy dish142. The system14can identify that the dish under the microscope is a wash dish150by recognizing a pattern of three separate drops134A,134B,134C that are centrally disposed on the dish150, or by reading and recognizing another characteristic on the dish150.

As the embryologist brings the pipette146holding the biopsy near the first wash drop134A of the wash dish150, the system14analyzes the received images and determines whether the wash dish150is an appropriate dish for the embryologist to deposit the biopsy, and whether the first wash drop134A is the correct drop in accordance with SOP of the IVF cycle. If the washing stage VI is the correct stage of the IVF cycle, the system14delivers (e.g., via the user interface30) a “correct” message, such as a visual, audible, and/or tactile indicator, for the embryologist to proceed with transferring the biopsy held in the pipette146to the first wash drop134A on the wash dish150. However, the embryologist will receive an “error” message if the dish or the location on the dish is incorrect or does not correlate with SOP.

At the sixth stage VI, the system14tracks the biopsy as the embryologist moves the biopsy from the first wash drop134A to a second wash drop134B and from the second wash drop134B to a third wash drop134C. As the biopsy moves from one drop to another, the system14processes the movement of the biopsy and records the new location of the biopsy in the wash drop. After picking up the biopsy from each wash drop, the system14receives and analyzes images of the wash dish150and pipette146, and identifies the status or condition of the pipette146holding the biopsy using the pipette identification model and/or the pipette-in-drop identification model. If, for example, the embryologist picks up the biopsy from the first wash drop134A and places the pipette146adjacent to the third wash drop134C (thereby skipping the second wash drop), the system14will recognize the movement of the pipette146as out of sequence compared to a stored order of washing steps according to SOP, and will deliver an “error” message.

At a seventh stage VII of the IVF cycle inFIG.2, a washed biopsy154is transferred to a PCR tube158having a unique identifier162(e.g., barcode, 2D barcode, QR code, numbers, letters, or a combination thereof). Again, before physically transferring the biopsy from the pipette146, the system14identifies a status or condition of the pipette146adjacent to the PCR tube158using the pipette identification model and stores the status data. The camera22captures a wide-view image of the PCR tube158under the microscope18and delivers the image to the system14. From the image, the system14identifies the PCR tube158as a different vessel than the wash dish150(e.g., via a PCR tube identification model). The system14also reads the unique identifier162of the PCR tube158and determines that the PCR tube158corresponds with the embryo116A from which the biopsy154was taken and is associated with the correct subject. The system14causes the user interface30to deliver a “correct” message, such as a visual, audible, or tactile indicator, to the embryologist to proceed with transferring the biopsy154held in the pipette146to the PCR tube158. The system14can also track when the pipette146goes into and comes out of the PCR tube158using the pipette-in-drop identification model. The PCR tube158containing the biopsy is then sent to genetic testing at an eighth stage VIII.

If a PCR tube158has been identified but the identifier162cannot be seen, the system14will prompt the embryologist to rotate the tube158until the identifier162can be seen.

The unique identifier162can be a pre-printed 2D barcode that is imaged by the camera22and processed by the system14. The system14records the transfer of the biopsy from the wash dish150to the PCR tube158and records the location of the biopsy154with the unique identifier162of the PCR tube158.

At a ninth stage IX, the embryo116A from the holding dish130is transferred to a pre-vitrification dish166to prepare the embryo116A for vitrification in the cryopreservation device32. Once again, before physically transferring the biological material between dishes, the system14first identifies a status or condition of the pipette146at the holding dish130as the pipette146is adjacent to the drop134at the first drop location1using the pipette identification model and stores the status data. After delivering a “correct” message to the embryologist, the system14then identifies a status or condition of the pipette146entering the drop134at the first drop location1using the pipette-in-drop identification model (i.e., pipette receiving embryo). After receiving the embryo116A in the pipette146, the embryologist replaces the holding dish130with the pre-vitrification dish166under the microscope18. The camera22captures a wide-view image of the pre-vitrification dish166, and the system14receives the image and identifies that the pre-vitrification dish166is a different dish from the holding dish130. The system14can identify that the dish under the microscope18is a pre-vitrification dish166by recognizing a pattern of two rows of three separate washing drops134D,134E,134F,134G,134H,134I that are centrally disposed on the dish166, or by reading and recognizing another characteristic on the dish166.

As the embryologist brings the pipette146holding the embryo116A near the first wash drop134D of the pre-vitrification dish166, the system14analyzes the received images and determines whether the pre-vitrification dish166is an appropriate dish for the embryologist to deposit the embryo116A, and whether the first wash drop134D is the correct drop in accordance with SOP of the IVF cycle. If the pre-vitrification processing stage is the correct stage of the IVF cycle100, the system14delivers (e.g., via the user interface30) a “correct” message, such as a visual, audible, or tactile indicator, for the embryologist to proceed with transferring the embryo116A held in the pipette146to the first wash drop134D on the pre-vitrification dish166. On the other hand, the embryologist will receive an “error” message if the dish or the location on the dish is incorrect or does not correlate with SOP.

Additionally at the ninth stage IX, the system14tracks the embryo116A as the embryologist moves the embryo116A from the first wash drop134D to a second wash drop134E, from the second wash drop134E, and to a third wash drop134F. Fourth, fifth, and sixth wash drops134G,134H,134I are used for another embryo. According to SOP, the embryo116A is placed in each drop for a pre-determined amount of time, and each drop may have a different wash time. After picking up the embryo from each pre-vitrification wash drop134D,134E,134F, the system14receives and analyzes images of the pre-vitrification dish166and pipette146, and identifies the status or condition of the pipette146holding the embryo116A using the pipette identification model and/or pipette-in-drop identification model at each drop. The system14initiates a timer for a set period of time the embryo116A should spend in each wash drop134D-I, and alerts the embryologist when the embryo116A should be retrieved and transferred to the next drop. If, for example, the embryologist picks up the embryo116A from the first wash drop134D and places the pipette146adjacent to the third wash drop134F (thereby skipping the second wash drop135E), the system14will recognize the movement of the pipette146as out of sequence compared to a stored order of pre-vitrification washing steps according to SOP, and will deliver an “error” message.

At a tenth stage X, the embryo116A is transferred from the third drop134F of the pre-vitrification dish166to a vitrification device (such as a VitriGuard® or Cryotop®)170having a unique identifier. The camera22captures a wide-view image of the vitrification device170under the microscope18and delivers the image to the system14. From the image, the system14identifies the vitrification device170as a different vessel than the pre-vitrification dish166(e.g., via a vitrification device identification model). The system14also reads the unique identifier of the vitrification device170and determines that the vitrification device170corresponds with the embryo116A. The system14causes the user interface30to deliver a “correct” message, such as a visual, audible, or tactile indicator, to the embryologist to proceed with transferring the embryo116A held in the pipette to the vitrification device170. The system14can also track when the pipette146goes into and comes out of the vitrification device170. The vitrification device170holding the embryo116A is then plunged into liquid nitrogen to vitrify the embryo, before it is placed in the cryopreservation device32, where the embryo116A is stored while the biopsy is tested.

Turning now toFIG.6, a flow chart represents an example method1100for tracking a subject's biological material in an IVF process, such as the process100ofFIG.2. The method1100is performed by the imaging system14via one or more computers and includes a step1102of receiving, from a camera22, an image of a dish38where the dish has a visual characteristic138. In the illustrated example, the dish38is holding a biological material at a drop location. The visual characteristic138may be one or more of a marking, drop pattern, barcode, name, number, a combination of characters, or other identifier that identifies a subject, biological material, dish type, dish orientation, and/or drop type. The system14then processes the image in step1104, using a drop identification model, to identify the drop134according to one or more visual characteristics138on the dish38. The system14assigns, in step1106, an identifier to the drop134associated with the drop location, and records in the memory the identifier (e.g., drop adjacent to marking) of the drop134associated with the drop location1in step1108. For example, the system14receives an image of the dish38from the camera22when the dish38is at the third stage III of the process100illustrated inFIG.2. Using the drop identification model of the detection model56, the system14processes the image of the drop to identify and assign an identifier (e.g., drop adjacent to marking138) to the drop134as being associated with the first drop location1.

In another step, the system14may further process the image, using a material identification model, to classify a type of biological material (e.g., one or more embryos, biopsy, embryo and biopsy) associated with the drop location on the dish38. The system14identifies the biological material associated with the drop location, and records an identifier (e.g., “embryo1of Subject X”) associated with the drop location1. For example, the system14receives an image of the dish38from the camera22when the dish38is at the third stage III of the process100illustrated inFIG.2. Using the material identification model of the detection model56, the system14processes the image of a biological material in a drop to classify the biological material as a first embryo116A associated with the first drop location1.

In another step, the system14may further process the image of the dish38, using a dish identification model, to uniquely identify the dish38according to the visual characteristic138and/or classify a type of dish according to the visual characteristic138and then identify (e.g., by recording in the memory) the dish according to that visual characteristic. For example, the system14processes the marking138(i.e., the visual characteristic) of the holding dish130(FIGS.3-5) to classify an orientation of the dish, and then maps each drop location1,2, and3to a different embryo116A,116B, and116C as each embryo is disposed in its respective drop134. The system14then stores this data in a memory to track the movement of each embryo116during the IVF process.

At the same time, the system14can process the image of the dish, using a subject identification model, to classify a subject identification associated with the dish, and record in the memory52the subject identification associated with the dish. For example, the system14can process a subject identifier (e.g., a unique ID associated with the patient) disposed on the dish38, and record that the dish38that is under the microscope18is associated with the subject. This ensures that the transfer of biological material of the subject remains with the dishes associated with the subject throughout the IVF process. In other examples, the system14communicates with the RFID reader33to associate the subject with the dish under the microscope18.

Additionally, the system14can process the image of the dish38having a drop pattern, using a drop pattern identification model, to classify a type of dish associated with the drop pattern. For example, the system14processes the drop pattern of the dish38to classify the dish38as a holding dish130by recognizing a circular drop arrangement and containing a plurality of embryos116A,116B, and116C in separate drops134. The system14can learn, using machine learning techniques (described below), how to recognize different dishes by identifying drop patterns and determining the likelihood of proper dish classification. The steps1102through1108of the method1100can be performed at various stages of the process100before retrieving an embryo or biopsy with the pipette146from any drop or vessel (e.g., dish, tube, device, etc.).

In another step, the system14may further process the image of the vitrification device170, using a vitrification device identification model, to classify a type of vitrification device according to the visual characteristic and then identify (e.g., by recording in the memory) the vitrification device according to that visual characteristic. In yet another step, the system14may further process the image of the PCR tube, using a PCR tube identification model, to classify a type of PCR tube according to the visual characteristic and then identify (e.g., by recording in the memory) the PR tube according to that visual characteristic.

Before the embryologist retrieves the biological material from the drop location, the method1100may further include a step of processing an image of the dish38and pipette146to identify a first status or condition of the pipette146at or near the drop location. After a predetermined time has passed, or after processing and identifying that the pipette146enters the drop at the drop location, the system14determines that the pipette146receives the biological material at the drop location. The system14records in the memory the first status or condition of the pipette146holding the biological material. For example at stage III of the process100illustrated inFIG.2, the system14determines that the pipette146retrieves the embryo116A at the drop location1and determines that the pipette146is now holding the embryo116A of the subject. The system14then records that the location of the embryo116A is now in the pipette146.

The method1100may further include a step of identifying a second status or condition of the pipette146holding the biological material at a second location. Before the biological material is delivered to the second location, the system14determines whether the second location for depositing the biological material correlates with SOP stored in a database of the memory52. This step includes receiving an image from the camera22of the second location (e.g., a different dish or vessel or a different drop on the same dish), and processing the image to classify the second location. The system14provides real-time feedback to an embryologist that the second drop location is the correct or incorrect drop location before the embryologist delivers the biological material to the second location. Once the biological material is delivered, the system14records a delivery status of the biological material from the pipette and to the second location.

For example at stage IV of the process100illustrated inFIG.2, the system14identifies that the pipette146holding the embryo116A is hovering over a different dish40, and processes the image of the second dish40and pipette146to identify that the pipette146is disposed above a biopsy dish142having a single, centrally-disposed drop134. In accordance with SOP of the IVF process stored in the database, the system14delivers a “correct” signal or message to the embryologist to proceed with delivering the embryo116A held in the pipette146to the drop134on the biopsy dish142. The system14then records the new location of the embryo116A associated with the subject. This method step can be repeated (e.g., identifying a third status or condition, a fourth status or condition, etc.) before delivering an embryo or biopsy using the pipette146from any drop or vessel (e.g., dish, tube, device) at subsequent stages of the IVF process. After each delivery, the system14records the delivery status and new location of the biological material.

As briefly discussed above, a machine learning model may be configured to process a model input that includes a set of drop patterns for a dish to generate a model output that characterizes a likelihood that the drop pattern is associated with a particular type of dish. A few examples of possible model outputs of the machine learning model are described next.

In some implementations, the model output of the machine learning model can include a hard classification that identifies the dish as being included in one category from a set of categories that includes: a culture dish120(i.e., indicating that the dish under the microscope has one or more drops containing one or more embryos each from a single subject), a holding dish130(i.e., indicating that the dish under the microscope has a plurality of drops, each drop containing one embryo), a biopsy dish142(i.e., indicating that the dish under the microscope has one drop containing a single embryo, a single biopsy, or a single embryo and a single biopsy), a wash dish150(i.e., indicating that the dish under the microscope has separate drops in a row), and a pre-vitrification dish166(i.e., indicating that the dish under the microscope has a plurality of rows of drops). These categories are determined specifically for the IVF process100ofFIG.2, and may vary according to the particular IVF lab.

In some implementations, the model output of the machine learning model can include a soft (probabilistic) classification that defines a score distribution over a set of categories. The set of categories can include a culture dish, a holding dish, a biopsy dish, a wash dish, and a pre-vitrification dish, as described above. The score for each category can define a likelihood (probability) that the dish is included in the category.

The machine learning model can have any appropriate machine learning model architecture that enables the machine learning model to perform its described functions. For instance, the machine learning model can be implemented, for example, as a neural network model, or a random forest model, or a support vector machine model, or a decision tree model, or a linear regression model, etc. In implementations, where the machine learning model is implemented as a neural network model, the machine learning model can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, or 50 layers) and connected in any appropriate configuration (e.g., as a linear sequence of layers). In implementations where the machine learning model is implemented as a decision tree model, the machine learning model can include any appropriate number of vertices, and can implement any appropriate splitting function at each vertex.

The machine learning model can include a set of machine learning model parameters. For instance, for a machine learning model implemented as a neural network model, the set of machine learning model parameters can define the weights and biases of the neural network layers of the machine learning model. As another example, for a machine learning model implemented as a decision tree, the set of machine learning model parameters can define parameters of a respective splitting function used at each vertex of the decision tree. To generate a model output, the machine learning model can process a model input in accordance with values of the set of machine learning model parameters.

A screening system can use a training system to train the machine learning model on a set of training examples. More specifically, the training system can determine trained values of the set of machine learning model parameters of the machine learning model by a machine learning training technique.

The training system uses a training engine to train the set of machine learning model parameters of the machine learning model on a set of training examples. Each training example can correspond to a dish (referred to for convenience as a “training dish”) and can include: (i) a model input that includes a set of drop patterns characterizing the dish, and (ii) a target dish classification of the dish under the microscope. For each training example, the training engine trains the machine learning model to process the model input of the training example to generate a model output that matches the target dish classification of the training dish. More specifically, the training engine trains the machine learning model, by a machine learning training technique, to optimize an objective function that measures an error between: (i) the model output generated by the machine learning model for the training dish, and (ii) the target dish classification of the training dish. The objective function can measure the error between a model output and a target dish classification in any appropriate way, e.g., as a squared error or as an absolute error.

The training engine can train the machine learning model using any machine learning training technique appropriate for the architecture of the machine learning model. For instance, if the machine learning model is implemented as a neural network model, then the training engine can train the machine learning model using stochastic gradient descent.

While the categories of the disclosed machine learning model includes culture dish, holding dish, biopsy dish, wash dish, and pre-vitrification dish, each dish having a particular pattern shown inFIG.2, the categories may be defined differently and according to a particular IVF process. For example, different stages of the IVF process100ofFIG.2can vary by IVF lab. For example, in some IVF labs, the biopsy dish may contain two or more separate drops disposed in a vertical column, each containing an embryo and/or biopsy. In this case, the machine learning model category “biopsy dish” would be associated with two or more separate drops disposed in a vertical column.

While the assembly10inFIG.1Ais described as having the camera22externally mounted to the microscope18, in certain examples, the imaging system may utilize a camera integral with the microscope or mounted to the microscope in a different way. Turning now toFIG.7, for example, an assembly210includes a microscope218including a microscope camera222and an imaging system214for tracking biological material in an IVF process, such as the IVF process100ofFIG.2, and for performing the method1100ofFIG.6. The second example assembly210is similar to the first assembly10ofFIG.1A, and the imaging system214is similar to the first imaging system14ofFIG.1B, and also includes a memory and a detection model. Thus, for ease of reference, and to the extent possible, the same or similar components of the second example assembly210will retain the same reference numbers as outlined above with respect to the first example assembly10, although the reference numbers will be increased by 200. However, the second example assembly210differs from the first example assembly10in the manner discussed below.

The second example microscope218includes a microscope camera222integrated with a head236of the microscope218. The microscope camera222may be a digital video camera that records the microscope image, and is configured to take magnified views of a dish under the lens244of the microscope218. As shown inFIGS.8and9, dishes238,240for use with the imaging system214are labeled with numbers239and/or coordinates241to help identify a location of a drop under the microscope218. Referring specifically toFIG.8, the dish238for use with the imaging system214includes nine separate drop locations1-9, each labeled with a number adjacent to a designated area235for a drop. Referring toFIG.9, the dish240for use with the imaging system214includes a plurality of spaced-apart coordinates241arranged in a grid and displays a combination of letters and numbers to indicate coordinate positions.

Additionally, the imaging system214is configured to receive an image of a magnified view of the dish240, as shown inFIG.10. In this case, the imaging system214processes the image to identify the dish240according to a visual characteristic on the dish with a drop location of the dish. The visual characteristics of the dishes240ofFIGS.9and10are labeled with coordinates241to identify the drop location (e.g., A3, A4, B2, B3, etc.).

The microscope camera222can send images of the biological material disposed in the drops on the dish, and the imaging system214can process the image to identify the biological material (e.g., embryo, biopsy, or both embryo and biopsy) associated with the drop location of the dish. For example, after the biopsy process, the biopsy dish142returns to the work surface226with the drop134holding both the embryo116A and a biopsy of the embryo116A. The imaging system214again receives and analyzes the images (taken by the microscope camera222) of the drop134holding both the embryo116A and the biopsy, and identifies that the biopsy dish142contains both a biopsy and the embryo116in the drop134.

While the imaging systems14,214described above rely on images obtained from either a camera mounted externally to the microscope or to a microscope camera of the microscope, in certain embodiments, the imaging system may utilize images from both types of cameras. Turning now toFIG.11, for example, a third example assembly310includes an imaging system314for tracking biological material in an IVF process, such as the process100depicted inFIG.2, and for performing the method1100ofFIG.6. The third example assembly310is similar to the assembly10ofFIG.1A, and the imaging system314is similar to the imaging system14ofFIG.1B, and also includes a memory and a detection model. Thus, for ease of reference, and to the extent possible, the same or similar components of the third example assembly310will retain the same reference numbers as outlined above with respect to the first example assembly10, although the reference numbers will be increased by 300. However, the third example assembly310differs from the first example assembly10in the manner discussed below.

Similar to the first example assembly10, the third example assembly310includes a wide FOV camera322A coupled to a body334of the microscope318. Similar to the second example microscope218, the third example microscope318includes a microscope camera322B integrated with a head336of the microscope318. The imaging system314includes the wide FOV and microscope cameras322A,322B that are configured to capture and send images of dishes, vessels, and objects underneath the microscope318or elsewhere on the work surface326to the imaging system314for processing and tracking.

At each stage of the process100, the system314receives a wide FOV image of a dish338, as shown inFIG.12, and a magnified image of the dish338, as shown inFIG.13. The system314can be used with any one of the different dishes shown inFIGS.3-5and8-10to accurately identify and track any transfers of biological material between different dishes, vessels, pipettes, and drop locations on the work surface326.

Using the pipette-in-drop identification model described above, the system314can distinguish when a pipette146enters or exits a drop134disposed on the dish338as well as when the pipette146receives the biological material. Referring toFIG.13, first, the system determines that the pipette146enters a layer of oil364on the dish338by identifying a first meniscus368adjacent to the pipette146. The imaging system314processes the presence of the first meniscus368created by the pipette146and layer of oil364to determine that the pipette146is about to retrieve or deliver a biological material to the drop134. The system314also determines that the pipette146enters the drop134by identifying a second meniscus372adjacent to the pipette and closer to a distal end of the pipette. The system314processes the presence of the second meniscus372created by the pipette146and the drop134. When the system314recognizes and identifies two menisci368,372, the system314determines that the pipette146has entered the drop134to deliver or retrieve a biological material. Further, using the microscope camera322, the imaging system314can determine whether the pipette146receives the biological material into the pipette146or delivers the biological material into the drop. In some examples, the microscope322camera may also be able to detect when the tip of the pipette146is in the drop134by seeing that it comes more clearly into focus, or bends as it touches the bottom of the dish, or some other indication. The system214including the microscope camera222ofFIG.7can also be configured to identify this level of detail, as well.

InFIGS.14-20, alternative systems for tracking biological material during an IVF process are illustrated. The assemblies ofFIGS.14-20may be configured for tracking biological material without using image recognition software or machine learning. Instead, the assemblies ofFIGS.14-20rely on RFID technology to identify the unique ID of each dish under the microscope. The unique drop identities for each drop on the dish is made up of RFID tag code and drop position. For ease of reference, and to the extent possible, the same or similar components of each assembly will retain the same reference numbers as outlined above with respect to the first example assembly10discussed above, although the reference numbers will be increased by 400 for the assembly410ofFIG.14, and 100 thereafter.

Turning first toFIG.14, an example assembly410for tracking a subject's biological material in a lab includes a microscope418, an RFID or barcode reader422coupled to the microscope418, a computer424coupled to the RFID tag or barcode reader422, and a user interface430coupled to the computer424. The RFID tag or barcode reader422is configured to read an RFID tag or barcode439on a dish438at a central location directly beneath the microscope418. To ensure the right drop is being read, the embryologist would need to align the drop with cross hairs through the scope. In practice, after an embryologist moves the dish438to align a narrow field of the RFID tag or barcode reader422with a drop440on the dish438, the RFID tag or barcode reader422reads the RFID tag or barcode439adjacent to the examined drop440, processes the RFID tag or barcode439associated with the examined drop440, and displays the dish438with the examined drop440highlighted on the user interface430. The embryologist can input information (e.g., type of biological material) related to the examined drop440by directly using the user interface430.

In the example ofFIG.14, the RFID or barcode reader422is attached to the lens of the microscope418. However, in other examples, the RFID or barcode reader422could be mounted under the glass448that the dish438sits on. In yet another example, an RFID or barcode reader may be integrated with a camera of the microscope.

InFIG.15, another example assembly510for tracking a subject's biological material includes a microscope518, a computer524, a user interface530coupled to the computer524, and a side-mounted RFID or barcode reader522coupled to the computer524. The RFID or barcode reader522has an attachment arm523, opaque working platform525, and a ledge527with an integrated RFID or barcode scanner529. The ledge527has a semi-circular cut-out that is shaped to receive a circular dish538. In particular, the RFID or barcode reader522is configured to work with the dish538having a plurality of drops540disposed in a circular arrangement. A circumferential wall543of the dish is perpendicularly disposed relative to the dish surface and displays mounted RFID tags or barcodes539associated with each drop540on an exterior surface. In one example, the barcodes539can be same in each dish (e.g., molded in), and then the combination of RFID tag and the drop location provides a unique drop ID. When the wall543of the dish528is moved up against the ledge527, an embryologist can rotate the dish538to align a particular drop540with a hole541in the working platform525. The opaque working platform525blocks the light underneath the dish538, and the hole541permits light to shine through to assist with aligning of the examined drop540. The RFID or barcode scanner529scans an RFID tag or barcode539disposed on the external surface of the circumferential wall543of the dish538, and sends information associated with the scanned RFID tag or barcode539to the computer524. The drops540on the dish538are peripherally disposed such that the dish438need only be rotated against the ledge527to align the hole541with a different drop540and the scanner529with the RFID tag or barcode539of the different drop540.

FIG.16depicts another example assembly610for tracking a subject's biological material, and includes a microscope618, a computer624, a platform625with integrated first and second readers622A,622B and coupled to the computer624, and a user interface630coupled to the computer624. The first and second readers622A,622B are perpendicularly disposed relative to one another, barcode readers, or other character readers. The assembly610is configured to map multiple drops640on a rectangular dish638by scanning X, Y coordinate markers on the dish638corresponding with the drops640. For example, the dish638includes X coordinate markings639on a first side643, and Y coordinate markings of a second side645of the dish638.

Initially, an embryologist will input into the user interface630the type of dish that will be examined under the microscope618. After aligning a drop640with a central location denoted by a hole641formed in the opaque platform625, the first reader622A reads an X coordinate on the first side643of the dish638and the second reader622B reads a Y coordinate on the second side645of the dish638. The readers622A,622B send the scanned X, Y coordinates to the computer624. The computer624then processes the data inputted by the embryologist and received from the readers622A,622B to map the drop640being examined on the dish638, which then is displayed on the user interface630

In yet another example inFIG.17, a biological material tracking assembly710includes a microscope718, an X-Y coordinate bracket assembly755, a computer724, and a user interface730coupled to the computer724. The X-Y coordinate bracket assembly755includes an L-shaped bracket758including a first arm757coupled to a base734of the microscope718via a coupler759, and a second arm761coupled to the first arm757and perpendicularly disposed relative to the first arm757. A movable frame763is coupled to the second arm761and includes an opening765sized and shaped to receive a dish738. In particular, the frame763includes a notch767or other female locking component that releasably receives and couples to a protrusion769or other male locking component extending from a circumference of the dish738.

Initially, an embryologist will input into the user interface730the type of dish that will be examined under the microscope718. To ensure the right drop is being read, the embryologist aligns the drop with cross hairs through the microscope718. Once the frame763receives the dish738, the dish738can move by sliding the frame763in an X direction along the second arm761and sliding the bracket758in a Y direction relative to the coupler759. The frame763is configured to move incrementally relative to location markers on the first and second arms757,761. Any movement in the X and Y directions is measured via one or more electronic measurement devices integrated into the L-shaped bracket758and/or coupler759. When an examined drop is underneath the microscope718, the electronic measurement devices integrated with the bracket assembly755sends the measured coordinates of the dish738to the computer724. The computer724then processes the data inputted by the embryologist and received from the electronic measurement devices to map the location of the examined drop. The drop being examined under the microscope718and may be displayed on the user interface730.

Similar to the tracking assembly710ofFIG.17, a tracking assembly810ofFIG.18includes a microscope818, a coordinate bracket assembly855coupled to a base834of the microscope818, a computer824coupled to the bracket assembly855, and a user interface830coupled to the computer824. However, unlike the coordinate bracket assembly755ofFIG.17, the coordinate bracket assembly855ofFIG.18is configured to measure radial coordinates of the dish838to infer drop location on the dish839. The bracket assembly855includes a frame863defining an opening865sized to receive the dish838and a notch867or other female locking component that releasably receives and couples to a protrusion869, or other male locking component, that extends from a circumference of the dish838. The bracket assembly855also includes a sliding arm857coupled to the frame863, and a coupler859that couples the arm857to a base834of the microscope818. An electronic measurement device is integrated into the coupler859to measure angular displacement of the frame863relative to the base834(i.e., movement in the G direction), and a different electronic measurement device is integrated into the arm857to measure radial displacement relative to the base834(i.e., movement in the R direction).

Initially, an embryologist will input into the user interface830the type of dish that will be examined under the microscope818. Once the dish838is placed in the opening865of the frame863, the dish838can move by swiveling the frame863relative to the coupler859and by sliding the frame863relative to the arm857. When an examined drop is underneath the microscope818(aligned using a cross-hairs through the microscope818, for example), the electronic measurement devices send the angular and radial coordinates of the dish838to the computer824. The computer824then processes the data inputted by the embryologist and received from the electronic measurement devices to map the location of the examined drop. The coordinates are mapped to the drops under the microscope818and displayed on the user interface830.

In the examples ofFIGS.17and18, each respective dish738,838includes a protrusion769,869that couples to a corresponding notch767,867or indentation of the frame763,863. However, in other examples, each respective dish738,838may include a notch or indentation (or other female locking component) and the frame763,863may include a protrusion (or other male locking component).

InFIG.19, a biological material tracking assembly910includes a microscope918, an infrared (IR) detector, a computer924coupled to the IR detector922, and a user interface930coupled to the computer924. The IR detector922includes first and second horizontal sensors971,973that emit IR light (i.e., an LED) and detect the light reflecting off of a scanned object. In particular, the IR detector922can detect an orientation of a known dish938by detecting a physical characteristic969, such as a protrusion, of the dish938. Initially, an embryologist will input into the user interface930the type of dish that will be examined under the microscope918. To ensure the right drop is being read, the embryologist aligns the drop with cross hairs through the microscope918. Once in place, the horizontal sensors971,973scan the dish938on both sides of the protrusion969to determine the position and orientation of the dish938. The sensors971,973sends the data to the computer924, which processes both the data inputted by the embryologist and received from the sensors971,973to map the location of the examined drop on the dish938The computer924displays the dish938and the highlighted examined drop940on the user interface930.

In yet another example inFIG.20, a tracking assembly1010includes a microscope1018, a capacitive screen1025below the microscope1018, a computer1024, a user interface1030coupled to the computer1024, and a compatible dish1038. The dish1038includes three contacts1069, each having a slight protrusion (as shown in magnified perspective and side views inFIG.20) extending from a bottom surface of the dish1038that contacts the capacitive screen1025when the dish1038is placed on the capacitive screen1025. To ensure the right drop is being read, the embryologist aligns the drop with cross hairs through the microscope1018. The capacitive screen1025recognizes the contacts, and registers the contacts of the dish1038to identify the position and orientation of the dish1038relative to a central location1041of the microscope1018. Specifically, computer1024processes the three contacts to infer the position and orientation of the dish1038relative to an examined drop1040, and then displays the highlighted drop1040on the user interface1030.

FIG.21is block diagram of an example computer system1200that can be used to perform operations described above. The system1200includes a processor1210, a memory1220, a storage device1230, and an input/output device1240. Each of the components1210,1220,1230, and1240can be interconnected, for example, using a system bus1250. The processor1210is capable of processing instructions for execution within the system1200. In one implementation, the processor1210is a single-threaded processor. In another implementation, the processor1210is a multi-threaded processor. The processor1210is capable of processing instructions stored in the memory1220or on the storage device1230.

The memory1220stores information within the system1200. In one implementation, the memory1220is a computer-readable medium. In one implementation, the memory1220is a volatile memory unit. In another implementation, the memory1220is a non-volatile memory unit.

The storage device1230is capable of providing mass storage for the system1200. In one implementation, the storage device1230is a computer-readable medium. In various different implementations, the storage device1230can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

The input/output device1240provides input/output operations for the system1200. In one implementation, the input/output device1240can include one or more network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and/or a wireless interface device, e.g., and 802.11 card. In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other input/output devices, e.g., keyboard, printer and display devices1260. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

The tracking assemblies10,210,310,410,510,610,710,810,910,1010ofFIGS.1A,7,11, and14-20may be integrated with a subject identification system that includes an RFID reader configured to read an RFID tag on each of the dishes that is placed on the work surface. The RFID reader33inFIG.1A, for example, is a large platform integrated with, or placed on top of, a work bench. As a dish is put onto the work surface, the RFID reader33automatically reads the RFID tag disposed on a bottom of the dish. The RFID tag identifies the subject of the dish. In some examples, a second, smaller reader may be placed around the microscope light source48to read the tag under the microscope18, as opposed to each dish on the RFID reader33. In another example, the RFID reader33may be divided into various zones to identify the dish relative to the various zones of the work surface26.

The imaging systems14,214,314described above with respect toFIGS.1A,1B,7, and11operate on a computer24,224,324and each includes one or more cameras, a memory52, and detection model56. However, in another example, the imaging system may have more or fewer components. In another example, the memory and or detection model of the imaging system may be integrated with one or more cameras, the microscope, the user interface, or the cloud instead of the local computer24,224,324.

In the assembly10ofFIG.1Aand the microscope318ofFIG.11, the wide FOV cameras22,322are mounted to the bodies34,334of the microscopes18,318, respectively. However, in other examples, the cameras22,322may not be directly mounted to the microscope, and may instead be coupled to a different mount or positioned on the work surface26,326.

The IVF cycle100ofFIG.2is an example process, and may include more or fewer stages. For example, the IVF cycle100was described tracking a single embryo116A and the biopsy taken from the embryo116A. However, in some examples, the imaging system may be configured to track multiple embryos processed at a time.

While the dish inFIGS.3-5includes a line as the characteristic, other example dishes for use with a wide field of view camera may include other visual characteristics such as, for example, dots, symbols, letters, numbers, and shapes created by painting, printing, etching, molding, labeling, or otherwise marking the dish directly.

In the illustrated example, the second stage of the IVF cycle depicts multiple embryos disposed in two drops on a culture dish. However, in other examples, the second stage of the IVF cycle includes a time lapse incubator. In this example, the third stage III involves transferring the embryos form the time lapse incubator to a holding dish, as shown inFIG.2.

In some examples, the biopsy dish142may contain more than one drop, and each drop may contain an embryo and associated biopsy. Similarly, the wash and pre-vitrification dishes150,166may be configured to contain drops for multiple biopsies and/or embryos.

In some examples, the imaging systems14,214,314may be configured to measure time that the embryo and/or biopsy resides in a particular drop. For example, at the sixth stage VI of the process100illustrated inFIG.2, the system14,214,314may recognize when the biopsy is delivered to each wash drop and initiate a timer for a set period of time the biopsy should spend in each wash drop.

At the seventh stage VII of the process100illustrated inFIG.2, the PCR tube is pre-labeled with a unique patient identifier. However, in other examples, the system may be configured to print a unique patient identifier when an empty PCR tube is scanned at the seventh stage. The embryologist places the biopsy in the tube immediately before or after printing and securing the label to the PCR tube to ensure accuracy. In other examples, the PCR tube may have an unassigned unique identifier attached which becomes associated with the biopsy once the biopsy is placed in the tube.

In some examples, the imaging systems14,214,314may provide a digital, visual guidance to provide feedback during the IVF process. In one example, a transparent LCD screen may be disposed under the dish, which could provide visual feedback and guidance to the embryologist while the embryologist is viewing the dish under the microscope. In another example, the imaging system14may include a microscope with an integrated graphical overlay that provides feedback and guidance while viewing the dish through the microscope. Specifically, graphical overlay may incorporate augmented reality (AR) technology. For example, the microscopes18,218,318may incorporate AR by providing a transparent screen disposed between an embryologist's eye and what is being read with the microscope. The AR technology may be coupled with the imaging systems14,214,314to give visual commands to the embryologist (e.g., highlighting a drop on the examined dish to identify where the drop should be deposited, crossing out drops that already contain biological material, crossing out entire dishes to indicate the incorrect dish is under the microscope, etc.). In another example, the embryologist could use a microscope configured with a display screen instead of eyepieces. In this case, graphical information could be overlaid onto that display screen.

In some example assemblies for tracking a subject's biological material in a lab during an IVF process, the imaging systems14,214,314may be replaced or combined with other components for inferring a position and orientation of the dish being examined. In some examples, the assembly, or specifically the microscope, may have components or features that can identify a central location so that the embryologist can identify a spot that is directly under the microscope. For example, the microscope may have a cross hair or other marker in the optical eyepiece or on the glass underneath the dish to denote a central location. The assembly may include components to block light underneath the dish except for the central location, or components that provide a colored light or laser at the center of the dish.

In some examples, the visual characteristics may include dish details, information added to the dish, information around the drops, and/or layout of different visual references relative to each other. For example, a dish may have an RFID tag on a bottom surface, and is specifically placed adjacent to a first drop location. The drops on the dish may be identified by their relative locations to the RFID tag.

The tracking assemblies10,210,310,410,510,610,710,810,910,1010ofFIGS.1A,7,11, and14-20may be used to track biological material outside of the IVF process. Further, while the imaging systems14,214,314described herein are used in an IVF process to track embryos and biopsies in an IVF lab, the imaging systems may be used to track different biological material in different processes. In such examples, a different technician may be working in the lab and interacting with the imaging system.

While the imaging systems14,214,314described above rely on images obtained from a camera mounted externally to the microscope, to a microscope camera of the microscope, or both types of cameras, in other embodiments, an imaging system may include additional multiple cameras set up through the lab space to track multiple dishes. For example, a plurality of spaced apart cameras are perpendicularly disposed relative to the horizontal work surface to image all dishes, for example, under a lab hood.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosure or of what may be claimed, but rather as descriptions of features that may be specific to particular examples of particular disclosures. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable subcombination. Moreover, although features may be described herein as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.