Patent Description:
Pathology is a medical discipline related to the study and diagnosis of disease. Most frequently pathology involves the analysis and examination of body-fluid, tissue, and cell samples. As a field of general study and research, pathology relates to four aspects of disease: (<NUM>) etiology, (<NUM>) pathogenesis, (<NUM>) morphologic changes, and (<NUM>) consequence of morphologic changes.

The field of pathology dates back to Antiquity. Many early societies possessed a rudimentary understanding of biological conditions as a result of dissection/examination. By the Hellenic period of Greece, a causal study of disease had emerged in human culture. Human understanding of disease through pathology continued to advance piecemeal as time progressed; for instance many advances in pathology are attributable to the medieval era of Islam.

However, modern pathology only emerged as a distinct field of study in the late <NUM>'s with the advent of microbiology. Now pathology is a major medical practice that is divided into a number of subdisciplines. In all of these subdiciplines, having a second opinion to check the work of a pathologist is helpful to eliminate erroneous diagnosis. Eliminating diagnostic error in pathology may result in a healthier population and reduce pathologists' exposure to liability.

<CIT> discloses an image analysis system that includes a processor and memory and displays an image to a first user. The image analysis system tracks gaze of the first user and collects initial gaze data for the first user. The initial gaze data includes a plurality of gaze points. The image analysis system identifies one or more ignored regions of the image based on a distribution of the gaze data within the image; and displays at least a first subset of the image. The first subset of the image is selected so as to include a respective ignored region of the one or more ignored regions and the first subset of the image is displayed in a manner that draws attention to the respective ignored region. In some embodiments, the ignored region is visually emphasized within the image. In some embodiments, only the first subset of the image is displayed.

<CIT> discloses a diagnostic imaging support device that includes an image data acquisition unit that acquires image data of high magnification of sample tissue, an image classifying unit that generates image data of low magnification from the image data of high magnification acquired by the image data acquisition unit, and classifies the generated image data of low magnification into group by each image data pattern of a plurality of pathological tissue, and an image evaluating unit that evaluates whether or not the image data of high magnification, which is to be a base of the image data of low magnification classified by the image classifying unit, is the pathological tissue of the classified group.

The present invention provides a method according to claim <NUM>. Further aspects of the invention are set out in the remaining claims.

Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles being described.

Embodiments of an apparatus and method for enhanced pathology diagnosis are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

The instant disclosure provides a system and method for enhancing pathology diagnosis. More specifically, these systems and methods may be used to aid a pathologist in the diagnosis of disease. A machine learning algorithm is trained to alert pathologists to regions of interest (e.g., diseased tissue, atypical cells, unusual growth, etc.) in a pathology sample when the machine learning algorithm recognizes these regions in the sample. Providing a second opinion to pathologists may both increase pathologist efficiency (by allowing the pathologist to spend less time on each slide), and decrease the probability of an inaccurate diagnosis (by alerting the pathologists to regions of interest in the sample).

<FIG> illustrates a system <NUM> for pathology diagnosis, in accordance with an embodiment of the disclosure. System <NUM> includes: microscope <NUM>, digital camera <NUM>, microphone <NUM>, screen <NUM>, processing apparatus <NUM>, network <NUM>, storage <NUM>, speaker <NUM>, first mechanical to electrical transducer <NUM>, and second mechanical to electrical transducer <NUM>.

In the illustrated embodiment, the user (e.g., a pathologist) is looking at a magnified pathology sample with microscope <NUM>. Microscope <NUM> is magnifying a pathology sample to form magnified pathology images. The magnified pathology images are recorded with digital camera <NUM> optically coupled to microscope <NUM>. Digital camera <NUM> is electrically coupled (wired or wirelessly) to processing apparatus <NUM> (e.g., a desktop computer, server, etc.) to send the magnified pathology images (still-frames or video) to processing apparatus <NUM>. Processing apparatus <NUM> compares the magnified pathology images to reference pathology images included in a pathology database (contained on remote or local storage <NUM>) to identify one or more regions of interest (e.g., outlined portion <NUM> on screen <NUM>) in the magnified pathology images. If there is a region of interest in the magnified pathology images, system <NUM> alerts the user of microscope <NUM> to the one or more regions of interest. This alert may be an audio alert (e.g., voice from speaker <NUM> saying "look in the lower left hand corner of the image there may be cancer"), visual alert (e.g., highlighting the region of interest <NUM> on screen <NUM>), or haptic alert (e.g., vibrating the stage of microscope <NUM>) in response to identifying the one or more regions of interest.

In one embodiment, a machine learning algorithm (see infra <FIG>) is used to identify the one or more regions of interest, and is trained to identify the one or more regions of interest using the reference pathology images in the pathology database. The pathology database may include many combinations of reference pathology images, annotations from a pathologist, and grading/diagnosis from a pathologist. On processing apparatus <NUM>, image processing and machine learning algorithms may be running so that image data from digital camera <NUM> is interpreted in real time and guidance can be provided to the user (pathologist) either visually on the monitor or by audio feedback (e.g., "prostate adenocarcinoma Gleason score <NUM>", "slide folded", "image out of focus", etc.). Headphones can be added to system <NUM> for convenience, and the conversation can be recorded and archived with respect to the video/images captured by digital camera <NUM>.

Reference images in the pathology database may include hundreds of thousands, or even millions, of images of pathology samples that the machine learning algorithm has been trained to recognize as diseased or heathy tissue. Moreover, the pathology database may also include annotations from a trained pathologist corresponding to the reference pathology images in the pathology database (see infra <FIG>). The machine learning algorithm may use the annotations from the pathologist in conjunction with the reference pathology images to train itself to identify the one or more regions of interest in the magnified pathology images. The pathology database may include transcribed text that the pathologist spoke when the reference pathology images were recorded. For example, the pathologist recording the images in the database may have said "this portion of the sample looks normal. " The machine learning algorithm may receive this statement from the database as either natural language or transcribed text, and know that the portion of the sample the pathologist is referring to has nothing interesting in it. The machine learning algorithm may note the characteristics of the sample that make it "normal" and recognize these characteristics in future images. Alternatively, the pathologist may have made a statement about the sample being malignant and the machine learning algorithm may note the characteristics of the malignant sample, and apply this knowledge of malignant samples to future magnified pathology images.

In one embodiment, the pathology database further includes slide position information and magnification information about the reference pathology images in the pathology database. For example, when reference samples were included in the database either as still photos or videos, the location of where the microscope was focusing (slide position information) may have been recorded-this may include the relative coordinates on the sample, the position of the microscope stage, etc. Similarly, the level of magnification used to see the features in the reference pathology images may also be recorded. For example, it may be useful for the machine learning algorithm to know what magnification was used when a determination of either "benign" or "malignant" was made, since certain features may only be visible under certain magnifications. The machine learning algorithm may use the slide position information and magnification information to identify the one or more regions of interest in the magnified pathology images.

In the illustrated embodiment, system <NUM> identifies areas of the pathology sample that are deemed to be insufficiently reviewed by the user, and inform the user of one or more regions of potential interest (e.g., a region of the sample that may appear to system <NUM> as containing diseased, cancerous, or otherwise interesting tissue/fluid/etc.) in the areas that were deemed to be insufficiently reviewed by the user. If the user of system <NUM> is examining a pathology sample and inadvertently passes over a region that system <NUM> recognizes as possibly important, system <NUM> alerts the user to the presence of the one or more regions of potential interest. Alerting the user may be achieved in the ways discussed above (e.g., via audio, visual, haptic, or other methods). The machine learning algorithm is trained to identify the areas of the pathology sample that are deemed to be insufficiently reviewed by the user based on the user viewing the areas for a threshold amount of time. This involves the machine learning algorithm tracking if a sample was in the user's field of view for a threshold amount of time. For example, if the user of system <NUM> spends less than one second looking at an area in the pathology sample, system <NUM> may presume that that area of the sample was passed over.

However, system <NUM> is more refined, and uses more than just time as the sole metric of if a portion of the sample was insufficiently reviewed. This is because a trained pathologist may simply need to glance at one area of a pathology sample and know immediately it is not of interest. Accordingly, system <NUM> may determine that a user has/has not viewed specific areas by employing gaze detection (illustrated as dashed lines from the eyepiece of microscope <NUM> in <FIG>), and does determine that a user has/has not viewed specific areas by employing location of the pathology sample (e.g., stage position or relative position on the sample) and level of magnification of microscope <NUM>. Thus, if the trained pathologist misses an area in the pathology sample because he/she never even looked at it, system <NUM> may use gaze detection (e.g., pupal dilation) and alert the pathologist that he/she passed over this portion of the sample.

In the illustrated embodiment, processing apparatus <NUM> is coupled (wired or wirelessly) to network <NUM> and/or storage <NUM>. Network <NUM> and storage <NUM> may be local or remote, and may or may not be distributed. Storage <NUM> may include RAM, ROM, hard disks, flash memory, or any other suitable memory system. Network <NUM> may include the internet or local area network. In one embodiment, processing apparatus <NUM> may be a distributed system on network <NUM>. One skilled in the art will appreciate that there are any number of ways to process/store data in accordance with the teachings of the present disclosure.

In the depicted embodiment, microphone <NUM> is electrically coupled to processing apparatus <NUM> (either wired or wirelessly) to record what the pathologist says. This information may be used to update the pathology database and further teach the machine learning algorithm. Additionally, images captured by digital camera <NUM> may also be used to update the pathology database along with the microscope stage position (provided by mechanical to electrical transducer <NUM>) and level of magnification (provided by mechanical to electrical transducer <NUM>).

In some embodiments, system <NUM> may be used to help pathologists transition to a fully digital microscopy environment, and a mouse is coupled to processing apparatus <NUM> and/or microscope <NUM>. The mouse may control a motorized stage on microscope <NUM>. The pathologist could choose to either physically move the slide (by turning nobs on the microscope) or the pathologist could move the mouse, and the motorized stage would move the slide to the corresponding location. In one embodiment, the motion of the stage could be captured to further inform the machine learning algorithm (e.g., to figure out which parts of the slide have been neglected/passed over).

<FIG> illustrates a pathology database <NUM> to train a machine learning algorithm <NUM>, in accordance with an embodiment of the disclosure. Pathology database <NUM> may be created using a system like system <NUM> of <FIG>, and may be stored on storage <NUM>. As shown in the depicted embodiment, pathology database <NUM> includes pathology images (e.g., video or still frames) that were captured by a digital camera (e.g., digital camera <NUM>). The pathology images are indexed with respect to their frame number, recording time, the voice annotation of the pathologist (transcribed), microscope stage position, magnification they were collected at, and location of pathologist gaze. One skilled in the art will appreciate that system <NUM> depicted in <FIG> can be used to create a database with any number of dimensions and inputs and is not restricted to those dimensions/inputs depicted here.

As illustrated a digital camera (e.g., digital camera <NUM>) optically coupled to a microscope (e.g., microscope <NUM>) may start recording images of pathology samples as a digital video or still frames. Each frame of the video is indexed with respect to its capture time. For example in the depicted embodiment, frame one was captured during the first three microseconds of recording, frame two was captured in the fourth through seventh microseconds of recording, etc. A microphone (e.g., microphone <NUM>) may also record the voice annotation of a user of the microscope. The vocal annotations may be converted into text and/or indexed to their respective recording time and video frame. In the depicted embodiment, while frame one was captured (during the first three microseconds of recording) the pathologist said the word "this"; in subsequent frames the pathologist stated "looks like lymphatic tissue and may be benign.

The system may also record the position of the microscope stage and index it with respect to the recording time and the magnified pathology images. In the depicted embodiment, the location of the stage is measured with X, Y coordinates from a (<NUM>,<NUM>) point which is the lower left hand position of the stage, and the stage movement is measured in microns. However, in other embodiments the stage axis may be oriented differently (e.g., the (<NUM>,<NUM>) point is located at the bottom right hand position of the stage), the units of measurement may be different (e.g., mm, cm, etc.), and the Z position of the stage may also be recorded. Furthermore, "stage position" should be broadly construed because it is used to identify specific locations on samples, which one skilled in the art will appreciate may be achieved in any number of ways. In one embodiment, stage position is determined optically with respect to the dimensions of the slide being imaged, and not with respect to the microscope hardware. As shown, the magnification that a specific frame was viewed with is also recorded with respect to recording time, transcribed text, stage position, and gaze quadrant.

The user's gaze may also be indexed with respect to the other dimensions/inputs illustrated and discussed. In the depicted embodiment, the gaze of the user/pathologist is measured in quadrants; meaning the image the user sees is subdivided into four sub-images, and the system records which sub-image the user was looking at during the recording time. This may be achieved with hardware/software installed in the microscope, or other external systems, as one skilled in the art will appreciate that there are many different ways to detect gaze. Moreover, while the embodiment depicted here only illustrates very generally where the pathologist/microscope user was looking, in other embodiments the exact coordinates that the user was looking at are recorded.

In one embodiment, indexing the magnified pathology images and the voice annotation may include tagging the voice annotation of the user to a region of interest in the magnified pathology images. For instance, in the embodiment depicted above, the pathologist's diagnosis of "benign" is associated with stage position coordinates (<NUM>, <NUM>) at 40x magnification, and he/she was looking in quadrants <NUM> and <NUM>. This allows the machine learning algorithm <NUM> to know exactly where the pathologist was looking when the determination of "benign" was made. Further the machine learning algorithm <NUM> knows the history of examination (how much of the slide had been examined up to that point). In the depicted embodiment, the processing apparatus may further include logic that when executed by the processing apparatus causes the processing apparatus to convert the pathologist's voice annotation to text, and the text is indexed with respect to recording time and the magnified pathology images, among the other dimensions/inputs mentioned and discussed. In another or the same embodiment, the pathologist may be able to review the pathology images collected and directly annotate the image to show a region of interest (e.g., circle the cancer cells on the digital image, place a star next to an unknown cell formation, etc.) to make teaching machine learning algorithm <NUM> easier.

It is worth noting that more than one pathologist may look at and annotate a pathology sample. Additional database rows and/or columns may be added so that information from both pathologists is captured. Both of the pathologists' input can then be compared to generate a ground truth/argument regarding what is known about the sample/slide. Redundancy of information about a pathology sample may make the diagnosis in pathology database <NUM> more robust, and provide a larger sample size to train machine learning algorithm <NUM>.

All of the inputs discussed above may be fed into machine learning algorithm <NUM> to train machine learning algorithm <NUM> to recognize regions of interest in pathology samples. Machine learning algorithm <NUM> may be based on a neural-network type approach or other methods such as association rule learning, deep learning, inductive logic programming, decision tree learning, support vector machines, Bayesian networks, reinforcement learning, clustering, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, or the like. Additionally, machine learning algorithm <NUM> may include a number of different algorithms running in parallel or at discrete intervals.

<FIG> illustrates what a pathologist might experience using system <NUM> of <FIG>, in accordance with an embodiment of the disclosure. Image <NUM> is what a user/pathologist might see when looking through microscope <NUM>. In one embodiment image <NUM> is projected onto screen <NUM>. After inspecting the pathology sample for a few minutes, a speaker (e.g., speaker <NUM> of <FIG>) may output several statements like statement <NUM> (letting the user/pathologist know they passed over a spot on the slide that has a potential region of interest) and statement <NUM> (telling the user a diagnosis of a structure in the one or more regions of interest, and outputting a confidence interval for the diagnosis). As shown the regions of interest and potential regions of interest may be highlighted (e.g., outlined) in image <NUM>. The outline may appear on a screen or in the eyepiece of the microscope. In one embodiment, highlighting may include laser light shining directly on the sample, or any other way to alert the pathologist examining the sample.

In addition to receiving guidance from the machine learning algorithm, a pathologist may be able to start a video chat application either by interacting with the GUI or by calling for help ("Ok microscope, call a breast pathologist"). The video feed from the microscope may then be sent to a remote proctor to view. The remote proctor may be able to communicate directly with the local pathologist and system <NUM>. More proctors/specialists can be added as needed. This proctoring capability can also be used for student evaluation.

The proctoring capability may be combined with the machine learning algorithm: if the system detects a rare diagnosis, it could suggest calling an expert or even automatically do it. This capability could also be expanded to enable consensus pathology, where each slide is always reviewed by a number of pathologists simultaneously and a diagnosis is made only when a consensus has been achieved (e.g., initially three pathologists are connected, if they disagree a fourth pathologist is added etc. until the desired level of agreement is reached).

<FIG> is a flow chart illustrating a method <NUM> of pathology diagnosis, in accordance with several embodiments of the disclosure. The order in which some or all of process blocks <NUM> - <NUM> appear in method <NUM> should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of method <NUM> may be executed in a variety of orders not illustrated, or even in parallel.

Block <NUM> illustrates magnifying a pathology sample with a microscope to form magnified pathology images. This may be done manually, by adjusting nobs on a microscope, or may be performed digitally using a computer system to control the microscope. In one embodiment, the microscope could also automatically zoom/move the slide so that the most relevant part of the slide (in order to make the diagnosis) is shown to the pathologist first. This feature could be combined with a slide feeding apparatus to greatly speed up pathologist workflow.

Block <NUM> shows recording the magnified pathology images with a digital camera optically coupled to the microscope. A processing apparatus may receive magnified pathology images from the digital camera either wirelessly or by wired transmission.

Block <NUM> describes comparing the magnified pathology images to reference pathology images included in a pathology database to identify one or more regions of interest in the magnified pathology images. In one embodiment, the one or more regions of interest include at least one of diseased portions of the pathology sample, or atypical cells in the pathology sample. The processing apparatus may run a machine learning algorithm to identify the one or more regions of interest, and the machine learning algorithm may be trained to identify regions of interest using the pathology database (see supra <FIG>). As stated above, the pathology database may include annotations from a pathologist corresponding to the reference pathology images in the pathology database. The machine learning algorithm may use the annotations from the pathologist in conjunction with the reference pathology images to identify one or more regions of interest in the magnified pathology images.

In one embodiment, the pathology database further includes magnification information, and location information to identify a position on the reference pathology images. The machine learning algorithm may be trained by, and use the location information and the magnification information about, the reference pathology images to identify the one or more regions of interest in the magnified pathology images. In one embodiment, the pathology database further includes a plurality of reference pathology images of the same disease to train the machine learning algorithm (e.g., the database may have many images of melanoma).

Block <NUM> shows alerting a user of the microscope to the one or more regions of interest in the magnified pathology images. This may involve the processing apparatus outputting instructions to a microphone to output speech, highlight a portion of a screen, or moving (e.g., vibrate) a piece of equipment. In one embodiment, the machine learning algorithm outputs a diagnosis of a structure in the one or more regions of interest. In another or the same embodiment, the machine learning algorithm may output a confidence interval for the diagnosis, based on the reference pathology images in the pathology database.

A tangible non-transitory machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-readable storage medium includes recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

Claim 1:
A method, comprising:
magnifying a pathology sample with a microscope to form magnified pathology images;
recording the magnified pathology images with a digital camera optically coupled to the microscope;
comparing the magnified pathology images to reference pathology images included in a pathology database to identify one or more regions of interest in the magnified pathology images, wherein a machine learning algorithm is used to identify the one or more regions of interest, and wherein the machine learning algorithm is trained to identify the one or more regions of interest using the reference pathology images in the pathology database; and
alerting a user of the microscope to the one or more regions of interest in the magnified pathology images, and further comprising:
identifying areas of the pathology sample that are deemed to be insufficiently reviewed by the user; and
informing the user of one or more regions of potential interest in the areas that are deemed to be insufficiently reviewed by the user, wherein the machine learning algorithm is trained to identify the areas of the pathology sample that are deemed to be insufficiently reviewed by the user by tracking if an area was in the user's field of view for a threshold amount of time, and wherein the areas that are deemed to be insufficiently reviewed are further determined by employing location on the pathology sample and level of magnification of the microscope.