Patent Description:
Pathologists typically examine tissue specimens prepared on a slide using a microscope. Often, before diagnosing the tissue specimens, pathologists screen the tissue specimens to separate normal tissue from diseased tissue. Diagnosis and classification of the diseased tissue follows thereafter. The screening process can consume a significant amount of time, which delays, and can reduce the amount of time available for, the diagnosis and classification of the specimens. Human error can also impede the screening process. <CIT> relates to the segmentation and feature-based classification of images of human tissue using machine learning. <NPL>, relates to a probability-based attribution of medical images to different classes.

The various embodiments described herein generally relate to an image diagnostic system and methods of operating thereof. The disclosed methods and systems can relate to histological images and/or histopathological images.

In accordance with the invention, there is provided a system for identifying one or more tissue types within an image patch according to a hierarchical histological taxonomy. The system includes a supervised digital pathology database having a set of training image patches stored thereon; and a processor in communication with the supervised digital pathology database and operable to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy; apply the feature extractor to the image patch to identify one or more tissue segments within the image patch; and apply the feature classifier to assign a tissue type to the one or more tissue segments identified by the feature extractor for the image patch and to generate a confidence score for assigned tissue type within the image patch with reference to the class confidence score, wherein applying the feature classifier comprises mapping one or more prediction models to each tissue segment within the image patch to identify the tissue type relative to the training feature vector and class confidence score for that tissue type; compare each confidence score generated by the feature classifier with a class confidence score generated for a normal tissue of that tissue type to determine whether the image patch is associated with normal tissue; and generate a confidence score vector containing the confidence scores for each tissue type identified for a feature vector by the feature classifier and a health state of each tissue type identified.

In some embodiments, the processor is operable to: develop a convolutional neural network based on the set of training image patches.

In some embodiments, the processor is operable to define an architecture of the convolutional neural network by operating a Network Architecture Search (NAS).

In some embodiments, the feature extractor is generated by applying one of local binary patterns (LBP), pre-trained convolutional neural network, local binary patterns (LBP), and Gabor filters, wavelet filters, and image differential filters.

In some embodiments, the processor is operable to: generate a feature vector identifying the one or more tissue segments of the image patch.

In some embodiments, the processor is operable to: evaluate the confidence score generated by the feature classifier by comparing the confidence score with a confidence threshold; and generate a quality indicator indicating the confidence score satisfies the confidence threshold, and otherwise, generate the quality indicator indicating the confidence score fails to satisfy the confidence threshold.

In some embodiments, the processor is operable to construct the supervised digital pathology database by: receiving one or more image patches of normal tissue; receiving, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to the hierarchical histological taxonomy; storing each labelled image patch in the supervised digital pathology database.

In some embodiments, the processor is further operable to: select a subset of the training image patches stored in the supervised digital pathology database for verification; and receive, via a verification user interface, one or more user inputs to verify one or more labels associated with each training image patch of the subset the training image patches.

In some embodiments, the processor is operable to: retrieve the set of training image patches from the supervised digital pathology database.

In some embodiments, the one or more image patches are generated from at least one whole slide image of a tissue specimen.

In some embodiments, the processor is further operable to: identify one or more regions of interest in the image patch from which to generate one or more image patches; and identify the one or more tissue types in each image patch of the one or more image patches.

In some embodiments, the system includes an image patch database for storing the one or more image patches.

In accordance with the invention, there is provided a method for identifying one or more tissue types within an image patch according to a hierarchical histological taxonomy. The method includes operating a processor to: access a set of training image patches stored in a supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy; apply the feature extractor to the image patch to identify one or more tissue segments within the image patch; and apply the feature classifier to assign a tissue type to the one or more tissue segments identified by the feature extractor for the image patch and to generate a confidence score for assigned tissue type within the image patch with reference to the class confidence score, wherein applying the feature classifier comprises mapping one or more prediction models to each tissue segment within the image patch to identify the tissue type relative to the training feature vector and class confidence score for that tissue type; compare each confidence score generated by the feature classifier with a class confidence score generated for a normal tissue of that tissue type to determine whether the image patch is associated with normal tissue; and generate a confidence score vector containing the confidence scores for each tissue type identified for a feature vector by the feature classifier and a health state of each tissue type identified.

In some embodiments, the method includes operating the processor to: develop a convolutional neural network based on the set of training image patches.

In some embodiments, an architecture of the convolutional neural network includes three convolutional blocks, a global max pooling layer, a single fully-connected layer, and a sigmoid layer.

In some embodiments, the method includes operating the processor to define an architecture of the convolutional neural network by operating a Network Architecture Search (NAS).

In some embodiments, the method includes operating the processor to generate the feature extractor by applying one of pre-trained convolutional neural network, local binary patterns (LBP), Gabor filters, wavelet filters, and image differential filters.

In some embodiments, the method includes operating the processor to: generate a feature vector identifying the one or more tissue segments of the image patch.

In some embodiments, the confidence score vector has a size corresponding to a number of tissue types identified by the feature classifier for the image patch. The confidence score can include a continuous value.

In some embodiments, the one or more prediction models includes a model based on at least one of a fully-connected (FC) network, a support vector machine (SVM), and a statistical regression.

In some embodiments, the feature classifier is based on a random-forest technique.

In some embodiments, the method includes operating the processor to: evaluate the confidence score generated by the feature classifier by comparing the confidence score with a confidence threshold; and generate a quality indicator indicating the confidence score satisfies the confidence threshold, and otherwise, generate the quality indicator indicating the confidence score fails to satisfy the confidence threshold.

In some embodiments, the quality indicator includes a Boolean value.

In some embodiments, the method includes operating the processor to construct the supervised digital pathology database by: receiving one or more image patches of normal tissue; receiving, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to the hierarchical histological taxonomy; storing each labelled image patch in the supervised digital pathology database.

In some embodiments, the method includes operating the processor to: select a subset of the training image patches stored in the supervised digital pathology database for verification; and receive, via a verification user interface, one or more user inputs to verify one or more labels associated with each training image patch of the subset the training image patches.

In some embodiments, the method includes operating the processor to: retrieve the set of training image patches from the supervised digital pathology database.

In some embodiments, the set of training image patches includes one or more image patches showing at least one normal tissue and the at least one normal tissue is labelled with a tissue type.

In some embodiments, the at least one normal tissue is labelled according to the hierarchical histological taxonomy.

In some embodiments, the method includes operating the processor to generate the one or more image patches from at least one whole slide image of a tissue specimen.

In some embodiments, the method includes operating the processor to: identify one or more regions of interest in the image patch from which to generate one or more image patches; and identify the one or more tissue types in each image patch of the one or more image patches.

In some embodiments, the one or more image patches includes at least one image patch having an overlapping portion with a neighbouring image patch.

In some embodiments, the method includes operating the processor to store the one or more image patches in an image patch database.

In some embodiments, the hierarchical histological taxonomy includes at least two hierarchical levels.

In some embodiments, the hierarchical histological taxonomy includes a set of morphological tissue type labels and a set of functional tissue type labels.

In accordance with the invention, there is provided a non-transitory computer-readable medium having instructions executable on a processor for implementing any one of the methods disclosed herein.

Several embodiments will be described in detail with reference to the drawings, in which:.

The drawings, described below, are provided for purposes of illustration, and not of limitation, of the aspects and features of various examples of embodiments described herein. For simplicity and clarity of illustration, elements shown in the drawings have not necessarily been drawn to scale. The dimensions of some of the elements may be exaggerated relative to other elements for clarity. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements or steps.

To diagnose a tissue specimen, a glass slide is prepared with the tissue specimen for microscopy scanning. The glass slide can then be converted into a digital slide by a digital pathology scanner, such as a whole slide imaging device. The digital slide is then examined to identify relevant regions, or regions of interest (ROIs), which are then diagnosed for disease. Screening for regions of interests can be a tedious and time-consuming visual recognition task as imaged pathological slides are very large in size (for example, many can have a size greater than 100MB). For example, a pathologist diagnosing a tissue specimen for adenocarcinoma will first screen for glandular tissue regions, and review those regions that appear abnormally disordered before assigning a diagnosis. In addition, pathologists are often required to diagnose hundreds to thousands of slides each day. As a result, diagnostic accuracy can suffer when the pathologists are fatigued.

The tissue specimen can relate to any biological systems, such as animal and plants.

Automating the screening process to separate normal tissue and diseased tissue can, therefore, increase the efficiency and reduce the turnaround time for diagnosing the tissue specimens. Normal tissues refer to tissues that are diagnosed to be healthy and unaffected by a known disease. Existing screening tools are typically limited to a specific task and cannot be extended to other tasks, such as the identification of other types of diseases or other grades of cancer, since the appearance and structure of abnormalities in diseased tissues can be highly variable. For example, a tool can be limited to identifying only metastatic breast cancer from sentinel lymph node biopsies. Also, existing tools typically require considerable quantities of abnormal tissue samples in order to generate a training data set.

Existing diagnostic computational tools require pixel-level annotation of histological tissue type (HTT) when segmenting whole slide images. However, pixel-level annotation is impractical due to the significant time and effort required by pathologists. In the methods and systems described herein, glass slides are digitized and then divided into image patches, or sub-images. An image patch corresponds to a portion of an image. The size of the image portion can vary and will typically relate to a small portion of the image. The number of image patches divided from an image can vary according to the size of the image, characteristics of the image, and/or features shown in the image. In some embodiments, the dimensions of an image patch can be selected based on the resolution necessary to show sufficient tissue information to facilitate the analysis of that tissue.

The disclosed systems can then develop a feature extractor and classifier for the image patches by training a neural network on a digital pathology database. The training datasets relate to normal tissues. By operating to identify normal tissues based on their appearance and/or structure, the disclosed systems can be extended to apply to any disease type as there is no requirement for data related to the abnormal tissue samples to serve as training datasets.

The image patches can then be labelled by the feature extractor and classifier according to a hierarchical histological taxonomy. Training a neural network on image patches labelled based on labels established according to the hierarchical histological taxonomy can, in some embodiments, capture more relevant information. Certain visual features which enable the distinction between more specific tissue types may not be shown, or not clearly shown, in some images while those visual features are visible in other images. To enable clear and informative labelling, a higher-level label is helpful for the images that do not show (or do not clearly show) those visual features and more specific labels are available for images that show those features.

Based on the patch-level annotations, the disclosed systems and methods can predict tissue types at the pixel level.

The methods and systems described herein can reduce the cognitive workload required of pathologists by narrowing the visual search area by highlighting (or segmenting) regions of diagnostic relevance. Pathologists are able to then focus on diagnosing the relevant regions of interest. The applications of the methods and systems disclosed herein can be extended to, but not limited to, toxicological analysis, forensic sciences, and other related fields.

Reference is first made to <FIG>, which illustrates an example block diagram <NUM> of an image diagnostic system <NUM> in communication with a digital pathology scanner <NUM>, an external data storage <NUM>, and a computing device <NUM> via a network <NUM>. Although only one digital pathology scanner <NUM> and one computing device <NUM> are shown in <FIG>, the image diagnostic system <NUM> can be in communication with a greater number of digital pathology scanners <NUM> and/or computing devices <NUM>. The image diagnostic system <NUM> can communicate with the digital pathology scanner(s) <NUM> and computing device(s) <NUM> over a wide geographic area via the network <NUM>.

The digital pathology scanner <NUM> can include any computing device that is capable of capturing image data, generating image data, and/or storing image data. The digital pathology scanner <NUM> can scan glass slides held by racks, for example.

The digital pathology scanner <NUM> can include a computing device <NUM>, in some embodiments. For example, the digital pathology scanner <NUM> can include a whole slide imaging scanner that digitizes glass slides. Whole slide imaging scanners are often used in digital pathology to digitize glass slides by scanning through the regions of interest with high magnification optics. Depending on the intended purpose of the digital glass slides, the whole slide imaging scanner can be operated using different magnification optics. The digital images obtained from whole slide imaging scanners are stored at high resolution. The digital images can be accessed by users, via a computer display, to interact (e.g., visualize, navigate, focus, mark, and classify, etc.) with the regions of interest within the digital slide.

The image diagnostic system <NUM> includes a processor <NUM>, a data storage <NUM>, and a communication component <NUM>. The image diagnostic system <NUM> can be implemented with more than one computer server distributed over a wide geographic area and connected via the network <NUM>. The processor <NUM>, the data storage <NUM> and the communication component <NUM> may be combined into a fewer number of components or may be separated into further components.

The processor <NUM> can be implemented with any suitable processor, controller, digital signal processor, graphics processing unit, application specific integrated circuits (ASICs), and/or field programmable gate arrays (FPGAs) that can provide sufficient processing power for the configuration, purposes and requirements of the image diagnostic system <NUM>. The processor <NUM> can include more than one processor with each processor being configured to perform different dedicated tasks.

The communication component <NUM> can include any interface that enables the image diagnostic system <NUM> to communicate with various devices and other systems. For example, the communication component <NUM> can receive an image generated by the digital pathology scanner and store the image in the data storage <NUM> or external data storage <NUM>. The processor <NUM> can then process the image according to the methods described herein. In some embodiments, the image diagnostic system <NUM> can receive data directly from the digital pathology scanner <NUM>. For example, the image diagnostic system <NUM> can receive images directly from the digital pathology scanner <NUM> as the image is being generated. This can further reduce delays in analyzing the images.

The communication component <NUM> can include at least one of a serial port, a parallel port or a USB port, in some embodiments. The communication component <NUM> may also include an interface to component via one or more of an Internet, Local Area Network (LAN), Ethernet, Firewire, modem, fiber, or digital subscriber line connection. Various combinations of these elements may be incorporated within the communication component <NUM>. For example, the communication component <NUM> may receive input from various input devices, such as a mouse, a keyboard, a touch screen, a thumbwheel, a track-pad, a track-ball, a card-reader, voice recognition software and the like depending on the requirements and implementation of the image diagnostic system <NUM>.

The data storage <NUM> can include RAM, ROM, one or more hard drives, one or more flash drives or some other suitable data storage elements such as disk drives. The data storage <NUM> can include one or more databases for storing data related to the hierarchical representation of tissue types, and/or image data related to normal tissues and/or non-disease artifacts.

In some embodiments, the data storage <NUM> can be used to store an operating system and programs. For instance, the operating system provides various basic operational processes for the processor <NUM>. The programs include various user programs so that a user can interact with the processor <NUM> to perform various functions such as, but not limited to, viewing and/or manipulating the images as well as retrieving and/or transmitting image data.

The data storage <NUM> can store images, including versions of the images at different resolutions, information about the glass slide from which the image was generated, classification information related to the images, information related to reports associated with the images, for example, diagnoses with respect to the image data, images of normal tissues, and images of tissues containing non-disease artifacts. Non-disease artifacts can be introduced during the preparation of the slide and can include, but is not limited to, cross-contamination, air bubbles, dust specks, folded tissue, crushed tissue, torn tissue, cut tissue, cracked tissue, tissue fixation, tissue thickness, and other defects.

The data storage <NUM> can store information related to image labels, such as but not limited to, text comments, audio recordings, markers, shapes, lines, free form mark-ups, and measurements.

The external data storage <NUM> can store data similar to that of the data storage <NUM>. The external data storage <NUM> can, in some embodiments, be used to store data that is less frequently used and/or older data. In some embodiments, the external data storage <NUM> can be a third party data storage stored with image data for analysis by the image diagnostic system <NUM>. The data stored in the external data storage <NUM> can be retrieved by the computing device <NUM> and/or the image diagnostic system <NUM> via the network <NUM>.

Images described herein can include any digital image of any reasonable size and resolution for histological and pathological applications. In some embodiments, the image diagnostic system <NUM> can apply image pre-processing to the images, such as but not limited to normalizing the pixel dimensions of an image and/or digital filtering for noise reduction, and storing the pre-processed image as a version of the original image. Example images can include a histological image of a tissue, or part of a tissue, and a histopathological image of a tissue, or a part of a tissue. Histological images relate to normal tissues, whereas histopathological images relate to abnormal and normal tissue within the same tissue section. The images can be generated using a microscope. Each image can have an intensity value associated with each pixel.

The computing device <NUM> can include any device capable of communicating with other devices through a network such as the network <NUM>. A network device can couple to the network <NUM> through a wired or wireless connection. The computing device <NUM> can include a processor and memory, and may be an electronic tablet device, a personal computer, workstation, server, portable computer, mobile device, personal digital assistant, laptop, smart phone, WAP phone, an interactive television, video display terminals, gaming consoles, and portable electronic devices or any combination of these.

The network <NUM> can include any network capable of carrying data, including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these, capable of interfacing with, and enabling communication between, the image diagnostic system <NUM>, the digital pathology scanner <NUM>, the external data storage <NUM>, and the computing device <NUM>.

Reference will now be made to <FIG>, which shows a flowchart 120A illustrating an example method of operating the image diagnostic system <NUM> for screening one or more glass slides.

At <NUM>, the image diagnostic system <NUM> operates to obtain a digital image of each glass slide. Glass slides can be stored on a slide rack. The image diagnostic system <NUM> can operate the digital pathology scanner <NUM> (such as a whole slide imaging device), to generate the digital images of the glass slides.

The digital pathology scanner <NUM> can be operated to generate the digital slides at low-resolution, such as at a magnification of <NUM> times (20X) or less. The image diagnostic system <NUM> can store the low-resolution digital image obtained at <NUM> into a low-resolution database, such as database <NUM> ("WSI <NUM>"). The database <NUM> can be provided with data storage <NUM>, <NUM>, for example.

At <NUM>, the image diagnostic system <NUM> then applies the methods disclosed herein to determine whether the digital image generated at <NUM> is associated with normal tissue or non-disease artifacts. As will be described, the methods applied by the image diagnostic system <NUM> can refer to images stored in databases <NUM> ("WSI <NUM>") and <NUM> ("WSI <NUM>"). For example, database <NUM> can store images related to normal tissues and database <NUM> can store images related to non-disease artifacts. The databases <NUM>, <NUM> can be provided with data storage <NUM>, <NUM>, for example.

At <NUM>, the image diagnostic system <NUM> can classify each digital image accordingly. For digital images classified as relating to normal tissue, the image diagnostic system <NUM> can classify the digital image as normal (at <NUM>) and store the relevant information at database <NUM> in association with the digital image. Similarly, for digital images classified as relating to non-disease artifacts, the image diagnostic system <NUM> can classify the digital image as non-disease artifact (at <NUM>) and store the relevant information at database <NUM> in association with the digital image.

At <NUM>, the image diagnostic system <NUM> can classify the remaining digital images to relate to abnormal tissue (at <NUM>). In some embodiments, the image diagnostic system <NUM> can also conduct the methods disclosed herein to identify one or more pathological case with the abnormal tissue within the digital image.

At <NUM>, the image diagnostic system <NUM> can operate the digital pathology scanner <NUM> to generate a high-resolution digital image of the glass slides corresponding to the digital images associated with abnormal tissue. The high-resolution digital image can be at a magnification of <NUM> times or greater, for example. The image diagnostic system <NUM> can use a different digital pathology scanner <NUM> than at <NUM>, in some embodiments.

The image diagnostic system <NUM> can then store the high-resolution digital images in a database <NUM> ("WSI <NUM>"). The database <NUM> can be provided with data storage <NUM>, <NUM>, for example.

In some embodiments, databases <NUM>, <NUM>, <NUM>, <NUM> can be provided together in one or more data storages or separately in different data storages. One or more of databases <NUM>, <NUM>, <NUM>, <NUM> can be provided together, while the others provided separately.

At <NUM>, the image diagnostic system <NUM> can indicate that the high-resolution digital images are available for review by a pathologist, for example. In some embodiments, the image diagnostic system <NUM> can generate a notification indicating the high-resolution digital images are ready for review.

Reference is now made to <FIG>, which is a flowchart 120B illustrating another example method of operating the image diagnostic system <NUM> for screening one or more glass slide. The method shown in <FIG> is generally similar to that shown in <FIG>, except that, at <NUM> and <NUM>, when the image diagnostic system <NUM> operates to obtain digital images at low-resolution and high-resolution, the image diagnostic system <NUM> can operate the same digital pathology scanner <NUM>.

As will be described, the image diagnostic system <NUM> can be operated to develop a neural network on a digital pathology database. The image diagnostic system <NUM> can receive image patches of normal tissues that are labelled according to an established histological taxonomy and store the labelled image patches into the digital pathology database as the training dataset. The digital pathology database can store labelled images and/or labelled image patches related to normal tissues, abnormal tissues and/or artifacts. This digital pathology database can be referred to as the supervised digital pathology database and can be provided in the data storage <NUM> and/or external data storage <NUM>. In some embodiments, as shown generally in <FIG>, the image diagnostic system <NUM> can operate to receive whole slide images of normal tissues (at <NUM>), divide the digitized slides of the whole slide images into image patches (at <NUM>) and provide a user interface from which to receive user inputs labelling each image patch according to the established histological taxonomy (at <NUM>).

The histological taxonomy can follow any histological classification system as long as the classification system is consistently applied throughout the digital pathology database. Generally, histological taxonomy can be developed by referring to existing histology documents that are used as educational materials for students in histology, knowledge of pathologists (e.g., via consultation with expert pathologists), and example tissue specimens. However, to extend the application of the image diagnostic system <NUM> beyond specific disease types, the labels and image data available via the digital pathology database is important. For example, some digital pathology databases include images that are labelled at the glass slide level and therefore, include minimal localized information. Digital pathology slide images are typically very large and by labeling at the glass slide level, the labels can be incomplete and exclude essential regions of interest. Some digital pathology databases include histopathological images that are specially labelled for certain diseases. As a result, the image diagnostic system <NUM> cannot use those labelled histopathological images to develop the neural network to be generalized to classify unexpected diseases or unseen organs.

The operation of the image diagnostic system <NUM> can benefit from a digital pathology database with image patches that are labelled according to a histological taxonomy system that involves a large range of tissue types. As will be described with reference to <FIG> and <FIG>, with the current microscopic technologies, there are only a limited number of tissue types that are observable. Further tissue types may be observable with evolving microscopic technologies. The scope of the image diagnostic system <NUM> is not limited to the existing observable tissue types. The example hierarchical histological taxonomy <NUM> illustrated in <FIG> and <FIG> can be adapted with additional knowledge from pathologists and data resulting from the use of new technologies. By operating the image diagnostic system <NUM> to develop the neural network on image patches labelled with a large range of tissue types, the image diagnostic system <NUM> can generate valuable results to pathologists that, in some embodiments, can be beyond the capabilities of pathologists achievable via the manual screening process.

Also, as described herein, operating the image diagnostic system <NUM> to develop the neural network on image patches labelled according to a hierarchical histological taxonomy can capture more relevant information. Certain visual features which enable the distinction between more specific tissue types may not be shown, or not clearly shown, in some images while those visual features are visible in other images. To enable clear and informative labelling, a higher-level label is helpful for the images that do not show (or do not clearly show) those visual features and more specific labels are available for images that show those features.

<FIG> illustrates at <NUM> an example hierarchical histological taxonomy table, and <FIG> illustrates a portion <NUM> of the hierarchical histological taxonomy table <NUM> embedded with example histological images for illustrative purposes.

In general, tissue types can be characterized with specific visual patterns and relative spatial relationships. For example, epithelial tissues can be characterized by linearly-arranged nuclei-dense cells lining a surface, connective proper tissues can be characterized by long fibrous cells with scant nuclei between other tissues, blood tissues can be characterized by circular or ovoid blobs which sometimes clump together and are often inside transport vessels, and skeletal tissues can be characterized by a woody material which sometimes contains layered rings and sometimes appears mottled with lighter regions, adipose tissues can be characterized by clustered rings or bubbles, muscular tissues can be characterized by parallel (e.g., longitudinal cut) or bundled (e.g., transverse cut) dense fibrous cells with oblong nuclei, nervous tissues can be characterized by wispy strands connecting branching circular blobs, glandular tissues can be characterized by epithelium-lined ovoid structures with or without inner openings, and transport vessel tissues can be characterized by string-like rings often containing blood.

In the example hierarchical histological taxonomy <NUM>, the tissue types are assigned into different types of varying specificity based on aspects related to their morphology and functionality, and how they relate to each other. Morphology relates to a tissue structure, whereas functionality refers to tissue functionality and relation to organs. Classification of tissues based on morphology is often used because even a small visual field is sufficient to identify the tissue structure, whereas classification by functionality is not as often used when labelling image patches since the organ of origin of the tissue specimen is usually unknown and a larger visual field is needed to provide the spatial context for understanding the tissue functionality.

In the example hierarchical histological taxonomy <NUM>, tissue types of differing specificity are organized vertically in a parent-child relationship. Tissue types with the same parent node and specificity are organized in a sibling relationship. For example, simple epithelial and stratified epithelial tissues are respective child nodes of the parent node, epithelial tissue. The hierarchical histological taxonomy <NUM> continues to classify the tissue types until leaf nodes are reached.

In some embodiments, the hierarchical histological taxonomy <NUM> can be encoded with a hierarchical encoding system so that each node is associated with a nodal code. For example, a nodal code can be generated based on a concatenation of its ancestor nodes' symbols (e.g. the stratified cuboidal epithelial node is encoded as "E. ", as its ancestors are symbolized by "E. " (stratified epithelial) and "E" (epithelial). When a label for an image patch cannot be assigned with a leaf node, a placeholder code can applied. For example, when a stratified epithelial tissue cannot be assigned a nodal code corresponding to a leaf node, the stratified epithelial undifferentiated leaf node, represented by the nodal code "E. X", can be used. Other manners of labelling undifferentiated tissue types can be applied.

The example hierarchical histological taxonomy <NUM> is organized into three hierarchical levels, a first hierarchical level <NUM>, a second hierarchical level <NUM> and a third hierarchical level <NUM>. The first hierarchical level <NUM> includes nine parent nodes corresponding to epithelial tissues, connective proper tissues, blood tissues, skeletal tissues, adipose tissues, muscular tissues, nervous tissues, glandular tissues, and transport vessel tissues.

The second hierarchical level <NUM> builds from the first hierarchical level <NUM>. For example, epithelial tissue is a parent node within the first hierarchical level <NUM> and is associated with child nodes associated with simple epithelial tissue and stratified epithelial tissue, and pseudostratified epithelial tissue in the second hierarchical level <NUM>.

The third hierarchical level <NUM> builds from the second hierarchical level <NUM>. For example, simple epithelial tissue is a child node within the second hierarchical level <NUM> and is associated with grandchild nodes associated with simple squamous epithelial tissue, simple cuboidal epithelial tissue, and simple columnar epithelial tissue in the third hierarchical level <NUM>.

As shown in <FIG> and <FIG>, it is impossible for a child node in the second hierarchical level <NUM> to be a leaf node (e.g., pseudostratified epithelial tissue). It is also possible for a parent node to be a leaf node (e.g., transport vessel tissue).

Continuing with reference to <FIG>, at <NUM>, the image diagnostic system <NUM> can receive, via a user interface, the user inputs labelling each image patch according to the example hierarchical histological taxonomy <NUM>. The user inputs can be received from one or more histology labeling experts who are trained to recognize tissues according to the hierarchical histological taxonomy <NUM>. The user interface can also receive multiple labels for each image patch and different labels from each user. Digital pathology slides contain visibly recognizable tissues and so, the image diagnostic system <NUM> can receive user inputs related to one or more histological tissue types.

After the image diagnostic system <NUM> receives the user inputs, the image diagnostic system <NUM> can store (<NUM>) the labelled image patches in the supervised digital pathology database in the data storage <NUM> and/or <NUM>. As quality control, the labelled image patches can be reviewed by experienced pathologists at <NUM>. The experienced pathologists can be board-certified and are usually very expensive. For the purpose of quality control, the image diagnostic system <NUM> can select a subset of labelled image patches for the experienced pathologists to verify.

With reference to the supervised digital pathology database in the data storage <NUM> and/or <NUM>, the image diagnostic system <NUM> can proceed with analyzing new whole slide images. As will be described with reference to <FIG>, the image diagnostic system <NUM> can, at least, identify tissue types within image patches and also outline individual tissue segments for tissue type quantification (e.g. measuring lymphocyte density or epithelium thickness); identify abnormal tissue regions (e.g. cancerous tumors, slide imperfections, staining debris, and out-of-focality) from class confidence scores generated by the neural network when identifying the tissue types; and encoding tissue types to facilitate feature searching within the supervised digital pathology database <NUM>, <NUM> (e.g. text-based image retrieval).

Reference will now be made to <FIG>, which shows a flowchart <NUM> of an example method of operating the image diagnostic system <NUM> to automatically identify one or more tissue types within an image patch of a whole slide image. <FIG> shows an example whole slide image <NUM>, <FIG> shows an example region of interest <NUM> of the whole slide image <NUM>, and <FIG> shows example image patches <NUM> selected from the region of interest <NUM> of the whole slide image <NUM>. As shown in <FIG>, the region of interest <NUM> can include multiple overlapping image patches <NUM>.

In <FIG>, an example training and validation phase is shown at <NUM> and an example classification phase is shown at <NUM>.

During the training and validation phase <NUM>, the image diagnostic system <NUM> accesses (<NUM>) the labelled image patches from the supervised digital pathology database <NUM>, <NUM>. The image diagnostic system <NUM> can retrieve the labelled image patches for the training and validation phase <NUM>, in some embodiments. The labelled image patches in the supervised digital pathology database <NUM>, <NUM> can be labelled according to the hierarchical histological taxonomy table <NUM> of <FIG>. The labelled image patches stored in the supervised digital pathology database <NUM>, <NUM> can serve as the training image patches during the training and validation phase <NUM>. The image diagnostic system <NUM> can develop a feature extractor with the training image patches and update the parameters of the feature extractor according to the hierarchical labels assigned to the training image patches.

At <NUM>, the image diagnostic system <NUM> operates to develop the feature extractor using the training image patches. The feature extractor will then be applied by the image diagnostic system <NUM> to other image patches to identify the tissue types shown therein.

The feature extractor can be trainable or non-trainable. A trainable feature extractor can be developed based on data-driven methods, such as the layers of a convolutional neural network (CNN) before the fully-connected layers (e.g., the convolution and pooling layers). A non-trainable feature extractor can include handcrafted methods, such as, but not limited to, local binary patterns (LBP), pre-trained convolutional neural network via transferable learning, wavelet filters, image differential filters and Gabor filters.

Convolutional neural networks generally include convolutional and pooling layers to downsample the training dataset, followed by fully connected layers that generate the desired predictions. The convolution operations involve element-wise matrix multiplication between filter values and the pixels in the image, and the resultant values are then summed. During the pooling layer, information collected by the convolution layer is simplified and condensed. The pooling operation can involve reducing the network parameters (e.g., size of a filter window, number of filters, a stride and padding), which can then reduce the required computation. Convolutional neural networks are deep, feed forward neural networks as they are trained through iterative passes at the training dataset. With each iteration, a further layer of the convolutional neural network is able to identify more complex features. Different architectures of the convolutional neural networks can be applied herein, such as, but are not limited to, Inception-V3, ResNet18, VGG16, or the architectures disclosed herein.

As described herein, the image diagnostic system <NUM> can develop a feature extractor based on a convolutional neural network that is trained on the digital pathology database <NUM>, <NUM>. This feature extractor can operate according to the hierarchical histological taxonomy associated with the digital pathology database <NUM>, <NUM>. The architecture of the convolutional neural network disclosed herein can be adapted for processing image patches at a suitable scan resolution and suitable patch size to maintain low computational complexity and high performance accuracy for image-feature transformation, development of data-augmentation methods to compensate for different shortcomings (such as, but not limited to, random effects of rotation, shifting, flipping, and cutting sections of tissue specimen during slide preparation in pathology laboratories, for example), image correction and enhancement techniques for data pre-processing (such as, but not limited to, optical deblurring and color enhancement, for example), and automating the engineering of the convolutional neural network architecture using adaptive techniques of Neural Architecture Search (NAS) methods (such as, but not limited to, searching space, searching strategy, and performance estimation strategy).

The different architectures of the convolutional neural networks can perform differently. For example, the VGG16 architecture can, in some embodiments, perform better than the Inception-V3 and ResNet18 architectures due to the reduced number of layers (i.e., there are <NUM> layers in VGG16 whereas there are <NUM> layers in Inception-v3 and <NUM> layers in ResNet18). It should also be noted that the Inception-V3 architecture is often characterized with a high prediction accuracy and low model complexity, compared to the VGG16 and ResNet18 architectures. The convolutional neural network architectures disclosed herein, in comparison with existing architectures, has a less complex architectural design that can obtain high accuracy predictions and high computational speed.

In some embodiments, the feature extractor can have an architecture consisting of three convolutional blocks, followed by a global max pooling layer, a single fully-connected layer, and a sigmoid layer. Each convolutional block consists of a single convolutional layer, a ReLU activation layer, and a batch normalization layer. Unlike the VGG16 architecture, the sigmoid layer replaces the softmax layer to facilitate multi-label prediction, batch normalization is added after each convolutional layer activation, and a global max pooling layer (which reduces overfitting) replaces the flattening layer since tissues are labeled regardless of their spatial extent. Also, unlike the VGG16 architecture, dropout is used between normalization and convolutional blocks. The batch normalization and dropout are added to regularize the neural network. In contrast to the VGG16 architecture, the architecture of the feature extractor disclosed herein do not include the last two convolutional layers and the two fully connected layers. This difference from the VGG16 architecture can improve classification performance, reduce training time, and increase segmentation resolution.

The feature extractor can generate a feature vector <NUM> that represents the tissue segments identified in the training image patch at an abstract level.

At <NUM>, the image diagnostic system <NUM> operates to develop a feature classifier using the training image patches. The feature classifier receives the feature vector <NUM> from the feature extractor to predict tissue types for the tissue segments identified in the feature vector <NUM>.

The feature classifier can be trainable. For example, the feature classifier can proceed to fit one or more prediction models to establish a mapping between the feature vector <NUM> and the class confidence scores for the various tissue types available from the training image patches. Example prediction models can include but are not limited to fully-connected (FC) network, support vector machine (SVM), and Statistical Regression.

In some embodiments, the image diagnostic system <NUM> can develop a non-trainable feature classifier, such as one based on a random-forest technique.

The feature classifier can also generate a class confidence score for each tissue type identified. The confidence score can be continuous-valued. The feature classifier can generate (<NUM>) a confidence score vector of a size corresponding to the number of tissue types identified for the feature vector <NUM>.

In some embodiments, the feature classifier can also apply a corresponding confidence threshold to each confidence score to verify the tissue type. For example, when the confidence score associated with the tissue type exceeds or equals a corresponding confidence threshold, the feature classifier can verify the tissue type identified. However, when the feature classifier determines that the confidence score is below the confidence threshold, the feature classifier can indicate that the identified tissue type is incorrect. The verification result can be boolean-valued.

To initiate the classification phase <NUM>, the image diagnostic system <NUM> receives (<NUM>) a whole slide image, such as <NUM>, from the digital pathology scanner <NUM>, or from data storage <NUM>, <NUM> or the computing device <NUM>. After receiving the whole slide image <NUM>, the image diagnostic system <NUM> can then divide (<NUM>) the whole slide image <NUM> into one or more image patches <NUM>, and store (<NUM>) the image patches <NUM> into a new image patch database, which can be provided in the data storage <NUM> and/or the external data storage <NUM>.

In the example shown in <FIG>, the image diagnostic system <NUM> divides the whole slide image <NUM> into nine image patches 502a to 502i. The number of image patches and the region of interest from which the image patches are selected can vary. The image patches 502a to 502i are overlapping in this example embodiment, which can, in some embodiments, improve the quality of the analysis to be conducted by the image diagnostic system <NUM>. The extent of the overlap between each image patch can vary with various factors, such as, characteristics of the whole slide image <NUM> and the region of interest <NUM>, for example. In some embodiments, the dimension of each image patch <NUM> can be defined by a number of pixels. The dimension of the image patches <NUM> can be varied with the applications of the image diagnostic system <NUM>, according to user definitions and/or other factors associated with the use of the image patches <NUM>.

Continuing with reference to <FIG>, the image diagnostic system <NUM> operates at <NUM> to apply the feature extractor developed during the training and validation phase <NUM> to each of the image patches 502a to 502i. Similar to the process described with respect to <NUM>, the feature extractor then generates a feature vector <NUM> for each of the image patches 502a to 502i. At <NUM>, the image diagnostic system <NUM> operates to apply the feature classifier developed during the training and validation phase <NUM> to each feature vector <NUM> generated by the feature extractor at <NUM>. Similar to the process described with respect to <NUM>, the feature classifier assigns a tissue type to each tissue segment identified in the feature vector <NUM> and generates (<NUM>) a confidence score for each tissue type identified. The feature classifier can also apply a corresponding confidence threshold to each confidence score to verify the identified tissue type, in some embodiments.

The classification phase <NUM> can be used as a testing phase by the image diagnostic system <NUM> to verify the performance of the feature extractor and feature classifier developed at <NUM> and <NUM>, respectively. For example, the image diagnostic system <NUM> can use the whole slide image <NUM> received at <NUM> as part of a testing dataset.

Referring now to <FIG>, which shows a flowchart <NUM> of an example method of operating the image diagnostic system <NUM> to identify normal image patches in order to separate normal image patches from abnormal image patches.

The abnormal image patches can include any image patch containing diseased tissues or artifacts (e.g., slide preparation artifacts, such as cross-contamination, an air bubble, and/or a dust speck, and/or tissue-related artifacts, such as a folded tissue, a crushed tissue, a torn tissue, a cut tissue, a cracked tissue, a tissue fixation artifact, and/or a tissue thickness artifact).

Most tissue specimens that a pathologist encounters will likely be healthy specimens. By operating the image diagnostic system <NUM> to screen for normal image patches and to exclude normal image patches from requiring further review by pathologists, the overall diagnostic process can be reduced significantly. As a result, pathologists can devote their time to diagnosing digital slides containing diseased tissues. Also, the appearances of diseased tissues are highly variable and can be difficult to generalize into certain appearances and structure - even with massive amounts of training dataset related to each type of diseased tissues. For example, the appearance and structure of cancerous tissues are highly variable according to their types and grades.

As shown in <FIG>, the image diagnostic system <NUM> can apply the classification method described with reference to <FIG> and evaluate (<NUM>) the sample confidence score <NUM> generated by the feature classifier <NUM> for the image region <NUM>, with reference to the class confidence score <NUM> generated by the feature classifier <NUM> for normal image patches to determine whether the identified tissue segment is normal (<NUM>). The class confidence score <NUM> can have a range, in some embodiments.

The sample confidence score <NUM> generated for the image patch <NUM> can represent a visual similarity between the sample tissue and the normal tissue. When the image diagnostic system <NUM> determines that the sample confidence score <NUM> is within an acceptable range of the class confidence score <NUM>, the image diagnostic system <NUM> can determine that the sample tissue is normal. However, when the image diagnostic system <NUM> determines that the sample confidence score <NUM> is not within the acceptable range of the class confidence score <NUM>, the image diagnostic system <NUM> can determine that the sample tissue is abnormal and generate a notification requiring further review of the sample tissue by a pathologist. In some embodiments, the image diagnostic system <NUM> can determine the sample tissue is normal only when the sample confidence score <NUM> equals or is greater than the class confidence score <NUM>.

The image diagnostic system <NUM> can also determine whether the whole slide image <NUM> received at <NUM> relates to normal tissues. Following <NUM>, the image diagnostic system <NUM> can group image patches <NUM> with a high confidence score together and image patches <NUM> with low confidence scores together. The values of the high confidence scores are compared to the class confidence scores <NUM> generated by the feature classifier at <NUM>. For example, the high confidence scores can include confidence scores above a confidence threshold and those high confidence scores are averaged and compared with an average of the class confidence scores <NUM>. When the average of the high confidence scores equal or exceed the average of the class confidence scores <NUM>, the image diagnostic system <NUM> can determine that the whole slide image <NUM> relates to normal tissues. However, when the average of the high confidence scores is below the average of the class confidence scores <NUM>, the image diagnostic system <NUM> can determine that the whole slide image <NUM> relates to abnormal tissues and requires further diagnosis.

<FIG> shows a flowchart <NUM> of an example method of operating the image diagnostic system <NUM> to predict pixel-level annotations for the whole-slide images.

With histopathological images, the process of semantic segmentation aims to label each pixel with diagnoses (e.g. cancer or non-cancer), or tissue or cell types (e.g. gland, nuclei, mitotic or non-mitotic). Semantic segmentation approaches for histopathology can be grouped into sliding patch approach, super-pixel approach, pixel-level approach, and weakly-supervised approach.

Sliding patch-based approaches are trained with training datasets and predict at the center pixel of a sliding patch to obtain finer predictions. Convolutional neural networks can be applied in sliding patch-based approaches, and commonly applied for mitosis, cellular, neuronal, and gland segmentation. Sliding patch-based approaches is a type of discriminative localization-based method that uses image-level annotations to generate discriminative object localizations as initial seeds (usually using a convolutional neural network and class activation mapping) and then improve these iteratively.

Super-pixel-based approaches are trained with training datasets and predict latent labels at the super-pixel level. Super-pixel-based approaches are a type of graphical model-based method, which involves extracting regions of homogeneous appearance and predicting latent variable labels from the image level for each region. Convolutional neural networks are typically applied in super-pixel-based approaches to scaled super-pixels for tissue type and nuclear segmentation.

Pixel-based approaches are trained with training datasets and predict at the pixel level and typically apply a fully convolutional network (FCN) with contour separation processing. Fully convolutional network methods can include multi-instance learning-fully convolutional network and Bayesian Fusion of Back Projected Probabilities (BFBP), for example. Pixel-based approaches are example self-supervised based methods, which involve generating interim segmentation masks from image-level annotations and learning pixel-level segmentations from them. Some self-supervised-based methods can iterate between fine and coarse predictions. Other methods produce class activation mappings or saliency maps as initial seeds and refine them to train a fully convolutional network.

Weakly-supervised approaches are trained at the image level and predict at the pixel level. Weakly-supervised approaches tend to involve a patch-based multi-instance learning (MIL) approach. Multi-instance learning based methods typically constrain their optimization to assign at least one pixel per image to each image label. Multi-instance learning based methods can include semantic texton forest (STF), and convolutional neural networks methods like multi-instance learning-inductive logic programming (MIL-ILP) and spatial transformer networks (SPN).

Fully supervised approaches to semantic segmentation, such as sliding patch-based approaches, super-pixel-based approaches and pixel-based approaches, can provide very accurate results because they are trained at the pixel level. However, fully supervised approaches are time-consuming and computationally expensive for labelling digital pathology images at the pixel-level. In contrast, weakly-supervised approaches can train at the patch-level but often provide less accurate results than fully-supervised approaches.

As will be described with reference to <FIG>, the image diagnostic system <NUM> can apply a weakly-supervised approach to semantic segmentation and predict pixel-level labels from the image-level annotations. Reference will be made also to <FIG>, which illustrates generally at <NUM> the image diagnostic system <NUM> applying the method <NUM> to an example image patch 502e.

At <NUM>, the image diagnostic system <NUM> receives the image patch 502e. The image diagnostic system <NUM> then applies the patch-level classification method described with reference to <FIG> at <NUM> to the image patch 502e. The image diagnostic system <NUM> can generate a feature vector <NUM> identifying tissue segments within the image patch 502e and tissue types with associated confidence scores for the feature vector <NUM>.

For example, as shown in <FIG>, example confidence scores <NUM> corresponding to identified tissue types within the image patch 502e are shown. In this example, for purpose of illustration, the confidence scores for morphological tissue <NUM> is shown separately from the confidence score for functional tissue 662f.

In respect of the image patch 502e, the image diagnostic system <NUM> identified four morphological tissues, namely lymphocytes tissue (H. ), loose connective tissue (C. ), simple columnar epithelial tissue (E. ), and smooth muscle tissue (M. ), and one functional tissue, namely exocrine gland tissue (G. The image diagnostic system <NUM> also generated a corresponding confidence score for each of the identified tissue, as generally illustrated at <NUM> and 662f. The image diagnostic system <NUM> assigned a confidence score of <NUM> for lymphocytes tissue (H. ), a confidence score of <NUM> for loose connective tissue (C. ), a confidence score of <NUM> for simple columnar epithelial tissue (E. ), a confidence score of <NUM> for smooth muscle tissue (M. ), and a confidence score of <NUM> for exocrine gland tissue (G.

At <NUM>, the image diagnostic system <NUM> generates a class activation map (CAM) based on the feature vector <NUM> and confidence score vector <NUM> generated at <NUM>. The class activation map can provide a visual representation of the confidence scores generated by the feature classifier without requiring the tissue type labels. Instead, the tissue types can be identified with different visual representations, for example, such as different colours or shading. <FIG> illustrate example class activation maps for a digital slide.

<FIG> shows an example class activation map <NUM> for a digital slide. The example class activation map <NUM> illustrates the tissue types assigned a confidence score equal to or greater than <NUM>%. Other confidence thresholds can be applied, depending on the application of the image diagnostic system <NUM>. The areas in light grey correspond to white adipose tissues, while the areas in dark grey correspond to non-descriptive tissues. The area referenced generally at <NUM> corresponds mostly to white adipose tissues (mostly light grey), whereas the area referenced generally at <NUM> corresponds mostly to non-descriptive tissues (mostly dark grey).

<FIG> shows another example class activation map <NUM> for the same digital slide represented in <FIG>. Similar to class activation map <NUM>, class activation map <NUM> illustrates the tissue types assigned a confidence score equal to or greater than <NUM>%. The areas in light grey correspond to exocrine gland tissues, while the areas in dark grey correspond to non-descriptive tissues. The area referenced generally at <NUM> corresponds mostly to exocrine glad tissues (mostly light grey), and the area referenced generally at <NUM> corresponds mostly to non-descriptive tissues (mostly dark grey).

The class activation map can, in some embodiments, include a pixel-level class activation map that can indicate, at the pixel level, the section of the image patch 502e that contributed to the respective confidence score <NUM>. The pixel-level interpretation of the confidence scores, in contrast to labels, can be more helpful for visualizing the tissues.

The image diagnostic system <NUM> can apply a gradient-class activation map (Grad-CAM) method, which is a weakly-supervised semantic segmentation (WSSS) method that generalizes the class activation map method. In some embodiments, the gradient-class activation map method can be a gradient-weighted class activation map method. The gradient-class activation map method, in comparison to similar weakly-supervised semantic segmentation methods, is simpler as no re-training is required and more versatile as it can be applied to any convolutional neural network architecture.

In some embodiments, the image diagnostic system <NUM> can vary the method with which the pixel-level class activation maps are generated for different tissue types. For example, the image diagnostic system <NUM> can vary the gradient-class activation maps method to account for different characteristics of morphological tissues and functional tissues.

For example, referring again to <FIG>, example pixel-level class activation maps are illustrated generally at <NUM>. The image diagnostic system <NUM> generates pixel-level class activation map 664a for exocrine gland tissue (G. ), pixel-level class activation map 664b for lymphocytes tissue (H. ), pixel-level class activation map 664c for loose connective tissue (C. ), pixel-level class activation map 664d for simple columnar epithelial tissue (E. ), and pixel-level class activation map 664e for smooth muscle tissue (M. Example scaled pixel-level class activation maps 664a' to 664e' are shown in <FIG> as well. In some embodiments, the image diagnostic system <NUM> can scale the class activation maps <NUM> to further distinguish the confident scores of interest.

For illustrative purposes, an area generally illustrated at <NUM> in the pixel-level class activation map 664c can be seen to have contributed to the confidence score generated by the feature classifier for loose connective tissue (C. ) within the image patch 502e. Area <NUM> is illustrated only as an example and as can be seen from the pixel-level class activation map 664c, other areas not specifically identified herein also contributed to the confidence score generated by the feature classifier.

At <NUM>, the image diagnostic system <NUM> conducts pixel-level segmentation on the pixel-level class activation maps 664a to 664e generated at <NUM> to outline tissue segments at each pixel of the image patch 502e. The image diagnostic system <NUM> can outline the contour of each tissue segment within each pixel-level class activation maps 664a to 664e.

<FIG> illustrates another example. <FIG> illustrates an example region of interest <NUM>. After the image diagnostic system <NUM> analyzes the region of interest <NUM> at <NUM>, the image diagnostic system <NUM> can generate a class activation map <NUM> (at <NUM>) representing the confidence scores assigned to regions identified as simple columnar epithelial tissues. Based on the class activation map <NUM>, the image diagnostic system <NUM> can conduct pixel-level segmentation to outline the tissue segments to generate a segmented image patch <NUM>. From comparing <FIG> with <FIG>, the pixel-level segmentation conducted at <NUM> can result in finer details and shapes that are lost at the patch-level identification at <NUM>.

The image diagnostic system <NUM> can further adjust the pixel-level class activation maps <NUM> generated at <NUM>.

The digital pathology database <NUM>, <NUM> on which the training dataset is stored typically does not include any image data with non-tissue labels. When the image diagnostic system <NUM> analyzes the image patch 502e, the image diagnostic system <NUM> can produce a background class activation map corresponding to the background for both morphological and functional tissue analyses, and a non-tissue class activation map corresponding to non-tissue characteristics for functional tissue analysis. The image diagnostic system <NUM> can then apply the background and non-tissue class activation maps to each of the tissue class activation maps to avoid making pixel predictions where no such tissue type exists within the digital pathology database <NUM>, <NUM>.

In digital pathology images, background pixels are typically associated with high intensity values (i.e. they appear white in the image), except for tissues that stain transparent, such as white adipose and brown adipose tissues, and exocrine glandular, endocrine glandular, and transport vessels tissues. To generate the background class activation map, the image diagnostic system <NUM> can apply a scaled-and-shifted sigmoid to a mean-RGB image to generate a smoothed high intensity background image. The image diagnostic system <NUM> can then subtract the appropriate transparent staining class activation maps from the smoothed high intensity background image to generate the background class activation map. The image diagnostic system <NUM> can apply a filter, such as a <NUM>-dimensional Gaussian blur Hµ,σ, to reduce the prediction resolution.

For non-tissue class activation map corresponding to non-tissue characteristics of functional tissues, non-functional pixels generally correspond to low confidence scores. The image diagnostic system <NUM> can generate a non-functional tissue class activation map corresponding to tissue pixels without a functional role. To generate this non-functional tissue class activation map, the image diagnostic system <NUM> can subtract the functional tissue class activation maps from the two-dimensional maximum of the other functional class activation maps, the class activation maps for white and brown adipose tissues, and the background class activation map. The image diagnostic system <NUM> can also scale the resulting non-functional tissue class activation map.

At <NUM>, the image diagnostic system <NUM> can apply segmentation post-processing to the identified tissue segments to increase visual homogeneity. The resultant class-specific gradient-class activation maps (CSGCs) at <NUM> can include predictions that poorly conform to object contours. In some embodiments, the post-processing can involve applying a fully-connected conditional random field (CRF) modeling method.

At <NUM>, the image diagnostic system <NUM> can generate pixel-level segmented image patches. Reference will now be made to <FIG>.

<FIG> illustrates an example image patch <NUM>. The image diagnostic system <NUM> applies the method <NUM> to example image patch <NUM> and generates the segmented image patches <NUM> of <FIG> and <NUM> of <FIG>. The segmented image patch <NUM> illustrates identified morphological tissues outlined and labeled, and the segmented image patch <NUM> illustrates identified functional tissues outlined and labeled. The labels may not be necessary as other visual aids can be used to identify the tissue types, such as colour code (which is also used in segmented image patches <NUM> and <NUM>) or shading.

For illustrative purposes, the method <NUM> is described with reference to the image patch 502e of <FIG>. The image diagnostic system <NUM> can also apply the method <NUM> to each of the other image patches 502a to 502d and 502f to 502i. <FIG> illustrate corresponding class activation maps <NUM> and <NUM> for the image patches <NUM> for specific tissue types. <FIG> illustrates corresponding class activation maps 850a to 850i for the loose connective tissues (C. ) and <FIG> illustrates corresponding class activation maps 860a to 860i for the exocrine gland tissues (G.

When stitching together each set of the class activation maps <NUM>, <NUM>, the image diagnostic system <NUM> can average the overlapping areas.

<FIG> illustrates example results from operating the image diagnostic system <NUM> to conduct method <NUM> on image patches of exocrine gland tissues of different health.

The example illustrated at <NUM> relates to an image patch 870a of healthy exocrine gland tissues. The examples illustrated at <NUM> to <NUM> relate to respective image patches 872a to 878a of exocrine gland tissues of worsening health. Example <NUM> relates to adenomatous exocrine gland tissues, example <NUM> relates to moderately differentiated exocrine gland tissues, example <NUM> relates to poorly-to-moderated differentiated exocrine gland tissues, and example <NUM> relates to poorly differentiated exocrine gland tissues.

Images 870b, 872b, 874b, 876b, 878b represent the training segmented image patches. As the exocrine gland tissue worsens in health (from 870b to 878b), the contour of the exocrine gland tissue becomes increasing misshapened.

Images 870c, 872c, 874c, 876c, 878c represent the segmented image patches and image 870d, 872d, 874d, 876d, 878d represent the respective class activation maps generated from operating the image diagnostic system <NUM> to conduct method <NUM> to respective image patches 870a, 872a, 874a, 876a, 878a. As the exocrine gland tissue worsens in health (from 870c to 878c), the image diagnostic system <NUM> generates segmented image patches 870c, 872c, 874c, 876c, 878c and class activation maps 870d, 872d, 874d, 876d, 878d that are increasingly dissimilar in comparison with the training segmented image patches 870b, 872b, 874b, 876b, 878b. Similarly, the respective confidence scores 870e, 872e, 874e, 876e, 878e decrease in value with the worsening exocrine gland tissue.

Referring now to <FIG>, which shows a flowchart <NUM> of an example method of operating the image diagnostic system <NUM> to generate encoded image patches.

At <NUM>, the image diagnostic system <NUM> can receive an image patch, such as image patch 502e.

The image diagnostic system <NUM> can apply the method <NUM> to the image patch 502e to conduct semantic segmentation to outline the various tissue segments identified therein. From applying the method <NUM>, the image diagnostic system <NUM> can generate binary masks corresponding to each identified tissue type. Each binary mask corresponds to pixel-level representations of the tissue segments of the identified tissue type within the image patch.

At <NUM>, the image diagnostic system <NUM> conducts a feature analysis of each binary mask to extract features related to the relevant tissue segment within the binary mask. Example features can include geometrical features related to the structure of the relevant tissue, such as shape (e.g., elongated, rectangular, tubular, circular, blob-like, etc.), and features related to the original image in the same image area associated with the identified shapes (e.g., statistical feature measures like gray-level and standard deviation, edge image features information, etc.). For example, if the image diagnostic system <NUM> detects a blob shape, the image diagnostic system <NUM> can also extract features related the image features inside the blob available from the original image. These features can be used by the image diagnostic system <NUM> to account for gray level image features, for example. In some embodiments, the image diagnostic system <NUM> can obtain the image patch 502e also when conducting <NUM>.

At <NUM>, the image diagnostic system <NUM> generates the encoded image patch for the image patch 502e based on the features identified at <NUM> for each binary mask.

<FIG> illustrates an example representation of an encoded image patch <NUM>. The encoded image patch <NUM> includes feature correspondence values <NUM> that indicate how similar a structure of a tissue type is to a feature identified at <NUM>. In this example, the encoded image patch <NUM> can be represented with a matrix with the columns <NUM> corresponding to the features identified at <NUM> and the rows <NUM> corresponding to the tissue types identified by the feature classifier. Feature correspondence value <NUM>, for example, indicates how similar the structure of the tissue type <NUM> is to the feature <NUM>.

In some embodiments, the image diagnostic system <NUM> can generate the encoded image patch <NUM> for each hierarchy level defined in hierarchical histological taxonomy used for labelling the image patch 502e. For example, when the hierarchical histological taxonomy <NUM> is used for labelling the image patch 502e, an encoded image patch <NUM> can be generated for each hierarchical level <NUM>, <NUM>, and <NUM> so that the image patch 502e can be represented by three different encoded image patches <NUM>. The encoded image patches <NUM> for the different hierarchical levels <NUM>, <NUM>, and <NUM> can be aggregated, in some embodiments.

The encoded image patch <NUM> can be an abstract representation of the tissue characteristics within the image patch 502e. By encoding the image patches <NUM> based on characteristics of the tissue, the encoded image patch <NUM> will include essential features of interest and minimal redundant information. This can facilitate comparison between image patches <NUM> and searching by the image diagnostic system <NUM> for specific features within the stored image patches <NUM>. For example, the image diagnostic system <NUM> can receive search parameters defining specific tissue features of interest and can then identify from the encoded image patches <NUM> stored in the data storage <NUM>, <NUM> the image patches <NUM> that satisfy the search parameters.

<FIG> shows a flowchart <NUM> of an example method of operating the image diagnostic system <NUM> to classify image patches <NUM> as different pathological cases, or diseases. Certain diseased tissues can be characterized by unique tissue arrangements in comparison with normal tissues. The image diagnostic system <NUM> can analyze image patches based on a structural analysis of specific tissues to generate a disease diagnosis.

The method <NUM> involves a pathological case database construction phase <NUM> and a pathological case classification phase <NUM>.

During the pathological case database construction phase <NUM>, the image diagnostic system <NUM> can receive (<NUM>) whole slide images of various pathological cases and receive (<NUM>) user inputs that labels regions associated with specific pathological cases within the whole slide images. The image diagnostic system <NUM> can provide a user interface from which the user inputs can be received. The user inputs can be received from experienced histology labeling experts, such as board-certified pathologists, who are trained to recognize tissues according to the hierarchical histological taxonomy <NUM>.

The image diagnostic system <NUM> can then divide each labelled whole slide image into multiple image patches. In some embodiments, the image diagnostic system <NUM> can identify the labelled regions and generate image patches for the labelled regions to reduce unnecessary computation.

At <NUM>, the image diagnostic system <NUM> organizes the image patches by the associated pathological case and stores the image patches associated with the same pathological case in association with each other and in association with the identified pathological case. For example, the image diagnostic system <NUM> can store image patches associated with the same pathological case in separate, dedicated databases provided by the data storage <NUM>, <NUM>.

The image diagnostic system <NUM> then generates an encoded image patch <NUM> according to the method <NUM> for each image patch stored at <NUM>. The image diagnostic system <NUM> then stores (<NUM>) each encoded image patch <NUM> associated with the same pathological case in association with each other and in association with the identified pathological case. As with the storage of image patches at <NUM>, the image diagnostic system <NUM> can store the encoded image patches <NUM> associated with the same pathological case in separate, dedicated databases provided by the data storage <NUM>, <NUM>. The stored encoded image patches <NUM> can be used by the image diagnostic system <NUM> as a pathological databank for different pathological cases.

The image diagnostic system <NUM> can then use the pathological databank to diagnose new whole scan images for pathological cases. For example, at <NUM>, the image diagnostic system <NUM> can receive a new whole scan image and can then divide the whole scan image into image patches at <NUM>. The image diagnostic system <NUM> applies the method <NUM> to each image patch to separate normal image patches from abnormal image patches. At <NUM>, the image diagnostic system <NUM> stores the identified abnormal image patches in the data storage <NUM>, <NUM>. The identified abnormal image patches can be stored in a separate, dedicated databases provided by the data storage <NUM>, <NUM>, in some embodiments.

The image diagnostic system <NUM> can then conduct the method <NUM> for each image patch stored at <NUM> to generate a respective encoded image patch. The image diagnostic system <NUM> can then compare (<NUM>) each encoded image patch generated at <NUM> with the information available in the pathological databank constructed at <NUM> to generate (<NUM>) a pathological case prediction.

It will be appreciated that numerous specific details are set forth in order to provide a thorough understanding of the example embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description and the drawings are not to be considered as limiting the scope of the embodiments described herein in any way, but rather as merely describing the implementation of the various embodiments described herein.

It should be noted that terms of degree such as "substantially", "about" and "approximately" when used herein mean a reasonable amount of deviation of the modified term such that the end result is not significantly changed. These terms of degree should be construed as including a deviation of the modified term if this deviation would not negate the meaning of the term it modifies.

In addition, as used herein, the wording "and/or" is intended to represent an inclusive-or. That is, "X and/or Y" is intended to mean X or Y or both, for example. As a further example, "X, Y, and/or Z" is intended to mean X or Y or Z or any combination thereof.

It should be noted that the term "coupled" used herein indicates that two elements can be directly coupled to one another or coupled to one another through one or more intermediate elements.

The embodiments of the systems and methods described herein may be implemented in hardware or software, or a combination of both. These embodiments may be implemented in computer programs executing on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. For example and without limitation, the programmable computers (referred to below as computing devices) may be a server, network appliance, embedded device, computer expansion module, a personal computer, laptop, personal data assistant, cellular telephone, smart-phone device, tablet computer, a wireless device or any other computing device capable of being configured to carry out the methods described herein.

In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements are combined, the communication interface may be a software communication interface, such as those for inter-process communication (IPC). In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.

Program code may be applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion.

Each program may be implemented in a high level procedural or object oriented programming and/or scripting language, or both, to communicate with a computer system. However, the programs may be implemented in assembly or machine language, if desired. Each such computer program may be stored on a storage media or a device (e.g. ROM, magnetic disk, optical disc) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Embodiments of the system may also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

Furthermore, the system, processes and methods of the described embodiments are capable of being distributed in a computer program product comprising a computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloadings, magnetic and electronic storage media, digital and analog signals, and the like. The computer useable instructions may also be in various forms, including compiled and non-compiled code.

Claim 1:
A system for identifying one or more tissue types within an image patch according to a hierarchical histological taxonomy, the system comprises:
a supervised digital pathology database having a set of training image patches stored thereon; and
a processor in communication with the supervised digital pathology database and operable to:
access the set of training image patches stored in the supervised digital pathology database;
develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches;
develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy;
apply the feature extractor to the image patch to identify one or more tissue segments within the image patch;
apply the feature classifier to assign a tissue type to the one or more tissue segments identified by the feature extractor for the image patch and to generate a confidence score for each assigned tissue type within the image patch with reference to the class confidence score, wherein applying the feature classifier comprises mapping one or more prediction models to each tissue segment within the image patch to identify the tissue type relative to the training feature vector and class confidence score for that tissue type;
compare each confidence score generated by the feature classifier with a class confidence score generated for a normal tissue of that tissue type to determine whether the image patch is associated with normal tissue; and generate a confidence score vector containing the confidence scores for each tissue type identified for a feature vector by the feature classifier and a health state of each tissue type identified.