Patent Description:
Microscopic imaging of tissue samples is a fundamental tool used for the diagnosis of various diseases and forms the workhorse of pathology and biological sciences. The clinically-established gold standard image of a tissue section is the result of a laborious process, which includes the tissue specimen being formalin-fixed paraffin-embedded (FFPE), sectioned to thin slices (typically -<NUM>-<NUM>), labeled/stained and mounted on a glass slide, which is then followed by its microscopic imaging using e.g., a brightfield microscope. All these steps use multiple reagents and introduce irreversible effects on the tissue. There have been recent efforts to change this workflow using different imaging modalities. Attempts have been made to imaged fresh, non-paraffin-embedded tissue samples using non-linear microscopy methods based on e.g., two-photon fluorescence, second harmonic generation, third-harmonic generation as well as Raman scattering. Other attempts have used a controllable super-continuum source to acquire multi-modal images for chemical analysis of fresh tissue samples. These methods require using ultra-fast lasers or super-continuum sources, which might not be readily available in most settings and require relatively long scanning times due to weaker optical signals. In addition to these, other microscopy methods for imaging non-sectioned tissue samples have also emerged by using UV-excitation on stained samples, or by taking advantage of the fluorescence emission of biological tissue at short wavelengths.

In fact, fluorescence signal creates some unique opportunities for imaging tissue samples by making use of the fluorescent light emitted from endogenous fluorophores. It has been demonstrated that such endogenous fluorescence signatures carry useful information that can be mapped to functional and structural properties of biological specimen and therefore have been used extensively for diagnostics and research purposes. One of the main focus areas of these efforts has been the spectroscopic investigation of the relationship between different biological molecules and their structural properties under different conditions. Some of these well-characterized biological constituents include vitamins (e.g., vitamin A, riboflavin, thiamin), collagen, coenzymes, fatty acids, among others.

While some of the above discussed techniques have unique capabilities to discriminate e.g., cell types and sub-cellular components in tissue samples using various contrast mechanisms, pathologists as well as tumor classification software are in general trained for examining "gold standard" stained tissue samples to make diagnostic decisions. Partially motivated by this, some of the above-mentioned techniques have been augmented to create pseudo-Hematoxylin and Eosin (H&E) images, which are based on a linear approximation that relates the fluorescence intensity of an image to the dye concentration per tissue volume, using empirically determined constants that represent the mean spectral response of various dyes embedded in the tissue. These methods also used exogenous staining to enhance the fluorescence signal contrast in order to create virtual H&E images of tissue samples. The conference proceedings article from <NPL>, discloses a method to virtually stain unstained specimens. The method utilizes dimension reduction and conditional adversarial generative networks (cGANs) which build highly non-linear mappings between input and output images.

In one embodiment, a system and method are provided that utilizes a trained deep neural network that is used for the digital or virtual staining of label-free thin tissue sections or other samples using their fluorescence images obtained from chemically unstained tissue (or other samples). Chemically unstained tissue refers to the lack of standard stains or labels used in histochemical staining of tissue. The fluorescence of chemically unstained tissue may include auto-fluorescence of tissue from naturally occurring or endogenous fluorophores or other endogenous emitters of light at frequencies different from the illumination frequency (i.e., frequency-shifted light). Fluorescence of chemically unstained tissue may further include fluorescence of tissue from exogenously added fluorescent labels or other exogenous emitters of light. Samples are imaged with a fluorescence microscope such as a wide-field fluorescence microscope (or a standard fluorescence microscope). The microscope may utilize a standard near-UV excitation/emission filter set or other excitation/emission light source/filter sets that are known to those skilled in the art. The digital or virtual staining is performed, in some embodiments, on a single fluorescence image obtained of the sample by using, in on preferred embodiment, a trained deep neural network.

In one embodiment, the trained deep neural network is a Convolutional Neural Network (CNN) which is trained using a Generative Adversarial Networks (GAN) model to match the corresponding brightfield microscopic images of tissue samples after they are labeled with a certain histology stain. In this embodiment, a fluorescence image of the unstained sample (e.g., tissue) is input to the trained deep neural network to generate the digitally stained image. Therefore, in this embodiment, the histochemical staining and brightfield imaging steps are completely replaced by the use of the trained deep neural network which generates the digitally stained image. As explained herein, the network inference performed by the trained neural network is fast, taking in some embodiments, less than a second using a standard desktop computer for an imaging field-of-view of ~ <NUM> × <NUM> using e.g., a <NUM>× objective lens. Using a <NUM>× objective for scanning tissue, a network inference time of <NUM> seconds/mm<NUM> was achieved.

The deep learning-based digital/virtual histology staining method using auto-fluorescence has been demonstrated by imaging label-free human tissue samples including salivary gland, thyroid, kidney, liver, lung and skin, where the trained deep neural network output created equivalent images, substantially matching with the images of the same samples that were labeled with three different stains, i.e., H&E (salivary gland and thyroid), Jones stain (kidney) and Masson's Trichrome (liver and lung). Because the trained deep neural network's input image is captured by a conventional fluorescence microscope with a standard filter set, this approach has transformative potential to use unstained tissue samples for pathology and histology applications, entirely bypassing the histochemical staining process, saving time and the attendant costs. This includes the cost of labor, reagents, the additional time involved in staining processes, and the like. For example, for the histology stains that were approximated using the digital or virtual staining process described herein, each staining procedure of a tissue section on average takes ~<NUM> (H&E) and <NUM>-<NUM> hours (Masson's Trichrome and Jones stain), with an estimated cost, including labor, of $<NUM>-<NUM> for H&E and >$<NUM>-<NUM> for Masson's Trichrome and Jones stain. Furthermore, some of these histochemical staining processes require time-sensitive steps, demanding the expert to monitor the process under a microscope, which makes the entire process not only lengthy and relatively costly, but also laborious. The system and method disclosed herein bypasses all these staining steps, and also allows the preservation of unlabeled tissue sections for later analysis, such as micro-marking of sub-regions of interest on the unstained tissue specimen that can be used for more advanced immunochemical and molecular analysis to facilitate e.g., customized therapies. Furthermore, the staining efficacy of this approach for whole slide images (WSIs) corresponding to some of these samples was blindly evaluated by a group of pathologists, who were able to recognize histopathological features with the digital/virtual staining technique, achieving a high degree of agreement with the histologically stained images of the same samples.

Further, this deep learning-based digital/virtual histology staining framework can be broadly applied to other excitation wavelengths or fluorescence filter sets, as well as to other microscopy modalities (such as non-linear microscopy) that utilize additional endogenous contrast mechanisms. In the experiments, sectioned and fixed tissue samples were used to be able to provide meaningful comparisons to the results of the standard histochemical staining process. However, the presented approach would also work with non-fixed, non-sectioned tissue samples, potentially making it applicable to use in surgery rooms or at the site of a biopsy for rapid diagnosis or telepathology applications. Beyond its clinical applications, this method could broadly benefit histology field and its applications in life science research and education.

In one embodiment, a method of generating a digitally stained microscopic image of a label-free sample includes providing a trained, deep neural network that is run using image processing software executed using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images or image patches of the same sample. The label-free sample may include tissues, cells, pathogens, biological fluid smears, or other micro-objects of interest. In some embodiments, the deep neural network may be trained using one or more tissue type/chemical stain type combinations. For example, this may include tissue type A with stain #<NUM>, stain #<NUM>, stain #<NUM>, etc. In some embodiments, the deep neural network may be trained using tissue that has been stained with multiple stains.

A fluorescence image of the sample is input to the trained, deep neural network. The trained, deep neural network then outputs a digitally stained microscopic image of the sample based on the input fluorescence image of the sample. In one embodiment, the trained, deep neural network is a convolutional neural network (CNN). This may include a CNN that uses a Generative Adversarial Network (GAN) model. The fluorescence input image of the sample is obtained using a fluorescence microscope and an excitation light source (e.g., UV or near UV emitting light source). In some alternative embodiments, multiple fluorescence images are input into the trained, deep neural network. For example, one fluorescence image may be obtained at a first filtered wavelength or wavelength range while another fluorescence image may be obtained at a second filtered wavelength or wavelength range. These two fluorescence images are then input into the trained, deep neural network to output a single digitally/virtually stained image. In another embodiment, the obtained fluorescence image may be subject to one or more linear or non-linear pre-processing operations selected from contrast enhancement, contrast reversal, image filtering which may be input alone or in combination with the obtained fluorescence image into the trained, deep neural network.

For example, a method of generating a digitally stained microscopic image of a label-free sample includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images or image patches of the same sample. A first fluorescence image of the sample is obtained using a fluorescence microscope and wherein fluorescent light at a first emission wavelength or wavelength range is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample. A second fluorescence image of the sample is obtained using a fluorescence microscope and wherein fluorescent light at a second emission wavelength or wavelength range is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample. The first and second fluorescence images may be obtained by using different excitation/emission wavelength combinations. The first and second fluorescence images of the sample are then input to the trained, deep neural network, the trained, deep neural network outputting the digitally stained microscopic image of the sample that is substantially equivalent to a corresponding brightfield image of the same sample that has been chemically stained.

In another embodiment, a system for generating digitally stained microscopic images of a chemically unstained sample includes a computing device having image processing software executed thereon or thereby, the image processing software comprising a trained, deep neural network that is executed using one or more processors of the computing device. The trained, deep neural network is trained with a plurality of matched chemically stained images or image patches and their corresponding fluorescence images or image patches of the same sample. The image processing software is configured to receive one or more fluorescence image(s) of the sample and output the digitally stained microscopic image of the sample that is substantially equivalent to a corresponding brightfield image of the same sample that has been chemically stained.

<FIG> schematically illustrates one embodiment of a system <NUM> for outputting digitally stained images <NUM> from an input microscope image <NUM> of a sample <NUM>. As explained herein, the input image <NUM> is a fluorescence image <NUM> of a sample <NUM> (such as tissue in one embodiment) that is not stained or labeled with a fluorescent stain or label. Namely, the input image <NUM> is an autofluorescence image <NUM> of the sample <NUM> in which the fluorescent light that is emitted by the sample <NUM> is the result of one or more endogenous fluorophores or other endogenous emitters of frequency-shifted light contained therein. Frequency-shifted light is light that is emitted at a different frequency (or wavelength) that differs from the incident frequency (or wavelength). Endogenous fluorophores or endogenous emitters of frequency-shifted light may include molecules, compounds, complexes, molecular species, biomolecules, pigments, tissues, and the like. In some embodiments, the input image <NUM> (e.g., the raw fluorescent image) is subject to one or more linear or non-linear pre-processing operations selected from contrast enhancement, contrast reversal, image filtering. The system includes a computing device <NUM> that contains one or more processors <NUM> therein and image processing software <NUM> that incorporates the trained, deep neural network <NUM> (e.g., a convolutional neural network as explained herein in one or more embodiments). The computing device <NUM> may include, as explained herein, a personal computer, laptop, mobile computing device, remote server, or the like, although other computing devices may be used (e.g., devices that incorporate one or more graphic processing units (GPUs)) or other application specific integrated circuits (ASICs). GPUs or ASICs can be used to accelerate training as well as final image output. The computing device <NUM> may be associated with or connected to a monitor or display <NUM> that is used to display the digitally stained images <NUM>. The display <NUM> may be used to display a Graphical User Interface (GUI) that is used by the user to display and view the digitally stained images <NUM>. In one embodiment, the user may be able to trigger or toggle manually between multiple different digital/virtual stains for a particular sample <NUM> using, for example, the GUI. Alternatively, the triggering or toggling between different stains may be done automatically by the computing device <NUM>. In one preferred embodiment, the trained, deep neural network <NUM> is a Convolution Neural Network (CNN).

For example, in one preferred embodiment as is described herein, the trained, deep neural network <NUM> is trained using a GAN model. In a GAN-trained deep neural network <NUM>, two models are used for training. A generative model is used that captures data distribution while a second model estimates the probability that a sample came from the training data rather than from the generative model. Details regarding GAN may be found in <NPL>), which is incorporated by reference herein. Network training of the deep neural network <NUM> (e.g., GAN) may be performed the same or different computing device <NUM>. For example, in one embodiment a personal computer may be used to train the GAN although such training may take a considerable amount of time. To accelerate this training process, one or more dedicated GPUs may be used for training. As explained herein, such training and testing was performed on GPUs obtained from a commercially available graphics card. Once the deep neural network <NUM> has been trained, the deep neural network <NUM> may be used or executed on a different computing device <NUM> which may include one with less computational resources used for the training process (although GPUs may also be integrated into execution of the trained deep neural network <NUM>).

The image processing software <NUM> can be implemented using Python and TensorFlow although other software packages and platforms may be used. The trained deep neural network <NUM> is not limited to a particular software platform or programming language and the trained deep neural network <NUM> may be executed using any number of commercially available software languages or platforms. The image processing software <NUM> that incorporates or runs in coordination with the trained, deep neural network <NUM> may be run in a local environment or a remove cloud-type environment. In some embodiments, some functionality of the image processing software <NUM> may run in one particular language or platform (e.g., image normalization) while the trained deep neural network <NUM> may run in another particular language or platform. Nonetheless, both operations are carried out by image processing software <NUM>.

As seen in <FIG>, in one embodiment, the trained, deep neural network <NUM> receives a single fluorescence image <NUM> of an unlabeled sample <NUM>. In other embodiments, for example, where multiple excitation channels are used (see melanin discussion herein), there may be multiple fluorescence images <NUM> of the unlabeled sample <NUM> that are input to the trained, deep neural network <NUM> (e.g., one image per channel). The fluorescence images <NUM> may include a wide-field fluorescence image <NUM> of an unlabeled tissue sample <NUM>. Wide-field is meant to indicate that a wide field-of-view (FOV) is obtained by scanning of a smaller FOV, with the wide FOV being in the size range of <NUM>-<NUM>,<NUM><NUM>. For example, smaller FOVs may be obtained by a scanning fluorescent microscope <NUM> that uses image processing software <NUM> to digitally stitch the smaller FOVs together to create a wider FOV. Wide FOVs, for example, can be used to obtain whole slide images (WSI) of the sample <NUM>. The fluorescence image is obtained using an imaging device <NUM>. For the fluorescent embodiments described herein, this may include a fluorescence microscope <NUM>. The fluorescent microscope <NUM> includes an excitation light source that illuminates the sample <NUM> as well as one or more image sensor(s) (e.g., CMOS image sensors) for capturing fluorescent light that is emitted by fluorophores or other endogenous emitters of frequency-shifted light contained in the sample <NUM>. The fluorescence microscope <NUM> may, in some embodiments, include the ability to illuminate the sample <NUM> with excitation light at multiple different wavelengths or wavelength ranges/bands. This may be accomplished using multiple different light sources and/or different filter sets (e.g., standard UV or near-UV excitation/emission filter sets). In addition, the fluorescence microscope <NUM> may include, in some embodiments, multiple filter sets that can filter different emission bands. For example, in some embodiments, multiple fluorescence images <NUM> may be captured, each captured at a different emission band using a different filter set.

The sample <NUM> may include, in some embodiments, a portion of tissue that is disposed on or in a substrate <NUM>. The substrate <NUM> may include an optically transparent substrate in some embodiments (e.g., a glass or plastic slide or the like). The sample <NUM> may include a tissue sections that are cut into thin sections using a microtome device or the like. Thin sections of tissue <NUM> can be considered a weakly scattering phase object, having limited amplitude contrast modulation under brightfield illumination. The sample <NUM> may be imaged with or without a cover glass/cover slip. The sample may involve frozen sections or paraffin (wax) sections. The tissue sample <NUM> may be fixed (e.g., using formalin) or unfixed. The tissue sample <NUM> may include mammalian (e.g., human or animal) tissue or plant tissue. The sample <NUM> may also include other biological samples, environmental samples, and the like. Examples include particles, cells, cell organelles, pathogens, or other micro-scale objects of interest (those with micrometer-sized dimensions or smaller). The sample <NUM> may include smears of biological fluids or tissue. These include, for instance, blood smears, Papanicolaou or Pap smears. As explained herein, for the fluorescent-based embodiments, the sample <NUM> includes one or more naturally occurring or endogenous fluorophores that fluoresce and are captured by the fluorescent microscope device <NUM>. Most plant and animal tissues show some autofluorescence when excited with ultraviolet or near ultra-violet light. Endogenous fluorophores may include by way of illustration proteins such as collagen, elastin, fatty acids, vitamins, flavins, porphyrins, lipofuscins, co-enzymes (e.g., NAD(P)H). In some optional embodiments, exogenously added fluorescent labels or other exogenous emitters of light may also be added. As explained herein, the sample <NUM> may also contain other endogenous emitters of frequency-shifted light.

The trained, deep neural network <NUM> in response to the input image <NUM> outputs or generates a digitally stained or labelled output image <NUM>. The digitally stained output image <NUM> has "staining" that has been digitally integrated into the stained output image <NUM> using the trained, deep neural network <NUM>. In some embodiments, such as those involved tissue sections, the trained, deep neural network <NUM> appears to a skilled observer (e.g., a trained histopathologist) to be substantially equivalent to a corresponding brightfield image of the same tissue section sample <NUM> that has been chemically stained. Indeed, as explained herein, the experimental results obtained using the trained, deep neural network <NUM> show that trained pathologists were able to recognize histopathologic features with both staining techniques (chemically stained vs. digitally/virtually stained) and with a high degree of agreement between the techniques, without a clear preferable staining technique (virtual vs. histological). This digital or virtual staining of the tissue section sample <NUM> appears just like the tissue section sample <NUM> had undergone histochemical staining even though no such staining operation was conducted.

<FIG> schematically illustrates the operations involved in a typical fluorescent-based embodiment. As seen in <FIG>, a sample <NUM> such as an unstained tissue section is obtained. This may be obtained from living tissue such as through a biopsy B or the like. The unstained tissue section sample <NUM> is then subject to fluorescent imaging using a fluorescence microscope <NUM> and generates a fluorescence image <NUM>. This fluorescence image <NUM> is then input to a trained, deep neural network <NUM> that then promptly outputs a digitally stained image <NUM> of the tissue section sample <NUM>. This digitally stained image <NUM> closely resembles the appearance of a brightfield image of the same tissue section sample <NUM> had the actual tissue section sample <NUM> be subject to histochemical staining. <FIG> illustrates (using dashed arrows) the conventional process whereby the tissue section sample <NUM> is subject to histochemical staining <NUM> followed by conventional brightfield microscopic imaging <NUM> to generate a conventional brightfield image <NUM> of the stained tissue section sample <NUM>. As seen in <FIG>, the digitally stained image <NUM> closely resembles the actual chemically stained image <NUM>. Similar resolution and color profiles are obtained using the digital staining platform described herein. This digitally stained image <NUM> may, as illustrated in <FIG>, be shown or displayed on a computer monitor <NUM> but it should be appreciated the digitally stained image <NUM> may be displayed on any suitable display (e.g., computer monitor, tablet computer, mobile computing device, mobile phone, etc.). A GUI may be displayed on the computer monitor <NUM> so that the user may view and optionally interact with the digitally stained image <NUM> (e.g., zoom, cut, highlight, mark, adjust exposure, and the like).

The system <NUM> and methods described herein was tested and demonstrated using different combinations of tissue section samples <NUM> and stains. Following the training of a CNN-based deep neural network <NUM> its inference was blindly tested by feeding it with the auto-fluorescence images <NUM> of label-free tissue sections <NUM> that did not overlap with the images that were used in the training or validation sets. <FIG> illustrates the results for a salivary gland tissue section, which was digitally/virtually stained to match H&E stained brightfield images <NUM> (i.e., the ground truth images) of the same sample <NUM>. These results demonstrate the capability of the system <NUM> to transform a fluorescence image <NUM> of a label-free tissue section <NUM> into a brightfield equivalent image <NUM>, showing the correct color scheme that is expected from an H&E stained tissue, containing various constituents such as epithelioid cells, cell nuclei, nucleoli, stroma, and collagen. Evaluation of both <FIG> show the H&E stains demonstrate a small island of infiltrating tumor cells within subcutaneous fibro-adipose tissue. Note the nuclear detail, including distinction of nucleoli (arrow) and chromatin texture, is clearly appreciated in both panels. Similarly, in <FIG>, the H&E stains demonstrate infiltrating squamous cell carcinoma. The desmoplastic reaction with edematous myxoid change (asterisk) in the adjacent stroma is clearly identifiable in both stains.

Next, the deep network <NUM> was trained to digitally/virtually stain other tissue types with two different stains, i.e., the Jones methenamine silver stain (kidney) and the Masson's Trichrome stain (liver and lung). <FIG> and <FIG> summarize the results for deep learning-based digital/virtual staining of these tissue sections <NUM>, which very well match to the brightfield images <NUM> of the same samples <NUM>, captured after the histochemical staining process. These results illustrate that the trained deep neural network <NUM> is capable of inferring the staining patterns of different types of histology stains used for different tissue types, from a single fluorescence image <NUM> of a label-free specimen (i.e., without any histochemical stains). With the same overall conclusion as in <FIG>, it was also confirmed by a pathologist that the neural network output images <FIG> and <FIG> correctly reveal the histological features corresponding to hepatocytes, sinusoidal spaces, collagen and fat droplets (<FIG>), consistent with the way that they appear in the brightfield images <NUM> of the same tissue samples <NUM>, captured after the chemical staining (<FIG>). Similarly, it was also confirmed by the same expert that the deep neural network output images <NUM> reported in <FIG> (lung) reveal consistently stained histological features corresponding to vessels, collagen and alveolar spaces as they appear in the brightfield images <NUM> of the same tissue sample <NUM> imaged after the chemical staining (FIGS. <NUM> and 6P).

The digitally/virtually-stained output images <NUM> from the trained, deep neural network <NUM> were compared to the standard histochemical staining images <NUM> for diagnosing multiple types of conditions on multiple types of tissues, which were either Formalin-Fixed Paraffin-Embedded (FFPE) or frozen sections. The results are summarized in Table <NUM> below. The analysis of fifteen (<NUM>) tissue sections by four board certified pathologists (who were not aware of the virtual staining technique) demonstrated <NUM>% non-major discordance, defined as no clinically significant difference in diagnosis among professional observers. The "time to diagnosis" varied considerably among observers, from an average of <NUM> seconds-per-image for observer <NUM> to <NUM> seconds-per-image for observer <NUM>. However, the intra-observer variability was very minor and tended towards shorter time to diagnosis with the virtually-stained slide images <NUM> for all the observers except observer <NUM> which was equal, i.e., ~<NUM> seconds-per-image for both the virtual slide image <NUM> and the histology stained slide image <NUM>. These indicate very similar diagnostic utility between the two image modalities.

After evaluating the differences in tissue section and stains, the ability of the virtual staining system <NUM> was tested in the specialized staining histology workflow. In particular, the autofluorescence distribution of <NUM> label-free samples of liver tissue sections and <NUM> label-free tissue sections of kidney were imaged with a <NUM>×/<NUM>. 75NA objective lens. All liver and kidney tissue sections were obtained from different patients and included both small biopsies and larger resections. All the tissue sections were obtained from FFPE but not cover slipped. After the autofluorescence scanning, the tissue sections were histologically stained with Masson's Trichrome (<NUM> liver tissue sections) and Jones' stain (<NUM> kidney tissue sections). The WSIs were then divided into training and test sets. For the liver slides cohort, <NUM> WSIs were used for training the virtual staining algorithm and <NUM> WSIs were used for blind testing; for the kidney slides cohort, <NUM> WSIs were used for training the algorithm and <NUM> WSIs were used for testing. The study pathologists were blinded to the staining techniques for each WSI and were asked to apply a <NUM>-<NUM> number grade for the quality of the different stains: <NUM> = perfect, <NUM> = very good, <NUM> = acceptable, <NUM> = unacceptable. Secondly, the study pathologists applied the same score scale (<NUM>-<NUM>) for specific features: nuclear detail (ND), cytoplasmic detail (CD) and extracellular fibrosis (EF), for liver only. These results are summarized in Table <NUM> (Liver) and Table <NUM> (Kidney) below (winner is bolded). The data indicates that the pathologists were able to recognize histopathologic features with both staining techniques and with a high degree of agreement between the techniques, without a clear preferable staining technique (virtual vs. histological).

Next, beyond the visual comparison provided in <FIG>, <FIG>, <FIG>, the results of the trained deep neural network <NUM> were quantified by first calculating the pixel-level differences between the brightfield images <NUM> of the chemically stained samples <NUM> and the digitally/virtually stained images <NUM> that are synthesized using the deep neural network <NUM> without the use of any labels/stains. Table <NUM> below summarizes this comparison for different combinations of tissue types and stains, using the YCbCr color space, where the chroma components Cb and Cr entirely define the color, and Y defines the brightness component of the image. The results of this comparison reveal that the average difference between these two sets of images is < ~<NUM>% and < -<NUM>%, for the chroma (Cb, Cr) and brightness (Y) channels, respectively. Next, a second metric was used to further quantify the comparison, i.e., the structural similarity index (SSIM), which is in general used to predict the score that a human observer will give for an image, in comparison to a reference image (Equation <NUM> herein). SSIM ranges between <NUM> and <NUM>, where <NUM> defines the score for identical images. The results of this SSIM quantification are also summarized in Table <NUM>, which very well illustrates the strong structural similarity between the network output images <NUM> and the brightfield images <NUM> of the chemically stained samples.

One should note that the brightfield images <NUM> of the chemically stained tissue samples <NUM> in fact do not provide the true gold standard for this specific SSIM and YCbCr analysis of the network output images <NUM>, because there are uncontrolled variations and structural changes that the tissue undergoes during the histochemical staining process and related dehydration and clearing steps. Another variation that was noticed for some of the images was that the automated microscope scanning software selected different auto-focusing planes for the two imaging modalities. All these variations create some challenges for the absolute quantitative comparison of the two sets of images (i.e., the network output <NUM> for a label-free tissue vs. the brightfield image <NUM> of the same tissue after the histological staining process).

An interesting by-product of the digital/virtual staining system <NUM> can be staining standardization. In other words, the trained deep neural network <NUM> converges to a "common stain" colorization scheme whereby the variation in the histologically stained tissue images <NUM> is higher than that of the virtually stained tissue images <NUM>. The colorization of the virtual stain is solely the result of its training (i.e., the gold standard histological staining used during the training phase) and can be further adjusted based on the preferences of pathologists, by retraining the network with a new stain colorization. Such "improved" training can be created from scratch or accelerated through transfer learning. This potential staining standardization using deep learning can remedy the negative effects of human-to-human variations at different stages of the sample preparation, create a common ground among different clinical laboratories, enhance the diagnostic workflow for clinicians as well as assist the development of new algorithms such as automatic tissue metastasis detection or grading of different types of cancer, among others.

Using the concept of transfer learning, the training procedure for new tissue and/or stain types can converge much faster, while also reaching an improved performance, i.e., a better local minimum in the training cost/loss function. This means, a pre-leamt CNN model deep neural network <NUM>, from a different tissue-stain combination, can be used to initialize the deep neural network <NUM> to statistically learn virtual staining of a new combination. <FIG> shows the favorable attributes of such an approach: a new deep neural network <NUM> was trained to virtually stain the auto-fluorescence images <NUM> of unstained thyroid tissue sections, and it was initialized using the weights and biases of another deep neural network <NUM> that was previously trained for H&E virtual staining of the salivary gland. The evolution of the loss metric as a function of the number of iterations used in the training phase clearly demonstrates that the new thyroid deep network <NUM> rapidly converges to a lower minimum in comparison to the same network architecture which was trained from scratch, using random initialization as seen in <FIG> compares the output images <NUM> of this thyroid network <NUM> at different stages of its learning process, which further illustrates the impact of transfer learning to rapidly adapt the presented approach to new tissue/stain combinations. The network output images <NUM>, after the training phase with e.g., ≥ <NUM>,<NUM> iterations, reveal that cell nuclei show irregular contours, nuclear grooves, and chromatin pallor, suggestive of papillary thyroid carcinoma; cells also show mild to moderate amounts of eosinophilic granular cytoplasm and the fibrovascular core at the network output image shows increased inflammatory cells including lymphocytes and plasma cells. <FIG> illustrates the corresponding H&E chemically stained brightfield image <NUM>.

The method of using the trained, deep neural network <NUM> can be combined with other excitation wavelengths and/or imaging modalities in order to enhance its inference performance for different tissue constituents. For example, melanin detection on a skin tissue section sample using virtual H&E staining was tried. However, melanin was not clearly identified in the output of the network, as it presents a weak auto-fluorescent signal at DAPI excitation / emission wavelengths measured in the experimental system described herein. One potential method to increase the autofluorescence of melanin is to image the samples while they are in an oxidizing solution. However, a more practical alternative was used where an additional autofluorescence channel was employed, originating from e.g., Cy5 filter (excitation <NUM> / emission <NUM>) such that the melanin signal can be enhanced and accurately inferred in the trained, deep neural network <NUM>. By training the network <NUM> using both the DAPI and Cy5 autofluorescence channels, the trained, deep neural network <NUM> was able to successfully determine where melanin occurs in the sample, as illustrated in <FIG>. In contrast, when only the DAPI channel was used (<FIG>), the network <NUM> was unable to determine the areas that contain melanin (the areas appear white). Stated differently, the additional autofluorescence information from the Cy5 channel was used by the network <NUM> to distinguish melanin from the background tissue. For the results that are shown in <FIG>, the images <NUM> were acquired using a lower resolution objective lens (<NUM>×/<NUM>. 45NA) for the Cy5 channel, to supplement the high-resolution DAPI scan (<NUM>×/<NUM>. 75NA), as it was hypothesized that most necessary information is found in the high-resolution DAPI scan and the additional information (for example, the melanin presence) can be encoded with the lower resolution scan. In this manner, two different channels were used with one of the channels being used at a lower resolution to identify the melanin. This may require multiple scanning passes of the sample <NUM> with the fluorescent microscope <NUM>. In yet another multi-channel embodiment, multiple images <NUM> may be fed to the trained, deep neural network <NUM>. This may include, for example, raw fluorescent images in combination with one or more images that have undergone linear or non-linear image pre-processing such as contrast enhancement, contrast reversal, and image filtering.

The system <NUM> and methods described herein show the ability to digitally/virtually stain label-free tissue sections <NUM>, using a supervised deep learning technique that uses a single fluorescence image <NUM> of the sample as input, captured by a standard fluorescence microscope <NUM> and filter set (in other embodiments multiple fluorescence images <NUM> are input when multiple fluorescence channels are used). This statistical learning-based method has the potential to restructure the clinical workflow in histopathology and can benefit from various imaging modalities such as fluorescence microscopy, non-linear microscopy, holographic microscopy, stimulated Raman scattering microscopy, and optical coherence tomography, among others, to potentially provide a digital alternative to the standard practice of histochemical staining of tissue samples <NUM>. Here, the method was demonstrated using fixed unstained tissue samples <NUM> to provide a meaningful comparison to chemically stained tissue samples, which is essential to train the deep neural network <NUM> as well as to blindly test the performance of the network output against the clinically-approved method. However, the presented deep learning-based approach is broadly applicable to different types and states of a sample <NUM> including un-sectioned, fresh tissue samples (e.g., following a biopsy procedure) without the use of any labels or stains. Following its training, the deep neural network <NUM> can be used to digitally/virtually stain the images of label-free fresh tissue samples <NUM>, acquired using e.g., UV or deep UV excitation or even nonlinear microscopy modalities. For example, Raman microscopy can provide very rich label-free biochemical signatures that can further enhance the effectiveness of the virtual staining that the neural network learns.

An important part of the training process involves matching the fluorescence images <NUM> of label-free tissue samples <NUM> and their corresponding brightfield images <NUM> after the histochemical staining process (i.e., chemically stained images). One should note that during the staining process and related steps, some tissue constitutes can be lost or deformed in a way that will mislead the loss/cost function in the training phase. This, however, is only a training and validation related challenge and does not pose any limitations on the practice of a well-trained deep neural network <NUM> for virtual staining of label-free tissue samples <NUM>. To ensure the quality of the training and validation phase and minimize the impact of this challenge on the network's performance, a threshold was established for an acceptable correlation value between the two sets of images (i.e., before and after the histochemical staining process) and eliminated the non-matching image pairs from the training/validation set to make sure that the deep neural network <NUM> learns the real signal, not the perturbations to the tissue morphology due to the chemical staining process. In fact, this process of cleaning the training/validation image data can be done iteratively: one can start with a rough elimination of the obviously altered samples and accordingly converge on a neural network <NUM> that is trained. After this initial training phase, the output images <NUM> of each sample in the available image set can be screened against their corresponding brightfield images <NUM> to set a more refined threshold to reject some additional images and further clean the training/validation image set. With a few iterations of this process, one can, not only further refine the image set, but also improve the performance of the final trained deep neural network <NUM>.

The methodology described above will mitigate some of the training challenges due to random loss of some tissue features after the histological staining process. In fact, this highlights another motivation to skip the laborious and costly procedures that are involved in histochemical staining as it will be easier to preserve the local tissue histology in a label-free method, without the need for an expert to handle some of the delicate procedures of the staining process, which sometimes also requires observing the tissue under a microscope.

Using a PC desktop, the training phase of the deep neural network <NUM> takes a considerable amount of time (e.g., ~<NUM> hours for the salivary gland network). However, this entire process can be significantly accelerated by using dedicated computer hardware, based on GPUs. Furthermore, as already emphasized in <FIG>, transfer learning provides a "warm start" to the training phase of a new tissue/stain combination, making the entire process significantly faster. Once the deep neural network <NUM> has been trained, the digital/virtual staining of a sample image <NUM> is performed in a single, non-iterative manner, which does not require a trial-and-error approach or any parameter tuning to achieve the optimal result. Based on its feed-forward and non-iterative architecture, the deep neural network <NUM> rapidly outputs a virtually stained image in less than one second (e.g., <NUM> sec, corresponding to a sample field-of-view of ~ <NUM> × <NUM>). With further GPU-based acceleration, it has the potential to achieve real-time or near real-time performance in outputting digitally/virtually stained images <NUM> which might especially be useful in the operating room or for in vivo imaging applications.

The digital/virtual staining procedure that is implemented is based on training a separate CNN deep neural network <NUM> for each tissue/stain combination. If one feeds a CNN-based deep neural network <NUM> with the auto-fluorescence images <NUM> having different tissue/stain combinations, it will not perform as desired. This, however, is not a limitation because for histology applications, the tissue type and stain type are pre-determined for each sample <NUM> of interest, and therefore, a specific CNN selection for creating the digitally/virtually stained image <NUM> from an auto-fluorescence image <NUM> of the unlabeled sample <NUM> does not require an additional information or resource. Of course, a more general CNN model can be learnt for multiple tissue/stain combinations by e.g., increasing the number of trained parameters in the model, at the cost of a possible increase in the training and inference times. Another avenue is the potential of the system <NUM> and method to perform multiple virtual stains on the same unlabeled tissue type.

A significant advantage of the system <NUM> is that it is quite flexible. It can accommodate feedback to statistically mend its performance if a diagnostic failure is detected through a clinical comparison, by accordingly penalizing such failures as they are caught. This iterative training and transfer learning cycle, based on clinical evaluations of the performance of the network output, will help optimize the robustness and clinical impact of the presented approach. Finally, this method and system <NUM> may be used for micro-guiding molecular analysis at the unstained tissue level, by locally identifying regions of interest based on virtual staining, and using this information to guide subsequent analysis of the tissue for e.g., micro-immunohistochemistry or sequencing. This type of virtual micro-guidance on an unlabeled tissue sample can facilitate high-throughput identification of sub-types of diseases, also helping the development of customized therapies for patients.

Formalin-fixed paraffin-embedded <NUM> thick tissue sections were deparaffinized using Xylene and mounted on a standard glass slide using Cytoseal™ (Thermo-Fisher Scientific, Waltham, MA USA), followed by placing a coverslip (Fisherfinest™, 24x50-<NUM>, Fisher Scientific, Pittsburgh, PA USA). Following the initial auto-fluorescence imaging process (using a DAPI excitation and emission filter set) of the unlabeled tissue sample, the slide was then put into Xylene for approximately <NUM> hours or until the coverslip can be removed without damaging the tissue. Once the coverslip is removed the slide was dipped (approximately <NUM> dips) in absolute alcohol, <NUM>% alcohol and then washed in D. water for ~<NUM>. This step was followed by the corresponding staining procedures, used for H&E, Masson's Trichrome or Jones stains. This tissue processing path is only used for the training and validation of the approach and is not needed after the network has been trained. To test the system and method, different tissue and stain combinations were used: the salivary gland and thyroid tissue sections were stained with H&E, kidney tissue sections were stained with Jones stain, while the liver and lung tissue sections were stained with Masson's trichrome.

In the WSI study, the FFPE <NUM>-<NUM> thick tissue sections were not cover slipped during the autofluorescence imaging stage. Following the autofluorescence imaging, the tissue samples were histologically stained as described above (Masson's Trichrome for the liver and Jones for the kidney tissue sections). The unstained frozen samples were prepared by embedding the tissue section in O. (Tissue Tek, SAKURA FINETEK USA INC) and dipped in <NUM>-Methylbutane with dry ice. The frozen section was then cut to <NUM> sections and was put in a freezer until it was imaged. Following the imaging process, the tissue section was washed with <NUM>% alcohol, H&E stained and cover slipped. The samples were obtained from the Translational Pathology Core Laboratory (TPCL) and were prepared by the Histology Lab at UCLA. The kidney tissue sections of diabetic and non-diabetic patients were obtained under IRB <NUM>-<NUM> (UCLA). All the samples were obtained after de-identification of the patient related information, and were prepared from existing specimen. Therefore, this work did not interfere with standard practices of care or sample collection procedures.

The label-free tissue auto-fluorescence images <NUM> were captured using a conventional fluorescence microscope <NUM> (IX83, Olympus Corporation, Tokyo, Japan) equipped with a motorized stage, where the image acquisition process was controlled by MetaMorph® microscope automation software (Molecular Devices, LLC). The unstained tissue samples were excited with near UV light and imaged using a DAPI filter cube (OSFI3-DAPI-5060C, excitation wavelength <NUM>/<NUM> bandwidth, emission wavelength <NUM>/<NUM> bandwidth) with a <NUM>×/<NUM>. 95NA objective lens (Olympus UPLSAPO 40X2/<NUM>. <NUM>) or <NUM>×/<NUM>. 75NA objective lens (Olympus UPLSAPO 20X/<NUM>. For the melanin inference, the autofluorescence images of the samples were additionally acquired using a Cy5 filter cube (CY5-4040C-OFX, excitation wavelength <NUM> / <NUM> bandwidth, emission wavelength <NUM> / <NUM> bandwidth) with a <NUM>×/<NUM>. 4NA objective lens (Olympus UPLSAPO10X2). Each auto-fluorescence image was captured with a scientific CMOS sensor (ORCA-flash4. <NUM> v2, Hamamatsu Photonics K. , Shizuoka Prefecture, Japan) with an exposure time of ~<NUM>. The brightfield images <NUM> (used for the training and validation) were acquired using a slide scanner microscope (Aperio AT, Leica Biosystems) using a <NUM>×/<NUM>. 75NA objective (Plan Apo), equipped with a <NUM>× magnification adapter.

Since the deep neural network <NUM> aims to learn a statistical transformation between an auto-fluorescence image <NUM> of a chemically unstained tissue sample <NUM> and a brightfield image <NUM> of the same tissue sample <NUM> after the histochemical staining, it is important to accurately match the FOV of the input and target images (i.e., unstained auto-fluorescence image <NUM> and the stained bright-filed image <NUM>). An overall scheme describing the global and local image registration process is described in <FIG> which was implemented in MATLAB (The MathWorks Inc. , Natick, MA, USA). The first step in this process is to find candidate features for matching unstained auto-fluorescence images and chemically stained brightfield images. For this, each auto-fluorescence image <NUM> (<NUM>×<NUM> pixels) is down-sampled to match the effective pixel size of the brightfield microscope images. This results in a <NUM>×<NUM>-pixel unstained auto-fluorescent tissue image, which is contrast enhanced by saturating the bottom <NUM>% and the top <NUM>% of all the pixel values, and contrast reversed (image 20a in <FIG>) to better represent the color map of the grayscale converted whole slide image. Then, a correlation patch process <NUM> is performed in which a normalized correlation score matrix is calculated by correlating each one of the <NUM>×<NUM>-pixel patches with the corresponding patch of the same size, extracted from the whole slide gray-scale image 48a. The entry in this matrix with the highest score represents the most likely matched FOV between the two imaging modalities. Using this information (which defines a pair of coordinates), the matched FOV of the original whole slide brightfield image <NUM> is cropped 48c to create target images 48d. Following this FOV matching procedure <NUM>, the auto-fluorescence <NUM> and brightfield microscope images <NUM> are coarsely matched. However, they are still not accurately registered at the individual pixel-level, due to the slight mismatch in the sample placement at the two different microscopic imaging experiments (auto-fluorescence, followed by brightfield), which randomly causes a slight rotation angle (e.g., -<NUM>-<NUM> degrees) between the input and target images of the same sample.

The second part of the input-target matching process involves a global registration step <NUM>, which corrects for this slight rotation angle between the auto-fluorescence and brightfield images. This is done by extracting feature vectors (descriptors) and their corresponding locations from the image pairs, and matching the features by using the extracted descriptors. Then, a transformation matrix corresponding to the matched pairs is found using the M-estimator Sample Consensus (MSAC) algorithm, which is a variant of the Random Sample Consensus (RANSAC) algorithm. Finally, the angle-corrected image 48e is obtained by applying this transformation matrix to the original brightfield microscope image patch 48d. Following the application of this rotation, the images 20b, 48e are further cropped by <NUM> pixels (<NUM> pixels on each side) to accommodate for undefined pixel values at the image borders, due to the rotation angle correction.

Finally, for the local feature registration operation <NUM>, an elastic image registration, which matches the local features of both sets of images (auto-fluorescence 20b vs. brightfield 48e), by hierarchically matching the corresponding blocks, from large to small. A neural network <NUM> is used to learn the transformation between the roughly matched images. This network <NUM> uses the same structure as the network <NUM> in <FIG>. A low number of iterations is used so that the network <NUM> only learns the accurate color mapping, and not any spatial transformations between the input and label images. The calculated transformation map from this step is finally applied to each brightfield image patch 48e. At the end of these registration steps <NUM>, <NUM>, <NUM>, the auto-fluorescence image patches 20b and their corresponding brightfield tissue image patches 48f are accurately matched to each other and can be used as input and label pairs for the training of the deep neural network <NUM>, allowing the network to solely focus on and learn the problem of virtual histological staining.

For the <NUM>× objective lens images (that were used for generating Table <NUM> and Table <NUM> data) a similar process was used. Instead of down-sampling the auto-fluorescence images <NUM>, the bright-field microscope images <NUM> were down-sampled to <NUM>% of their original size so that they match with the lower magnification images. Furthermore, to create whole slide images using these <NUM>× images, additional shading correction and normalization techniques were applied. Before being fed into the network <NUM>, each field-of-view was normalized by subtracting the mean value across the entire slide and dividing it by the standard deviation between pixel values. This normalizes the network input both within each slide as well as between slides. Finally, shading correction was applied to each image to account for the lower relative intensity measured at the edges of each field-of-view.

In this work, a GAN architecture was used to learn the transformation from a label-free unstained auto-fluorescence input image <NUM> to the corresponding brightfield image <NUM> of the chemically stained sample. A standard convolutional neural network-based training learns to minimize a loss/cost function between the network's output and the target label. Thus, the choice of this loss function <NUM> (<FIG> and <FIG>) is a critical component of the deep network design. For instance, simply choosing an ℓ<NUM> -norm penalty as a cost function will tend to generate blurry results, as the network averages a weighted probability of all the plausible results; therefore, additional regularization terms are generally needed to guide the network to preserve the desired sharp sample features at the network's output. GANs avoid this problem by learning a criterion that aims to accurately classify if the deep network's output image is real or fake (i.e., correct in its virtual staining or wrong). This makes the output images that are inconsistent with the desired labels not to be tolerated, which makes the loss function to be adaptive to the data and the desired task at hand. To achieve this goal, the GAN training procedure involves training of two different networks, as shown in <FIG> and <FIG>: (i) a generator network <NUM>, which in this case aims to learn the statistical transformation between the unstained auto-fluorescence input images <NUM> and the corresponding brightfield images <NUM> of the same samples <NUM>, after the histological staining process; and (ii) a discriminator network <NUM> that learns how to discriminate between a true brightfield image of a stained tissue section and the generator network's output image. Ultimately, the desired result of this training process is a trained deep neural network <NUM>, which transforms an unstained auto-fluorescence input image <NUM> into a digitally stained image <NUM> which will be indistinguishable from the stained brightfield image <NUM> of the same sample <NUM>. For this task, the loss functions <NUM> of the generator <NUM> and discriminator <NUM> were defined as such:
<MAT>
where D refers to the discriminator network output, zlabel denotes the brightfield image of the chemically stained tissue, zoutput denotes the output of the generator network. The generator loss function balances the pixel-wise mean squared error (MSE) of the generator network output image with respect to its label, the total variation (TV) operator of the output image, and the discriminator network prediction of the output image, using the regularization parameters (λ, a) that are empirically set to different values, which accommodate for ~<NUM>% and -<NUM>% of the pixel-wise MSE loss and the combined generator loss (ℓgenerator ), respectively. The TV operator of an image z is defined as:
<MAT>
where p, q are pixel indices. Based on Eq. (<NUM>), the discriminator attempts to minimize the output loss, while maximizing the probability of correctly classifying the real label (i.e., the brightfield image of the chemically stained tissue). Ideally, the discriminator network would aim to achieve D(zlabel) = <NUM> and D(zoutput) = <NUM>, but if the generator is successfully trained by the GAN, D(zoutput) will ideally converge to <NUM>.

The generator deep neural network architecture <NUM> is detailed in <FIG>. An input image <NUM> is processed by the network <NUM> in a multi-scale fashion, using down-sampling and up-sampling paths, helping the network to learn the virtual staining task at various different scales. The down-sampling path consists of four individual steps (four blocks #<NUM>, #<NUM>, #<NUM>, #<NUM>), with each step containing one residual block, each of which maps a feature map xk into feature map xk+<NUM> :
<MAT>
where CONV{. } is the convolution operator (which includes the bias terms), k1, k2, and k3 denote the serial number of the convolution layers, and LReLU[. ] is the non-linear activation function (i.e., a Leaky Rectified Linear Unit) that was used throughout the entire network, defined as:
<MAT>.

The number of the input channels for each level in the down-sampling path was set to: <NUM>, <NUM>, <NUM>, <NUM>, while the number of the output channels in the down-sampling path was set to: <NUM>, <NUM>, <NUM>, <NUM>. To avoid the dimension mismatch for each block, the feature map xk was zero-padded to match the number of the channels in xk+<NUM> The connection between each down-sampling level is a <NUM>×<NUM> average pooling layer with a stride of <NUM> pixels that down-samples the feature maps by a factor of <NUM> (<NUM>-fold for in each direction). Following the output of the fourth down-sampling block, another convolutional layer (CL) maintains the number of the feature maps as <NUM>, before connecting it to the up-sampling path. The up-sampling path consists of four, symmetric, up-sampling steps (#<NUM>, #<NUM>, #<NUM>, #<NUM>), with each step containing one convolutional block. The convolutional block operation, which maps feature map yk into feature map yk+<NUM>, is given by: <MAT>
where CONCAT(. ) is the concatenation between two feature maps which merges the number of channels, US {. } is the up-sampling operator, and k4, k5, and k6, denote the serial number of the convolution layers. The number of the input channels for each level in the up-sampling path was set to <NUM>, <NUM>, <NUM>, <NUM> and the number of the output channels for each level in the up-sampling path was set to <NUM>, <NUM>, <NUM>, <NUM>, respectively. The last layer is a convolutional layer (CL) mapping <NUM> channels into <NUM> channels, represented by the YCbCr color map. Both the generator and the discriminator networks were trained with a patch size of <NUM>×<NUM> pixels.

The discriminator network, summarized in <FIG>, receives three (<NUM>) input channels, corresponding to the YCbCr color space of an input image 40YCbCr, 48YCbCr. This input is then transformed into a <NUM>-channel representation using a convolutional layer, which is followed by <NUM> blocks of the following operator:
<MAT>
where k1, k2, denote the serial number of the convolutional layer. The number of channels for each layer was <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. The next layer was an average pooling layer with a filter size that is equal to the patch size (<NUM>×<NUM>), which results in a vector with <NUM> entries. The output of this average pooling layer is then fed into two fully connected layers (FC) with the following structure:
<MAT>
where FC represents the fully connected layer, with learnable weights and biases. The first fully connected layer outputs a vector with <NUM> entries, while the second one outputs a scalar value. This scalar value is used as an input to a sigmoid activation function D(z) = <NUM>/ (<NUM> + exp(-z)) which calculates the probability (between <NUM> and <NUM>) of the discriminator network input to be real/genuine or fake, i.e., ideally D(zlabel) = <NUM> as illustrated by output <NUM> in <FIG>.

The convolution kernels throughout the GAN were set to be <NUM>×<NUM>. These kernels were randomly initialized by using a truncated normal distribution with a standard deviation of <NUM> and a mean of <NUM>; all the network biases were initialized as <NUM>. The learnable parameters are updated through the training stage of the deep neural network <NUM> by back propagation (illustrated in dashed arrows of <FIG>) using an adaptive moment estimation (Adam) optimizer with learning rate <NUM>×<NUM>-<NUM> for the generator network <NUM> and <NUM>×<NUM>-<NUM> for the discriminator network <NUM>. Also, for each iteration of the discriminator <NUM>, there were <NUM> iterations of the generator network <NUM>, to avoid training stagnation following a potential over-fit of the discriminator network to the labels. A batch size of <NUM> was used in the training.

Once all the fields-of-view have passed through the network <NUM>, the whole slide images are stitched together using the Fiji Grid/Collection stitching plugin (see, e.g., <NPL>), which is incorporated herein by reference). This plugin calculates the exact overlap between each tile and linearly blends them into a single large image. Overall, the inference and stitching took ~<NUM> minutes and <NUM> seconds, respectively, per cm<NUM> and can be substantially improved using hardware and software advancements. Before being shown to the pathologists, sections which are out of focus or have major aberrations (due to e.g., dust particles) in either the auto-fluorescence or bright-field images are cropped out. Finally, the images were exported to the Zoomify format (designed to enable viewing of large images using a standard web browser; http://zoomify. com/) and uploaded to the GIGAmacro website (https://viewer. com/) for easy access and viewing by the pathologists.

The other implementation details, including the number of trained patches, the number of epochs and the training times are shown in Table <NUM> below. The digital/virtual staining deep neural network <NUM> was implemented using Python version <NUM>. The GAN was implemented using TensorFlow framework version <NUM>. Other python libraries used were os, time, tqdm, the Python Imaging Library (PIL), SciPy, glob, ops, sys, and numpy. The software was implemented on a desktop computer with a Core i7-<NUM> CPU @ <NUM> (Intel) and 64GB of RAM, running a Windows <NUM> operating system (Microsoft). The network training and testing were performed using dual GeForce® GTX 1080Ti GPUs (NVidia).

Claim 1:
A method of generating a digitally stained microscopic image (<NUM>) of a label-free sample (<NUM>) comprising:
training a deep neural network (<NUM>) that is executed by image processing software (<NUM>) using one or more processors (<NUM>) of a computing device (<NUM>), wherein the deep neural network (<NUM>) is trained with a plurality of matched pairs of chemically stained images or image patches and their corresponding autofluorescence images or image patches of the same sample and wherein the matched pairs are subject to global registration process (<NUM>) based on extracted feature vectors and their corresponding locations from the matched pairs and a subsequent local registration process (<NUM>, <NUM>) that learns, using a neural network, the color mapping between the globally registered matched pairs followed by local feature registration (<NUM>) which matches local features of the matched pairs;
obtaining one or more autofluorescence images (<NUM>) of the sample (<NUM>) using a fluorescence microscope (<NUM>) and one or more excitation light sources and one or more emission filters, wherein fluorescent light is emitted from endogenous fluorophores or other endogenous emitters of frequency-shifted light within the sample (<NUM>);
inputting the one or more autofluorescence images (<NUM>) of the sample (<NUM>) to the trained, deep neural network (<NUM>); and
the trained, deep neural network (<NUM>) outputting the digitally stained microscopic image (<NUM>) of the sample (<NUM>) that is substantially equivalent to a corresponding brightfield image (<NUM>) of the same sample (<NUM>) that has been chemically stained (<NUM>).