Source: http://www.google.es/patents/US7953264?dq=flatulence
Timestamp: 2013-05-26 04:35:44
Document Index: 207913558

Matched Legal Cases: ['Application No. 60', 'art 200', 'art 200', 'Application No. 200680010148', 'Application No. 200680010148', 'Application No. 06', 'Application No. 06719']

Patente US7953264 - Classifying image features - Google PatentesB�squeda Im�genes Maps Play YouTube Noticias Gmail Drive M�s » B�squeda avanzada de patentes | Historial web | Iniciar sesi�n B�squeda avanzada de patentesPatentesMethods are disclosed for classifying different parts of a sample into respective classes based on an image stack that includes one or more images....http://www.google.es/patents/US7953264?utm_source=gb-gplus-sharePatente US7953264 - Classifying image features N�mero de publicaci�nUS7953264 B2Tipo de publicaci�nConcesi�n N�mero de solicitud12/477,330 Fecha de publicaci�n31 May 2011 Fecha de presentaci�n3 Jun 2009 Fecha de prioridad27 Ene 2005Tambi�n publicado comoCN101151623ACN101151623BCN101916359ACN101916359BEP1846869A1EP1846869B1EP2237189A2US7555155US8280140US20060245631US20090297016US20110255753WO2006081547A1 InventoresKirk William GossageClifford C. HoytRichard Levenson Cesionario originalCambridge Research & Instrumentation, Inc.Cambridge Research And Instrumentation, Inc. Clasificaci�n de EE.UU.382/128600/407356/36 Clasificaci�n internacionalG01N1/00G06K9/00A61B5/05 Clasificaci�n cooperativaG06K9/00127G06K9/6247 Clasificaci�n europeaG06K9/00BG06K9/62B4PReferenciasCitas de patentes (12)Otras citas (13)Enlaces externosUSPTO Cesi�n de USPTO EspacenetClassifying image featuresUS 7953264 B2 Resumen Methods are disclosed for classifying different parts of a sample into respective classes based on an image stack that includes one or more images.
CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. application Ser. No. 11/342,272, filed on Jan. 27, 2006 now U.S. Pat. No. 7,555,155, which claims priority to U.S. Provisional Patent Application No. 60/647,729 entitled �METHOD FOR CLASSIFYING LABELED PATHOLOGY AND CYTOLOGY TISSUE SECTIONS� by Richard Levenson and Clifford C. Hoyt, filed on Jan. 27, 2005. The contents of the prior application are incorporated herein by reference in their entirety.
TECHNICAL FIELD This invention relates to classifying tissue samples.
BACKGROUND Chromogenic staining techniques have been developed empirically to impart visual contrast to various elements within tissue samples. Staining techniques and protocols can produce mixtures of dyes in different tissue elements, and human observers, using microscopes and other imaging devices, have learned to distinguish these staining patterns as typical for particular elements. Modern targeted staining methods, which can be specific to chemical moieties and/or molecular structural arrangements, can produce stained tissues in which two or more chromogenic or fluorescent stains apparently overlap spatially. In fact, the perceived overlap can result because the multiple stains truly are bound within a common structure in the sample, or because, due to the method of preparation, a structure within the sample containing one stain overlaps with a second structure containing a different stain. In either case, it may be difficult to distinguish the presence and relative distribution of the multiple stains and the structures to which they are bound, especially when the stains employed have similar spectral absorption and/or emission characteristics.
SUMMARY In general, in a first aspect, the invention features a method that includes classifying different parts of a sample into respective classes based on an image stack that includes one or more images. For example, the sample can be a tissue section.
DESCRIPTION OF DRAWINGS FIG. 1 is a schematic diagram of a system for acquiring spectral images of a sample, and for classifying the sample.
As used herein, the term �classifying� refers to identifying different regions of an image of a sample that share a set of common characteristics, wherein at least some of the steps in the procedure are performed in an automated fashion by electronic components. The set of common characteristics can include signal strength, shape, spectral and textural features, for example. The identification of such regions in a sample image effectively identifies the corresponding regions in the sample as sharing a set of common features, and more generally that the sample region is of a specific known state or type based on its expression of these features. At least some of the steps in the classification procedure are performed in automated fashion by electronic components. For example, in many embodiments, steps that include spectral unmixing of images, generating composite images, and classifying regions of images into one or more classes are performed by electronic components. However, some operator intervention may occur in other steps. In particular, in some embodiments, steps such as the selection of reference regions corresponding to various classes for training a machine-based classifier may be performed manually by a system operator.
In certain embodiments, spectral images of a sample are �unmixed� into images that each correspond to a spectral index of a respective constituent of the sample. These unmixed images can then by processed by the classifier. The use of the unmixed images as the input into the classifier may improve the efficiency and/or accuracy of the classification.
In certain embodiments, one or more composite images can be generated from spectral images, prior to classification. As explained in more detail later, composite images generally include �flattened� spectral information; that is, composite images contain spectral information that is encoded as variations in a spatial intensity image of a sample. The use of the composite image as an input into the classifier may improve the efficiency and/or accuracy of the classification.
In general, both light conditioning optics 104 and light collecting optics 110 include configurable spectral filter elements. Therefore, spectral resolution can be provided either on the excitation side of sample 108 (e.g., via light conditioning optics 104) or on the emission side of sample 108 (e.g., via light collecting optics 110), or both. In any case, the result of collecting multiple, spectrally resolved images of sample 108 is an �image stack� where each image in the stack is a two-dimensional image of the sample corresponding to a particular wavelength. Conceptually, the set of images can be visualized as forming a three-dimensional matrix, where two of the matrix dimensions are the spatial length and width of each of the images, and the third matrix dimension is the spectral wavelength (emission or excitation) to which the image corresponds. For this reason, the set of spectrally resolved images can be referred to as a �spectral cube� of images. As used herein, a �pixel� in such a set of images (or image stack or spectral cube), refers to a common spatial location for each of the images. Accordingly, a pixel in a set of images includes a value associated with each image at the spatial location corresponding to the pixel.
While each spectral image described above typically refers to a particular wavelength or range of wavelengths (e.g., a spectral band), more generally, each spectral image can correspond to a spectral index that may include one or more wavelength bands, or some more complex spectral distribution. For example, such an image can be generated by using a spectral comb filter. Generally, the image cube will include several spectral images, for example, 10 or more. However, in some embodiments, the image cube may include fewer images, for example, only two or three spectral images. One such example is an red-green-blue (RGB) color image, in which each pixel includes a value associated with the strength of each of the red, green, and blue colors. Such information may be displayed as a single color image, rather than as a set of separate images; however, the information content is the same as that in the set of images, and therefore we use the expression �spectral images� to refer to both cases.
FIG. 2 is a flow chart 200 showing steps involved in classifying a sample. Step 202 includes acquiring a set of one or more images (e.g., a spectral cube) of a sample, as discussed above. Step 204, which is optional, includes spectrally unmixing some or all of the images in the spectral cube to generate an unmixed set of images (i.e., an �unmixed spectral cube�). Spectral unmixing is a technique that quantitatively separates contributions in an image that arise from spectrally different sources. For example, a sample may contain three different types of structures, each labeled with a different dye. The three different dyes may each have different absorption spectra. Typically, the individual absorption spectra of the dyes are known before they are used, or they can be measured. Images of the specimen under illumination will contain, in the most general case, spectral contributions from each of the three dyes. A similar situation arises, for example, in samples containing multiple different fluorescence labels, each of which contribute to measured fluorescence emissions.
Spectral unmixing decomposes one or more images that include contributions from multiple spectral sources into a set of component images (the �unmixed images�) that correspond to contributions from each of the spectral entities within the sample. Thus, if the sample includes three different dyes, each specific to a particular structural entity, then an image of the sample can be separated into three unmixed images, each unmixed image reflecting contributions principally from only one of the dyes.
The functions F and G can be termed the �spectral eigenstates� for the system because they correspond to the pure spectra for the spectral sources in the sample, which are combined in varying proportions to produce the measured spectral images of the sample. Thus, the sample spectrum is a weighted superposition corresponding to separate contributions from the two spectral sources.
In the above discussion, the number of spectral sources is two (i.e., F and G). In general, however, unmixing techniques are not restricted to any particular number of sources. For example, a sample can generally contain m different spectral sources. If the number of wavelengths at which data is collected is n�that is, k=1 . . . n�then matrix E is an n�m matrix instead of an n�2 matrix, as in the above discussion. The unmixing algorithm can then be employed in the same manner as described above to isolate specific contributions at each pixel location in an image from each of the m spectral eigenstates.
θ = cos - 1 ⁡ [ I 1 � I 2  I 1  ⁢  I 2  ] ( 4 ) Sets of spectra for which θ is small for two members are not as easily separated into their components. Physically, the reason for this is easily understood: if two spectra are only marginally different, it is harder to determine the relative abundance of each.
Various data analysis techniques can also be used for determining component spectra for spectral unmixing, such as principal component analysis (PCA), which identifies the most orthogonal spectral eigenvectors from an image cube and yields score images showing the weighting of each eigenvector throughout the image. This may be done in combination with other mathematical processing, and there are other known techniques for identifying low-dimensionality spectral vectors, such as projection pursuit, a technique described, for example, in L. Jimenez and D. Landgrebe, �Hyperspectral Data Analysis and Feature Reduction Via Projection Pursuit�, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, No. 6, pp. 2653-2667, November 1999, the entire contents of which are incorporated herein by reference. Other techniques include independent component analysis (ICA) and end-member detection algorithms, for example.
These techniques are typically not well-suited to the applications in the life sciences. For example, some techniques are optimized for spectral imaging data sets that contain spectra with dense spectral shapes and well-defined narrow peaks. In some techniques the spectral ranges are large compared to the individual spectral features and peaks that are used for analysis. The presence of peaks, or the ratio of peaks may be then used to classify �end-members� to be separated. Unfortunately, the components in biological samples typically do not have such well-defined, narrow peaks.
There are some techniques, sometimes called �convex-hull� algorithms, that estimate what the true end-members are even if they do not exist in a pure form in the image, but the effectiveness is dependent on how close signal spectra in the image cube are to the end-members.
One technique that can be used to extract spectral eigenstates (or representations thereof) without a priori knowledge of all of the eigenstates involves considering the signal spectrum I(λk) for a given pixel, and subtracting from it the maximum amount of a first spectral source F(λk) while leaving the remaining signal that is positive definite in all spectral channels. That is, one defines a so-called �remainder spectrum� Ua(λk) for each pixel as
Additional spectral unmixing techniques are described in PCT Patent Publication No. WO2005/040769 entitled �SPECTRAL IMAGING OF BIOLOGICAL SAMPLES� by Richard Levenson et al., the contents of which are incorporated herein by reference.
In order for the spectral unmixing techniques disclosed herein to effectively separate contributions in sample images that are due to different spectral eigenstates, Equation (1) should be at least approximately correct. That is, the measured spectral data should be approximately described as a linear superposition of weighted eigenstates. This approximation holds for many samples and spectral measurement techniques, especially darkfield measurement techniques. For example, sample images arising from fluorescent or luminescent chemical labels within the sample typically satisfy the linearity assumption. In some cases however, such as for some brightfield measurement techniques, the linearity approximation may not be satisfied. For example, when images are captured that arise from illumination light that is transmitted through a sample that includes light-absorbing components, the linearity assumption in Equation (1) may not be correct. Instead, the intensity of the measured light may be reduced with an exponential dependence on the concentration of the light-absorbing components. In such cases, transformation of the images may first be necessary before unmixing techniques can be used. As an example, for sample images measured in a transmission mode, the measured image intensities can be transformed into optical densities (e.g., by applying a logarithmic function) in order to apply linear unmixing techniques. Optical density techniques are further described, for example, in U.S. application Ser. No. 10/226,592 (Publication No. US 2003/0081204 Al) entitled �SPECTRAL IMAGING� by Paul J. Cronin and Peter J. Miller, filed Aug. 23, 2002, the entire contents of which are incorporated herein by reference.
Application of the unmixing techniques discussed above provides a set of unmixed images from a multi-spectral data set. Returning now to FIG. 2, in a second optional step in flow chart 200, step 206 includes generating one or more composite images using the spectral cube images and/or unmixed spectral cube images. Composite images are generated as a means to �flatten� or compress spectral information into a two-dimensional grayscale image. In other words, in terms of a 3D spectral matrix of image data, generating a composite image corresponds roughly to compressing or packing the information from two or more layers into a single layer. Since both spectral cube and unmixed spectral cube image data can be used, the technique can conceptually include packing multiple layers from different spectral cubes into a single layer.
As an example, consider a 3D spectral cube of images, where each image has width x, height y, and an index k that corresponds to a wavelength λk. If there are a total of N different images in the cube (i.e., data recorded at N different wavelengths) then the spectral cube I can be represented, as described previously, as a matrix I(x,y,k). Compressing spectral information from two or more images in the spectral cube to create a composite image C is equivalent to adding the image layers together. In some embodiments, prior to adding the layers together, each layer is scaled according to a weighting function �(k). The spectral compression operation is then performed according to
C ⁡ ( x , y ) = ∑ k = m n ⁢ f ⁡ ( k ) � I ⁡ ( x , y , k ) ( 7 ) which yields composite image C(x,y) from layers m through n of the spectral image cube. The weighting function �(k) is generally chosen to emphasize different spectral features in the composite image; that is, to create contrast between features arising from the different layers of the spectral cube that contribute to the overall intensity distribution in the composite image.
A wide variety of weighting functions can be chosen in order to produce the desired contrast. In general, in some embodiments, a monotonically increasing or decreasing function is chosen for �(k), such as a linear ramp function or a sigmoidal function. In other embodiments, �(i) can be a dual ramp function (i.e., decreasing to a point and then increasing, or increasing to a point and then decreasing) or another function, such as one or more Gaussian functions. The weight function can generally be selected as desired, and can be applied to a batch series of samples, or can be selected individually for each sample prior to classification. System 100 can include a storage medium to store weighting functions for particular types of samples, so that a weighting function appropriate for a sample undergoing classification can be recalled as needed.
The neural network has one or more input nodes, by which it receives information about the region to be classified. An input is termed a �feature vector,� where each element of the feature vector corresponds to a specific input node of the neural network. The elements of the feature vector are functions of the signal values at one or more pixels in the area being classified. Examples of suitable functions for producing the feature vector are described further below.
FIG. 11 is a schematic diagram showing an example of a neural network that can be used in the classification methods disclosed herein. The network includes an input layer, one hidden layer, and an output layer. Inputs to the neural network are feature vectors �m, and coupling strengths between nodes are given by γk,l values. The outputs from the neural network are the classes associated with an image or image stack.
Typical topological parameters for networks used in the processing of tissue sample images include one hidden layer with 5 nodes, a learning parameter of 0.2, and a momentum factor of 0.5. The structure of neural networks are described, for example, in Christopher M. Bishop, �Neural Networks for Pattern Recognition�, Oxford University Press, 1995.
Referring again to FIG. 2, after selecting the set of images according to which the sample will be classified (the �classification image set�), step 210 includes training the classifier using images from the classification image set.
Many different numerical quantities can be calculated in order to provide a sufficiently distinguishable description of the ROI. For example, in some embodiments, the feature vector corresponding to a selected ROI for a particular class can include 10 different calculations for each of the images in the image stack, thereby resulting in vector with 10Ni elements, where Ni is the number of images in the image stack. The first four of the ten calculations can be texture analysis features obtained from spatial gray level dependency matrices (SGLDMs), which are also referred to as co-occurrence matrices. For example, such matrices are described in R. M. Haralick, K. Shanmugam, and I. Dinstein, �Textural features for image classification�, IEEE Trans. Syst., Man, Cybern., vol. SMC-3, pp. 610-621, 1973. A SGLDM is a spatial histogram of an image (or a portion thereof) that quantifies a distribution of gray scale values within the image. SGLDMs can be calculated, for example, from an estimate of the second-order joint conditional probability densities, sθ(i,j|d,θ). Each value of this conditional probability density represents the probability of a pixel having a gray level value i being d pixels away from a pixel having a gray level value j in a direction described by θ. If an image includes Ng gray levels, then an Ng�Ng matrix sθ(i,j|d,θ) can be created. Optionally, the matrix can be summed over a set of directions θ for a selected distance d. For example, in some embodiments, a single direction θ=0� can be selected. In other embodiments, for example, four directions can be employed: θ=0�, 45�, 90�, and 135�. In general, any number of directions can be selected for analysis of the texture features in a particular ROI.
E = ∑ i = 0 N g - 1 ⁢ ∑ j = 0 N g - 1 ⁢ [ s θ ⁡ ( i , j | d ) ] 2 ⁢ ⁢ S = ∑ i = 0 N g - 1 ⁢ ∑ j = 0 N g - 1 ⁢ s θ ⁡ ( i , j | d ) ⁢ log ⁡ [ s θ ⁡ ( i , j | d ) ] ⁢ ⁢ H = ∑ i = 0 N g - 1 ⁢ ∑ j = 0 N g - 1 ⁢ 1 1 + ( i - j ) 2 ⁢ s θ ⁡ ( i , j | d ) ( 8 ) R = ∑ i = 0 N g - 1 ⁢ ∑ j = 0 N g - 1 ⁢ ( i - j ) 2 ⁢ s θ ⁡ ( i , j | d ) ( 9 ) can be computed as a sum of co-occurrence matrices over the four directions in each ROI. Textural features can then be calculated from each SGLDM. For example, four different textural features that can be calculated from each SGLDM include energy (E), entropy (S), local (10) homogeneity (H), and inertia (R). The inertia value is also referred to as �contrast�. In this example, then, four SGLDM features for the set of angles θ can be calculated as follows for each ROI:
Step 604 includes choosing the number of neural network classification features N�. Initially, the value of N�typically consists of all the features that were calculated, for all image planes, which is the number of elements in the feature vector. Subsequent iterations of the optimization sequence can reduce the value of N� according to the classification performance of the neural network.
In order to assess the relative significance of each of the N� features to the performance of the neural network, the mean feature value μj for each of the j classification features is calculated in step 610. Calculation of a mean feature value can be accomplished, for example, by calculating a mean value of the elements in a feature vector corresponding to a particular class. The elements in the feature vector can be weighted equally or differently in performing the calculation of μj.
In a further step 612, the weighted contribution Wj of each feature j of the N� total features under consideration by the neural network is calculated according to
W j = ∑ k = 1 N f ⁢ μ j ⁢ γ k ( 12 ) where the γk values are the node-to-node coupling constants within the neural network. Using Equation (12), the weighted contributions of each of the features (which generally correspond to classes) can be evaluated. In step 614, classification feature s having the smallest weighted contribution Ws is identified as the �weakest� classification feature and removed from the set of classification features considered by the neural network.
In logic step 620, the classification accuracy score is compared against a selected accuracy threshold. If the accuracy score is higher than the threshold, then the removed feature is deemed to be insignificant enough that it can be permanently removed from consideration by the neural network. The number of neural network classification features N� is reduced by one in step 622 and logical flow returns to step 610, where new mean feature values are calculated for the newly reduced set of classification features in the neural network. In some embodiments, before logical flow returns to step 610, the neural network can be retrained in order to adapt to the smaller number of features. This step is not necessary, but may be employed in some embodiments to improve the accuracy and/or speed of classification.
In some embodiments, the histogram data can be used to �flag� particular regions of the sample according to classification. For example, if the histogram data for a particular pixel includes even a single instance in which the pixel was classified as belonging to a particular class, steps can be taken to ensure that the pixel is positively identified. Warning messages or sounds can be produced, or a sample image having the identified pixels highlighted for easy identification can be displayed. Flagging techniques can be particularly useful when tissue samples are examined for the presence of harmful agents or structures such as pathogens and cancer cells.
EXAMPLES The following examples are intended to be exemplary of the systems and methods disclosed herein, but should not in any way be construed as limiting the scope of the subsequent claims.
Citas de patentes Patente citada Fecha de presentaci�n Fecha de publicaci�n Solicitante T�tuloUS599102825 Mar 199723 Nov 1999Applied Spectral Imaging Ltd.Spectral bio-imaging methods for cell classificationUS642113130 Jun 200016 Jul 2002Cambridge Research & Instrumentation Inc.Birefringent interferometerUS646668711 Feb 199815 Oct 2002The University Of Iowa Research FoundationMethod and apparatus for analyzing CT images to determine the presence of pulmonary tissue pathologyUS67767605 Mar 200317 Ago 2004Alfred E. Mann Institute For Biomedical Engineering At The University Of Southern CaliforniaMulti-mode processing for ultrasonic imagingUS692023931 Mar 200319 Jul 2005Chromavision Medical Systems, Inc.Method and apparatus for automated image analysis of biological specimensUS692489312 May 20032 Ago 2005Marine Biological LaboratoryEnhancing polarized light microscopyUS2003008120423 Ago 20021 May 2003Cambridge Research & Instrumentation, Inc.Spectral imagingUS2003013814024 Ene 200224 Jul 2003Tripath Imaging, Inc.Method for quantitative video-microscopy and associated system and computer software program productUS200302232484 Jun 20024 Dic 2003Cambridge Research & Instrumentation, Inc.Multispectral imaging systemUS2005006544023 Sep 200324 Mar 2005Levenson RichardSpectral imaging of deep tissueWO1998043042A116 Mar 19981 Oct 1998Applied Spectral Imaging Ltd.Spectral bio-imaging methods for cell classificationWO2005040769A223 Sep 20046 May 2005Cambridge Research And Instrumentation, Inc.Spectral imaging of biological samplesOtras citasReferencia1Bishop, Christopher M., "The Multi-Layer Perceptron", Neural Networks for Pattern Recognition (Oxford University Press, 1995), pp. 116-163.2Chinese Office Action for Application No. 200680010148.7 dated Dec. 19, 2008.3Chinese Office Action for Application No. 200680010148.7 dated Jul. 3, 2009.4European Office Action with regard to Application No. 06 719 860.6-1224 dated Nov. 7, 2007.5European Office Action/Summons to Attend Oral Proceedings with regard to Application No. 06719 860.6-1224 / 1846869 dated Feb. 3, 2011.6Haralick, R.M. et al., "Textural features for image classification", IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-3: 610-621 (1973).7International Search Report and Written Opinion for International Application No. PCT/US2006/003197 dated May 22, 2006.8Jimenez, L. et al., "Hyperspectral Data Analysis and Feature Reduction Via Projection in Pursuit", IEEE Transactions on Geoscience and Remote Sensing 37(6): 2653-2667 (1999).9Kenneth P. Camilleri et al., "Spectral Unmixing of Mixed Pixels for Texture Boundary Refinement", IEEE, pp. 1084-1087 (2000).10Paul M. Mather, "Classification 'What is or is not a cow for the public to decide'", Computer Processing of Remotely-Sensed Images: An Introduction, Third Edition, John Wiley & Sons, Ltd. (2004).11Paul M. Mather, "Classification �What is or is not a cow for the public to decide�", Computer Processing of Remotely-Sensed Images: An Introduction, Third Edition, John Wiley & Sons, Ltd. (2004).12Selim Aksoy et al., "Textual Features for Image Database Retrieval", IEEE, pp. 45-49 (1998).13Stephen W. Wharton, "A Contextual Classification Method for Recognizing Land Use Patterns in High Resolution Remotely Sensed Data", Pattern Recognition, vol. 15, No. 4, pp. 317-324 (1982).GirarImagen originalP�gina principal de Google - Sitemap - Descargas masivas de USPTO - Pol�tica de privacidad - Condiciones de servicio - Acerca de Google Patentes - Danos tu opini�nDatos proporcionados por IFI CLAIMS Patent Services©2012 Google