Image analysis device

An image analysis device may include a memory storing learning data for executing image analysis, and may obtain cell image data representing a cell image including a plurality of cell objects, sequentially identify plural pieces of partial image data from the cell image data, sequentially execute a center determination process on each of the plural pieces of partial image data, classify at least one cell corresponding to at least one cell object among a plurality of cell objects by using results of the center determination process on the plural pieces of partial image data and classification data included in learning data, and output a classification result.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Entry under 35 U.S.C. §371 of International Patent Application No. PCT/JP2017/017559, filed May 9, 2017, entitled IMAGE ANALYSIS DEVICE, the entire disclosure of which is hereby incorporated by reference herein in its entirety and for all purposes.

TECHNICAL FIELD

The disclosure herein discloses art that classifies a cell corresponding to a cell object by analyzing an image including the cell object.

BACKGROUND ART

In recent years, histopathologic diagnosis or cytological diagnosis using an image analysis device has been practiced (for example, Japanese Patent Application Publication No. 2011-527055). In these techniques, data for classifying cells is learned by an image analysis device in advance and image data obtained from a histopathologic specimen or a cytological specimen is inputted to the image analysis device, to obtain a cell classification result.

SUMMARY OF INVENTION

Technical Problem Normally in pathological diagnosis assistance or automatic cell analysis using an image analysis device, plural pieces of partial image data are identified sequentially from inputted cell image data. Then, for each of the plural pieces of partial image data, a cell that corresponds to a cell object included in a partial image represented by the partial image data is classified. However, for example, when a cell is located at a position deviated from the center of a partial image, there is a possibility that the cell cannot accurately be classified by using such partial image. The disclosure herein discloses art for improving accuracy of cell classification.

An image analysis device disclosed herein may comprise: a memory that stores learning data for image analysis, the learning data including determination data and classification data, the determination data being for determining whether a center of an analysis target image matches a center of a cell object, the classification data being for classifying a cell corresponding to a cell object; an obtaining unit configured to obtain cell image data representing a cell image including a plurality of cell objects; a first image identifying unit configured to identify plural pieces of partial image data sequentially from the cell image data; a determining unit configured to execute a center determination process for each of the plural pieces of partial image data sequentially, the center determination process including determining, by using the determination data included in the learning data, whether a center of a partial image represented by process target partial image data matches a center of a cell object; a classifying unit configured to classify at least one cell corresponding to at least one cell object among the plurality of cell objects by using a result of the center determination process for each of the plural pieces of partial image data and the classification data included in the learning data; and an output unit configured to output a classification result.

According to the above configuration, the image analysis device executes the center determination process for each of the plural pieces of partial image data by using the determination data. By doing so, the image analysis device can suitably determine whether the center of each partial image matches the center of the corresponding cell object. Further, the image analysis device classifies at least one cell based on the result of the center determination process and the classification data. Due to this, accuracy of cell classification can be improved.

The image analysis device may further comprise a generating unit configured to generate binary image data by binarizing the cell image data. The first image identifying unit may identify the plural pieces of partial image data sequentially by repeating detection and identification, the detection being for detecting a position of a candidate object, which is a cell object candidate, from a binary image represented by the binary image data, the identification being for identifying partial image data corresponding to the detected position from the cell image data. The image analysis device may further comprise a second image identifying unit, wherein in a case where it is determined in the center determination process for first partial image data among the plural pieces of partial image data that a center of a first partial image represented by the first partial image data does not match a center of a cell object and the first partial image includes a target cell object, the second image identifying unit identifies second partial image data that represents a second partial image including the target cell object from the cell image data, wherein a center of the second partial image matches a center of the target cell object. The classifying unit may classify a cell corresponding to the target cell object by using the second partial image data and the classification data. According to this configuration, the image analysis device can classify a cell using the second partial image data. Since the center of the second partial image matches the center of the target cell object, the accuracy of the cell classification is improved.

The image analysis device may further comprise: a position identifying unit configured to identify a center position of the target cell object included in the first partial image in the case where it is determined in the center determination process for the first partial image data that the center of the first partial image does not match the center of the cell object and the first partial image includes the target cell object; and a changing unit configured to change the binary image data by writing a predetermined mark at a position corresponding to the center position in the binary image represented by the binary image data. The second image identifying unit may detect a position of the predetermined mark from a changed binary image represented by the changed binary image data so as to identify the second partial image data corresponding to the position of the predetermined mark from the cell image data. According to this configuration, the image analysis device can be suppressed from classifying the same cell multiple times.

In a case where it is determined that the center of the first partial image does not match the center of the cell object and the first partial image includes only one target cell object, the position identifying unit may identify one center position. The changing unit may change the binary image data by writing one predetermined mark at one position corresponding to the one center position in the binary image represented by the binary image data. The second image identifying unit may detect a position of the one predetermined mark from the changed binary image represented by the changed binary image data so as to identify the second partial image data corresponding to the position of the one predetermined mark from the cell image data. According to this configuration, the image analysis device can suitably identify one piece of the second partial image data when the first partial image includes only one target cell object.

In a case where it is determined that the center of the first partial image does not match the center of the cell object and the first partial image includes two or more target cell objects, the position identifying unit may identify two or more center positions. The changing unit may change the binary image data by writing two or more predetermined marks at two or more positions corresponding to the two or more center positions in the binary image represented by the binary image data. The second image identifying unit may detect each of positions of the two or more predetermined marks from the changed binary image represented by the changed binary image data so as to identify two or more second partial image data corresponding to the positions of the two or more predetermined marks from the cell image data. According to this configuration, the image analysis device can suitably identify two or more second partial image data when the first partial image includes two or more target cell objects.

In a case where the two or more target cell objects include a first cell object having a first center position that is a first distance apart from the center of the first partial image and a second cell object having a second center position that is a second distance apart from the center of the first partial image, the position identifying unit may identify the two or more center positions including the first center position and the second center position. The second distance may be longer than the first distance. The changing unit may change the binary image data by writing the two or more predetermined marks at the two or more positions corresponding to the two or more center positions including the first center position and the second center position in the binary image represented by the binary image data. According to this configuration, the image analysis device can suitably write the predetermined marks even when distances between the center of the first partial image and each of the center positions of the cell objects are not constant.

The first image identifying unit may identify sequentially the plural pieces of partial image data that represent a plurality of partial images obtained by scanning the cell image with a predetermined interval. According to this configuration, the image analysis device can identify the respective pieces of partial image data while suppressing processing load.

In a case where it is determined in the center determination process for process target partial image data among the plural pieces of partial image data that a center of a partial image represented by the process target partial image data matches a center of a cell object, the classifying unit may classify a cell corresponding to the cell object. In a case where it is determined in the center determination process for process target partial image data among the plural pieces of partial image data that a center of a partial image represented by the process target partial image data does not match a center of a cell object, the classifying unit may not execute the classification using the partial image. According to this configuration, the image analysis device does not execute the classification when determining that the center of the partial image does not match the center of the cell object in the center determination process. Thus, when the cell object is located at a position deviated from the center of the partial image, the image analysis device does not classify the cell corresponding to this cell object. Due to this, erroneous classification of cells can be suppressed.

The image analysis device may further comprise: a generating unit configured to generate binary image data by binarizing the cell image data; a position identifying unit, wherein in a case where it is determined in the center determination process for process target partial image data among the plural pieces of partial image data that a center of a partial image represented by the process target partial image data does not match a center of a cell object and the partial image includes a target cell object, the position identifying unit identifies a center position of the target cell object; and a changing unit configured to change the binary image data by writing a predetermined mark at a position corresponding to the center position in the binary image represented by the binary image data. The classifying unit may: detect a position of the predetermined mark from a changed binary image represented by the changed binary image data; identify partial image data corresponding to the detected position from the cell image data; and classify a cell corresponding to the target cell object included in a partial image represented by the partial image data by using the partial image data and the classification data. According to this configuration, the image analysis device can classify the cell by using the partial image data corresponding to the position of the predetermined mark. Here, since the predetermined mark is written at the position corresponding to the center position of the target cell object, the center of the partial image represented by that partial image data can match the center of the target object cell. Due to this, the accuracy of the cell classification is improved.

The learning data may be data for executing image analysis according to a convolutional neural network or image analysis according to a large-scale network including a convolutional neural network as a partial structure thereof. According to this configuration, the image analysis device can execute image analysis using the convolutional neural network according to the learning data, that is, the image analysis that uses deep learning and artificial intelligence. Especially, the larger a volume of the learning data is, the more accurately the image analysis device can execute the image analysis.

The technique disclosed herein can be applied to an image analysis method. Moreover, a control method, a computer program and a non-transitory computer-readable medium storing the computer program for implementing the above image analysis device are also novel and useful.

(Configuration of Image Analysis Device:FIG. 1).FIG. 1shows a configuration of an image analysis device10. The image analysis device10is provided with an operation unit12, a display unit14, an input unit16, and a controller30. The respective units12to30are connected to a bus line (reference sign omitted). The operation unit12is provided with a mouse and a keyboard, for example. A user can provide various instructions to the image analysis device10by operating the operation unit12. The display unit14is a display configured to display various types of information.

The input unit16is a device configured to input cell image data that represents a cell image including a plurality of cell objects to the image analysis device10. The input unit16may be a communication interface configured to execute wired or wireless communication, or a memory interface to which a USB memory or the like is to be inserted. For example, the input unit16may execute wired or wireless communication with a device that stores cell image data captured by a microscope, a Whole Slide Image, or a virtual side, and may receive the cell image data from this device, by which the cell image data may be inputted to the image analysis device10. Further, for example, the input unit16may read out cell image data from a memory storing the cell image data, by which the cell image data may be inputted to the image analysis device10.

The controller30is provided with a CPU32and a memory34. The CPU32is configured to execute various processes according to programs38,40stored in the memory34. The memory34stores an OS program38for realizing basic operations of the image analysis device10and an analysis program40for executing image analysis according to a convolutional neural network (hereinbelow termed “CNN (abbreviation of Convolutional Neural Network)”). For example, the image analysis device10is realized by installing the analysis program40to a general-purpose PC or a server. The analysis program40may execute image analysis according to a large-scale network (such as GoogLeNet (registered trademark), Residual Network) that includes CNN as its partial structure. Further, the memory34stores learning data42for executing image analysis according to the analysis program40. The learning data42may be provided by a vendor who sells the analysis program40, or may be generated by the user of the image analysis device10. In the former case, the learning data42is stored in the memory34upon installation of the analysis program40. In the latter case, the learning data42is stored in the memory34by the user of the image analysis device10after installation of the analysis program40.

The learning data42includes determination data44for determining whether a center of an analysis target image matches a center of a cell object, and classification data46for classifying a cell corresponding to the cell object. The determination data44is data in which image data is associated with a center position of a cell object included in an image represented by the image data for each of a plural pieces of image data. The center position is expressed in a phase (that is, an angle) of the cell object in the present embodiment, however, it may be expressed in coordinates in a variant. The classification data46is data in which image data is associated with a type of cell object included in the image represented by the image data for each of the plural pieces of image data.

(Process of Image Analysis Device10:FIG. 2). Next, a process which the CPU32of the image analysis device10executes according to the analysis program40will be described with reference toFIG. 2. In S10, the CPU32obtains cell image data that represents a cell image100including a plurality of cell objects102,104, via the input unit16. The cell image data is bitmap data constituted of a plurality of pixels having multilevel RGB values (e.g., 256 levels). A file format of the bitmap data is not limited to BMP (Microsoft Windows (registered trademark) Bitmap Image), and may be JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or the like. The cell image data may be generated as described below. For example, bronchioalveolar lavage fluid, which is a specimen collected from a patient, is applied to a slide glass and Giemsa stain is performed thereon to produce a pathological specimen. Then, the pathological specimen is captured by a microscope to generate cell image data. The pathological specimen is not limited to the above. For example, the specimen may be a blood specimen, a biological specimen, or the like, and the stain method may be a Papanicolaou stain method, a hematoxylin and eosin stain method, an immunohistochemistry stain method, or an immunofluorescence stain method. Further, an unstained phase-contrast microscopic image of cultured cells may be used.

In S15, the CPU32binarizes the cell image data obtained in S10. Specifically, for each of the plurality of pixels constituting the cell image data, the CPU32calculates a luminance value from the RGB value of the pixel (e.g., luminance value Y=0.299×R+0.587×G+0.114×B), and then determines that a pixel value of the pixel is “1” in a case where the luminance value is greater than a threshold (such as127), while determines that the pixel value of the pixel is “0” in a case where the luminance value is equal to or less than the threshold. By doing so, the CPU32generates binary image data constituted of a plurality of pixels each having “1” or “0”. Hereinbelow, a pixel having the pixel value “1” and a pixel having the pixel value “0” are respectively termed an “ON pixel” and an “OFF pixel”.FIG. 2shows a binary image110represented by the binary image data, and in the binary image110, the ON pixels are expressed in white and the OFF pixels are expressed in black. As a result, an ON pixel group112indicating a portion of the cell object102that is relatively deeply stained is expressed in white.

In S20, the CPU32executes object detection. Specifically, from the binary image110represented by the binary image data generated in S15, the CPU32detects a center position (that is, coordinates) of one ON pixel group (such as112or114) constituted of a predetermined number of ON pixels or more adjacent to each other as a position of a candidate object which is a candidate of cell object. In doing so, a region dividing method such as a watershed method may be used together.

In S25, the CPU32identifies partial image data that corresponds to the position of the candidate object detected in S20, from the cell image data. Due to this, partial image data of a target of processes of S30and S35to be described later (hereinbelow termed “target partial image data”) is identified. Specifically, the CPU32identifies the target partial image data, which represents a rectangular image having a predetermined size of which center is the position (that is, the coordinates) detected in S20, from the cell image data. In the example ofFIG. 2, the CPU32identifies target partial image data representing a partial image122from the cell image data in a case of having detected the position of the ON pixel group112in S20, while the CPU32identifies target partial image data representing a partial image124from the cell image data in a case of having detected the position of the ON pixel group114in S20.

In S30, the CPU32executes a center determination process on the target partial image data by using the determination data44included in the learning data42. The center determination process includes determining whether a center of a partial image represented by target partial image data (hereinbelow termed a “target partial image”) matches a center of cell object. Specifically, in a case where a phase of cell object (such as 30° inFIG. 1) included in a target partial image can be identified by executing CNN using the determination data44, the CPU32determines that the center of the target partial image does not match the center of the cell object (NO in S30) and proceeds to S35. On the other hand, in a case where a phase of cell object included in a target partial image cannot be identified, the CPU32determines that the center of the target partial image matches the center of the cell object (YES in S30), skips S35and proceeds to S40. For example, since the cell object is located at the center in the partial image122, it is determined that the center of the partial image122matches the center of the cell object (YES in S30). Further, for example, since the cell object is located not at the center but at the upper portion in the partial image124, it is determined that the center of the partial image124does not match the center of the cell object (NO in S30).

In S35, the CPU32writes a predetermined mark in the binary image110. Specifically, the CPU32firstly identifies the phase of cell object, which is the determination result of S30, as a center position of the cell object in the target partial image. Then, the CPU32changes the binary image data by writing the predetermined mark at a position corresponding to the identified center position (that is, the identified phase) in the binary image110. The predetermined mark has a shape that can be detected in object detection of S45to be described later, and in this embodiment, it is a circular mark having a white center portion and a black peripheral portion. In the example ofFIG. 2, the CPU32generates binary image data representing a changed binary image130by writing a predetermined mark132in the binary image110. In a variant, the predetermined mark is not limited to the circular mark, and may have a rectangular shape or another shape.

In S40, the CPU32determines whether the detection (that is, S20) has been completed for all objects within the binary image110. The CPU32proceeds to S45in a case of determining that the detection has been completed for all the objects (YES in S40), while the CPU32returns to S25in a case of determining that the detection has not been completed yet for all the objects (NO in S40).

S45and S50are the same as S20and S25except that the changed binary image data is used. Since the mark132is written in S35as above, the CPU32can identify in S50partial image data representing a partial image150that is a rectangular image having the mark132located at a center thereof. That is, in this partial image data, the center of the partial image150matches the center of the cell object. Since the predetermined mark has the black peripheral portion as described above, it is not integrated with other objects. Due to this, the CPU32can correctly detect the mark132.

In S55, the CPU32executes CNN using the partial image data identified in S50and the classification data46to classify the cell that corresponds to the cell object included in the partial image represented by the partial image data. Since the center of the partial image150matches the center of the cell object as described above, the CPU32can accurately classify the cell corresponding to this cell object.

S60is the same as S40. The CPU32proceeds to S65in a case of determining that the detection has been completed for all the objects (YES in S60), while the CPU32returns to S45in a case of determining that the detection has not been completed yet for all the objects (NO in S60).

In S65, the CPU32outputs a classification result of S55. Specifically, the CPU32causes the display unit14to display the classification result. The classification result may, for example, be information including a number of target cells, information including numbers of classified cells by types thereof, or a score indicating expression of a specific protein in the cell. Further, it may be a grade of malignancy or an expression level of gene mutation predicted from the cell classification result, or a prediction of prognosis of the patient. Results of the center determination process of S30and the classification process of S55may be added to the determination data44and the classification data46. By doing so, accuracy of CNN can be improved.

(Details of Processes of S30and S35inFIG. 2;FIGS. 3 and 4). Next, details of the processes of S30and S35inFIG. 2will be described with reference toFIGS. 3 and 4.FIG. 3shows examples of partial image identified in S25ofFIG. 2.FIG. 4shows positions and shapes of the predetermined mark written in S35according to the center positions of cell objects. Further,FIG. 4also shows the phases indicating the center positions of cell objects.

A partial image302ofFIG. 3Aincludes a cell object corresponding to one macrophage. The center of the partial image302does not match the center of the cell object, and thus 330° is identified in S30as the center position of the cell object. Then, in S35, the position and shape of the predetermined mark corresponding to the phase 330° inFIG. 4Aare identified, and the predetermined mark having the identified shape (that is, the circular mark having the black peripheral portion) is written at a position in the binary image corresponding to the identified position.

A partial image306ofFIG. 3Bincludes a cell object corresponding to one neutrophil. Since neutrophils include segmented nuclei, it had been difficult to correctly identify the centers of neutrophils by conventional image analyses. In the present embodiment, the determination data44includes data in which image data that represents an image including a cell object corresponding to a neutrophil is associated with a center position of this cell object. Due to this, 30° is identified in S30as the center position of the cell object, and the predetermined mark according to the position and shape corresponding to the phase 30° inFIG. 4Ais written in S35. As a result, in the partial image data identified in S50, the center of a partial image represented by this partial image data matches the center of the cell object corresponding to the neutrophil. Due to this, the neutrophil can accurately be classified in S55based on this partial image data.

A partial image310ofFIG. 3Cincludes two cell objects corresponding to two adjacent lymphocytes. Conventional image analyses divide each of such two or more cells by a region dividing method and classify them, however, they may fail to divide the cells correctly. In the present embodiment, the determination data44includes data in which image data that represents an image including two cell objects corresponding to two lymphocytes is associated with center positions of these two cell objects. Due to this, 120° is identified in S30as the center positions, and two predetermined marks according to the position and shape corresponding to phase 120° inFIG. 4Bare written in S35. As a result, the respective positions of the two predetermined marks are detected in S45. Accordingly, partial image data corresponding to the position of one of the predetermined marks is identified in S50(that is, the center of the partial image matches the center of the cell object corresponding to one of the lymphocytes), and the one of the lymphocytes is classified in S55. After this, partial image data corresponding to the position of the other predetermined mark is identified in S50(that is, the center of the partial image matches the center of the cell object corresponding to the other lymphocyte), and the other lymphocytes is classified in S55. As above, even when a partial image identified in S25includes two cell objects, each of the two cells can accurately be classified.

A partial image314ofFIG. 3Dincludes two cell objects corresponding to a lymphocyte and a macrophage adjacent to each other. In the partial image314, a distance L2 between the center of the partial image314and the center of the cell object corresponding to the lymphocyte is larger than a distance L1 between the center of the partial image314and the center of the cell object corresponding to the macrophage, thus the distances from the center of the partial image314to the respective centers of the cell objects are not constant. In the present embodiment, the determination data44includes data in which image data that represents an image including two cell objects corresponding to a lymphocyte and a macrophage is associated with the center positions of these two cell objects. Due to this, 300° is identified in S30as the center positions, and two predetermined marks according to the position and shape corresponding to phase 300° inFIG. 4Care written in S35. In the present embodiment, since a size of the cell object corresponding to the macrophage is larger than a size of the cell object corresponding to the lymphocyte, the predetermined mark corresponding to the former cell object is larger than the predetermined mark corresponding to the latter cell object, as shown inFIG. 4C. However, in a variant, the two predetermined marks may have the same size. In S45, the respective positions of the two predetermined marks are detected. Thus, two pieces of partial image data according to the positions of the two predetermined marks are sequentially identified in S50, and the one lymphocyte and the one macrophage are sequentially classified in S55. As above, even when distances between the center of partial image and the centers of cell objects are different, the respective cells can accurately be classified.

A partial image318ofFIG. 3Eincludes three cell objects corresponding to three lymphocytes. In the present embodiment, the determination data44includes data in which image data that represents an image including three cell objects corresponding to three lymphocytes is associated with the center positions of these three cell objects. Due to this, 0° is identified in S30as the center positions, and three predetermined marks according to the position and shape corresponding to phase 0° inFIG. 4Dare written in S35. As a result, the respective positions of the three predetermined marks are detected in S45. Accordingly, three pieces of partial image data according to the positions of the three predetermined marks are sequentially identified in S50, and the three lymphocytes are sequentially classified in S55. As above, even when a partial image identified in S25includes three cell objects, the three cells can accurately be classified. Although the present embodiment describes examples that the number of cell objects included in a partial image is up to three, however, the determination data44for four or more cell objects may be used in a variant.

(Case A:FIG. 5). Next, specific cases realized by the process ofFIG. 2will be described with reference toFIGS. 5 to 7.FIG. 5shows diagrams for explaining Case A in which cell image data that is obtained from a blood smear prepared by executing Giemsa stain on bone marrow blood is inputted to the image analysis device10.

FIG. 5Ashows a cell image obtained in S10. A sign500shows a cell object corresponding to one neutrophil.FIG. 5Bshows a binary image generated in S15. The image analysis device10uses the binary image ofFIG. 5Bto execute a first round of object detection (S20). In doing so, a region dividing method such as a watershed method is used. As a result, for example, three candidate objects502,504,506are identified sequentially for the cell object500. As a result, the identification of partial image data (S25), the determination that the center of the partial image does not match the center of the cell object (NO in S30), and the writing of marks507,508,509to the binary image (S35) are executed for each of the three candidate objects502,504,506.FIG. 5Cshows a changed binary image obtained by a plurality of marks including the marks507,508,509being written. In this example, positions of the three marks507,508,509overlap. In such a situation, the mark508, which was written last, is brought to the foreground in the present embodiment. In a variant, a mark that is highly likely the center of a cell object may be identified, and this mark may be brought to the foreground.

A sign503inFIG. 5Bis also identified as a candidate object (S20). In this case, in the center determination process for partial image data identified for the candidate object503, it is determined that the partial image does not include a cell object corresponding to a neutrophil which is the analysis target cell of the present case. As a result, NO is determined in S30, however, no mark is written in S35(this case is omitted fromFIG. 2).

The image analysis device10uses the changed binary image ofFIG. 5Cto execute a second round of object detection (S45). As a result, for example, the mark508is detected as a candidate object for the cell object500. As a result, partial image data having a center that matches the center of the cell object500is identified in S50, and the neutrophil corresponding to the cell object500is classified in S55. The image analysis device10executes the same processes to cell objects other than the cell object500, as a result of which a plurality of cells corresponding to a plurality of cell objects510to540can accurately be classified, as shown inFIG. 5D.

As described above, in the case of sequentially identifying the three candidate objects502,504,506in the first round of object detection, the image analysis device10sequentially writes the three marks507,508,509. Here, as a comparative example, it is considered to employ a configuration in which the image analysis device10sequentially identifies three pieces of partial image data corresponding to the positions of the three marks507,508,509from the cell image data, instead of writing the three marks507,508and509, and sequentially classifies the cells by using the three pieces of partial image data. However, according to the configuration of the comparative example, the three pieces of partial image data including the same cell are sequentially identified and thus the same cell is classified three times. Contrary to this, in the present embodiment, the image analysis device10writes the three marks507,508,509such that the mark508is brought to the foreground. Due to this, the image analysis device10can detect the mark508in the second round of object detection without detecting the marks507,509which do not have sufficient areas for object detection, and classify the cell using the partial image data corresponding to the mark508. Due to this, the unnecessity that three pieces of partial image data including the same cell are sequentially identified and the same cell is classified three times can be eliminated. However, in a variant, the configuration of the above-described comparative example may be employed.

(Case B:FIG. 6). Next, specific Case B will be described with reference toFIG. 6. In Case B, cell image data that is obtained from a pathological specimen prepared by executing hematoxylin and eosin stain on a gastric tissue specimen is inputted to the image analysis device10.

FIG. 6Ashows a cell image obtained in S10, and a sign600shows a cell object corresponding to one cancer cell.FIG. 6Bshows a binary image generated in S15. In a cancer cell, its nucleus may not be stained uniformly and only an edge of the nucleus may be stained in a dark tone. Due to this, in the first round of object detection (S20), two candidate objects602,603are identified sequentially for the cell object600. As a result, the image analysis device10writes two marks corresponding to the two candidate objects602,603(S35).FIG. 6Cshows a changed binary image. Although the positions of the aforementioned two marks overlap, a sign is given only to a mark604which was written last. The image analysis device10uses the changed binary image to execute the second round of object detection (S45), identifies partial image data having a center that matches the center of the cell object600(S50), and classifies the cancer cell corresponding to the cell object600(S55). The image analysis device10executes the same processes to cell objects other than the cell object600, as a result of which a plurality of cells corresponding to a plurality of cell objects610to680is classified, as shown inFIG. 6D.

(Case C:FIG. 7). Next, specific Case C will be described with reference toFIG. 7. In Case C, cell image data that is obtained from a pathological specimen prepared by executing immunohistochemistry stain using anti-PD-L1 antibodies on an epithelial tissue specimen is inputted to the image analysis device10. PD-L1 is a protein expressed on a cell membrane and is expressed on macrophages and cancer cells (in particular, squamous cancer cells).

FIG. 7Ashows a cell image obtained in S10, and a sign700shows a cell object corresponding to one squamous cancer cell having PD-L1 expressed on its cell membrane. In the immunohistochemistry stain, when a protein is expressed on a cell membrane, the stained cell membrane may be detected as a candidate object in the object detection.FIG. 7Bshows a binary image generated in S15. Since the cell membrane is stained in the cell object700, two candidate objects702,704are identified sequentially in the first round of object detection (S20). As a result, the image analysis device10writes two marks corresponding to the two candidate objects702,704(S35).FIG. 7Cshows a changed binary image. Although the positions of the aforementioned two marks overlap, a sign is given only to a mark706which was written last. The image analysis device10uses the changed binary image to execute the second round of object detection (S45), identifies partial image data having a center that matches the center of the cell object700(S50), and classifies the squamous cancer cell corresponding to the cell object700(S55).

Cases A to C described above are examples of specific cases, and the image analysis device10may be adapted to leucocyte count in peripheral blood, classification of cells in other celomic fluids (such as pleural fluid, ascites fluid), cervical cytology, aspiration biopsy cytology for thyroid and mammary glands, and the like. Moreover, it may be adapted to classification of unstained cultured cells observed by using a phase-contrast microscope.

(Corresponding Relationships). The process of S10and the process of S15are respectively examples of processes executed by “obtaining unit” and “generating unit”. The processes of S20and S25are an example of processes executed by “first image identifying unit”. The process of S30is an example of processes executed by “determining unit” and “position identifying unit”. The process of S35is an example of a process executed by “changing unit”. The processes of S45and S50are an example of processes executed by “second image identifying unit”. The process of S55and the process of S65are respectively examples of processes executed by “classifying unit” and “output unit”.

The partial image124and the partial image150ofFIG. 2are respectively examples of “first partial image” and “second partial image”. The lymphocyte and the macrophage in the partial image314ofFIG. 3Dare respectively examples of “first cell object” and “second cell object”, and the distance L1 and the distance L2 are respectively examples of “first distance” and “second distance”.

(Second Embodiment;FIG. 8). In the present embodiment, the CPU32executes a process ofFIG. 8, instead of the process ofFIG. 2, according to the analysis program40. S100is the same as S10ofFIG. 2. In S105, the CPU32identifies partial image data representing partial images that are obtained by scanning a cell image represented by the cell image data obtained in S100in a matrix pattern with a predetermined interval. Specifically, the CPU32determines a plurality of coordinates (such as a sign812) in the cell image at the predetermined interval as shown in a cell image810, and identifies the partial image data representing the partial images each of which is a rectangular image of a predetermined size having the coordinates as its center. As above, since the partial image data is identified in the matrix pattern without execution of the object detection from binary image data in the present embodiment, the process of generating binary image and the like can be omitted and processing load can thereby be reduced.

S110is the same as S30ofFIG. 2except that target partial image data which is a target of the center determination process is the partial image data identified in S105. In a case of determining that the center of the target partial image matches the center of a cell object (YES in S110), the CPU32classifies the cell similarly to S55ofFIG. 2. On the other hand, in a case of determining that the center of the target partial image does not match the center of the cell object (NO in S110), the CPU32does not execute S115(that is, does not execute the classification) and proceeds to S120. In S120, the CPU32determines whether the identification of partial image data has been completed for all the determined coordinates in the cell image. The CPU32proceeds to S125in a case of determining that the identification has been completed (YES in S120), while the CPU32returns to S105in a case of determining that the identification has not been completed yet (NO in S120). S125is the same as S65ofFIG. 2.

(Specific Case:FIG. 9). Next, a specific case realized by the process ofFIG. 8will be described with reference toFIG. 9. In the present case, cell image data that is obtained from a blood smear prepared by executing Giemsa stain on bone marrow blood is inputted to the image analysis device10.

FIG. 9Ashows a cell image obtained in S100. The image analysis device10sequentially identifies plural pieces of partial image data (S105). As a result, for example, plural partial images including five partial images900to940ofFIG. 9Bare sequentially identified, and the center determination process is executed for each of the partial images (S110). For each of the four partial images900to930, it is determined that the center of the partial image matches the center of cell object (YES in S110), as a result of which cells corresponding to the cell objects included in the partial images are classified (S115). On the other hand, for the partial image940, it is determined that the center of this partial image does not match the center of cell object (NO in S110), and the classification therefor is not executed. Due to this, erroneous cell classification can be suppressed.

(Corresponding Relationships). The process of S100, the process of S105, the process of S110, the process of S115, and the process of S125are respectively examples of processes executed by “obtaining unit”, “first image identifying unit”, “determining unit”, “classifying unit”, and “output unit”.

(Third Embodiment;FIG. 10). In the present embodiment, the CPU32executes a process ofFIG. 10, instead of the process ofFIG. 2, according to the analysis program40. S200and S202are the same as S10and S15ofFIG. 2. S205is the same as S105ofFIG. 8. Topmost rectangles indicated by broken lines in a cell image1000ofFIG. 10show partial images sequentially identified in S205. S210is the same as S30ofFIG. 2. The CPU32proceeds to S215in a case of determining that the center of a target partial image does not match the center of cell object (NO in S210), while the CPU32skips S215and proceeds to S220in a case of determining that the center of the target partial image matches the center of cell object (YES in S210). S215is the same as S35ofFIG. 2. In this example ofFIG. 10, the CPU32generates changed binary image data representing a changed binary image1020by writing two marks. S220is the same as S120.

S225and S230are the same as S20and S25ofFIG. 2. Lowermost rectangles indicated by broken lines in the cell image1000ofFIG. 10show partial images sequentially identified in S230. S235to S245are the same as S55to S65.

(Specific Case:FIG. 11). Next, a specific case realized by the process ofFIG. 10will be described with reference toFIG. 11. In the present case, cell image data that is the same as that ofFIG. 9is inputted to the image analysis device10.FIG. 11Ais the same asFIG. 9A. When the processes of S205to S220are executed on a cell image ofFIG. 11A, a changed binary image ofFIG. 11Bis obtained. When S225and S230are executed using this changed binary image, partial images900to950corresponding to the positions of marks are identified as shown inFIG. 11C. In each of the partial images900to950, the center of the partial image matches the center of cell object. Due to this, each cell can accurately be classified (S235).

(Corresponding Relationships). The process of S200, the process of S202, and the process of S205are respectively examples of processes executed by “obtaining unit”, “generating unit”, and “first image identifying unit”. The process of S210is an example of a process executed by “determining unit” and “position identifying unit”. The process of S215and the process of S245are respectively examples of processes executed by “changing unit” and “output unit”. The processes of S225to S235are an example of processes executed by “classifying unit”.

Specific examples of the present disclosure have been described in detail, however, these are mere exemplary indications and thus do not limit the scope of the claims. The art described in the claims include modifications and variations of the specific examples presented above.

Moreover, technical features described in the description and the drawings may technically be useful alone or in various combinations, and are not limited to the combinations as originally claimed. Further, the art described in the description and the drawings may concurrently achieve a plurality of aims, and technical significance thereof resides in achieving any one of such aims.

REFERENCE SIGNS LIST