Patent ID: 12185905

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments will be described in detail with reference to the accompanying drawings.

Overall Configuration of Endoscope System

FIG.1illustrates an overview of an overall configuration of an endoscope system9including a medical image processing apparatus according to an embodiment. As illustrated inFIG.1, the endoscope system9includes an endoscope10, which is an electronic endoscope, a light source apparatus11, an endoscope processor apparatus12, a display apparatus13, a medical image processing apparatus14, an operating unit15, and a display16.

The endoscope10captures a time-series medical image and is a flexible endoscope, for example. The endoscope10has an insertion part20, a handheld operating unit21, and a universal cord22. The insertion part20is inserted into a subject and has a distal end and a base end. The handheld operating unit21is disposed continuously with the base end side of the insertion part20and held by a user (physician) to perform various operations. The universal cord22is disposed continuously with the handheld operating unit21.

The insertion part20is entirely formed to have a small diameter and an elongated shape. The insertion part20is constituted by a soft part25, a bending part26, and a distal end part27, which are disposed continuously with each other in this order from the base end side to the distal end side. The soft part25has flexibility. The bending part26is bendable by an operation of the handheld operating unit21. An imaging optical system (objective lens), which is not illustrated, an imaging element28, and the like are incorporated in the distal end part27.

The imaging element28is an imaging element of a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type. Image light of a part to be observed is incident on an imaging surface of the imaging element28through an observation window and the objective lens. The observation window, which is not illustrated, is open on a distal end surface of the distal end part27, and the objective lens, which is not illustrated, is disposed behind the observation window. The imaging element28captures the image light of the part to be observed, which is incident on the imaging surface (converts the image light into an electric signal) and outputs an image signal.

The handheld operating unit21is provided with various operating members to be operated by a user. Specifically, the handheld operating unit21is provided with two types of bending operation knobs29to be used for a bending operation of the bending part26, an air/water supply button30for air supply/water supply operations, and a suction button31for a suction operation. The handheld operating unit21is further provided with a still image pick-up command unit32for issuing a command for capturing a still image39of the part to be observed and a treatment tool introduction port33for inserting a treatment tool (not illustrated) into a treatment tool insertion path (not illustrated) that penetrates through the insertion part20.

The universal cord22is a connection cord for connecting the endoscope10to the light source apparatus11. The universal cord22contains a light guide35that penetrates through the insertion part20, a signal cable36, and a fluid tube (not illustrated). In addition, an end portion of the universal cord22is provided with a connector37athat is connected to the light source apparatus11and a connector37bthat branches off from the connector37aand is connected to the endoscope processor apparatus12.

Since the connector37ais connected to the light source apparatus11, the light guide35and the fluid tube (not illustrated) are inserted into the light source apparatus11. Thus, through the light guide35and the fluid tube (not illustrated), necessary illumination light, water, and gas are supplied from the light source apparatus11to the endoscope10. As a result, the part to be observed is irradiated with the illumination light from an illumination window (not illustrated) on the distal end surface of the distal end part27. In accordance with a pressing operation on the above-described air/water supply button30, the gas or water is injected from an air/water supply nozzle (not illustrated) on the distal end surface of the distal end part27to the observation window (not illustrated) on the distal end surface.

Since the connector37bis connected to the endoscope processor apparatus12, the signal cable36is electrically connected to the endoscope processor apparatus12. Thus, through the signal cable36, an image signal of the part to be observed is output from the imaging element28of the endoscope10to the endoscope processor apparatus12, and also, a control signal is output from the endoscope processor apparatus12to the endoscope10.

The light source apparatus11supplies the illumination light through the connector37ato the light guide35of the endoscope10. As the illumination light, light in various wavelength ranges in accordance with an observation purpose, such as white light (light in a white wavelength range or light in a plurality of wavelength ranges), light in one or more specific wavelength ranges, or a combination thereof is selected. Note that the specific wavelength range is narrower than the white wavelength range.

A first example of the specific wavelength range is, for example, a blue range or a green range in a visible range. The wavelength range of the first example includes a wavelength range of greater than or equal to 390 nm and less than or equal to 450 nm or greater than or equal to 530 nm and less than or equal to 550 nm, and light of the first example has a peak wavelength in the wavelength range of greater than or equal to 390 nm and less than or equal to 450 nm or greater than or equal to 530 nm and less than or equal to 550 nm.

A second example of the specific wavelength range is, for example, a red range in a visible range. The wavelength range of the second example includes a wavelength range of greater than or equal to 585 nm and less than or equal to 615 nm or greater than or equal to 610 nm and less than or equal to 730 nm, and light of the second example has a peak wavelength in the wavelength range of greater than or equal to 585 nm and less than or equal to 615 nm or greater than or equal to 610 nm and less than or equal to 730 nm.

A third example of the specific wavelength range includes a wavelength range in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients, and light of the third example has a peak wavelength in the wavelength range in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. The wavelength range of the third example includes a wavelength range of 400±10 nm, 440±10 nm, 470±10 nm, or greater than or equal to 600 nm and less than or equal to 750 nm, and light of the third example has a peak wavelength in the wavelength range of 400±10 nm, 440±10 nm, 470±10 nm, or greater than or equal to 600 nm and less than or equal to 750 nm.

A fourth example of the specific wavelength range is the wavelength range (from 390 nm to 470 nm) of excitation light that is used for observing fluorescence (fluorescence observation) emitted by a fluorescent material in a living body and that excites the fluorescent material.

A fifth example of the specific wavelength range is the wavelength range of infrared light. The wavelength range of the fifth example includes a wavelength range of greater than or equal to 790 nm and less than or equal to 820 nm or greater than or equal to 905 nm and less than or equal to 970 nm, and light of the fifth example has a peak wavelength in the wavelength range of greater than or equal to 790 nm and less than or equal to 820 nm or greater than or equal to 905 nm and less than or equal to 970 nm.

The endoscope processor apparatus12controls operations of the endoscope10through the connector37band the signal cable36. In addition, based on the image signal acquired from the imaging element28of the endoscope10through the connector37band the signal cable36, the endoscope processor apparatus12generates a moving image38that is a time-series medical image formed of time-series frame images38a(seeFIG.2) including a photographic subject image. The frame rate of the moving image38is, for example, 30 fps (frame per second).

Furthermore, if the still image pick-up command unit32is operated in the handheld operating unit21of the endoscope10, concurrently with the generation of the moving image38, the endoscope processor apparatus12acquires one frame image38ain the moving image38as the still image39in accordance with the timing of an imaging command.

The moving image38and the still image39are medical images obtained by imaging the inside of the subject, that is, a living body. In addition, if the moving image38and the still image39are images obtained with the above-described light in the specific wavelength range (special light), both are special light images. In addition, the endoscope processor apparatus12outputs the generated moving image38and the still image39to each of the display apparatus13and the medical image processing apparatus14.

Note that the endoscope processor apparatus12may generate (acquire) the special light image having information on the above-described specific wavelength range, based on a usual light image obtained with the above-described white light. In this case, the endoscope processor apparatus12functions as a special light image acquiring unit. Then, the endoscope processor apparatus12obtains a signal in the specific wavelength range by performing calculation based on RGB color information of red, green, and blue or CMY color information of cyan, magenta, and yellow included in the usual light image.

Based on, for example, at least one of the usual light image obtained with the above-described white light or the special light image obtained with the above-described light in the specific wavelength range (special light), the endoscope processor apparatus12may generate a feature quantity image such as a known oxygen saturation image. In this case, the endoscope processor apparatus12functions as a feature quantity image generating unit. Note that each of the moving image38and the still image39including the above-described in-living-body image, the usual light image, the special light image, and the feature quantity image is a medical image obtained by converting results of imaging or measuring of a human body into an image for the purpose of image diagnosis or inspection.

The display apparatus13is connected to the endoscope processor apparatus12and displays the moving image38and the still image39input from the endoscope processor apparatus12. A user operates the insertion part20back and forth, for example, while viewing the moving image38displayed on the display apparatus13, and, if a lesion or the like is found at the part to be observed, the user operates the still image pick-up command unit32to capture a still image of the part to be observed for diagnosis, biopsy, or the like.

Configuration of Medical Image Processing Apparatus

The medical image processing apparatus14reports a region of interest included in a time-series medical image to a user, and, for example, a personal computer is used as the medical image processing apparatus14in this embodiment. In addition, a keyboard, a mouse, or the like connected to the personal computer via wired or wireless connection is used as the operating unit15, and any monitor, such as a liquid crystal monitor that can be connected to the personal computer, is used as the display16(example of a display unit).

FIG.2is a block diagram illustrating an electric configuration of the medical image processing apparatus14. The medical image processing apparatus14illustrated inFIG.2is mainly constituted by a time-series image acquiring unit40, a region-of-interest detecting unit41, a region-of-interest information acquiring unit42, a coordinates calculating unit43, a total-time measuring unit44, an emphasis-degree setting unit45, an identical-region-of-interest determining unit46, a control unit47, a display control unit48, and a storage unit49.

Based on a program (medical image processing program)51stored in the storage unit49, the control unit47generally controls the time-series image acquiring unit40, the region-of-interest detecting unit41, the region-of-interest information acquiring unit42, the coordinates calculating unit43, the total-time measuring unit44, the emphasis-degree setting unit45, the identical-region-of-interest determining unit46, and the display control unit48and functions as part of these units.

The storage unit49is a part that stores detection results obtained by the region-of-interest detecting unit41and stores a captured still image39, and also stores information or the like related to various controls of a region-of-interest storage unit50that stores a feature quantity and a total time of the region of interest in association with each other, the program51, and the medical image processing apparatus14.

The time-series image acquiring unit40acquires, from the endoscope processor apparatus12(FIG.1), the moving image38(moving image38captured by the endoscope10in this example), formed of the time-series frame images38aincluding a photographic subject image, by using an image input/output interface, which is not illustrated, connected to the endoscope processor apparatus12via wired or wireless connection. In addition, if the above-described still image39is captured while the moving image38is being captured by the endoscope10, the time-series image acquiring unit40acquires the moving image38and the still image39from the endoscope processor apparatus12.

Note that, instead of directly acquiring the moving image38from the endoscope processor apparatus12, the time-series image acquiring unit40may acquire the moving image38via any information storage medium, such as a memory card or a hard disk apparatus. In addition, the time-series image acquiring unit40may acquire, via the Internet, the moving image38uploaded on a server, database, or the like on the Internet.

The region-of-interest detecting unit41is an image processing unit that detects the region of interest from the moving image38captured during observation of the inside of the subject. The region-of-interest detecting unit41calculates a feature quantity of the frame images38a(or the frame images38adecimated at certain intervals) of the moving image38and includes a convolutional neural network (CNN) that performs recognition processing of the region of interest within an image.

As examples of the region of interest, there are a polyp, cancer, a colon diverticulum, inflammation, an endoscopic mucosal resection (EMR) scar, an endoscopic submucosal dissection (ESD) scar, a clipped part, a bleeding point, perforation, an atypical vessel, a treatment tool, and the like.

The region-of-interest detecting unit41can further acquire a recognition result of, for example, category classification as to whether the detected region of interest belongs to which of a plurality of categories about the lesion, such as “tumorous”, “non-tumorous”, and “others”.

Note that the region-of-interest detecting unit41is not limited to the one that detects the region of interest by the CNN, but may detect the region of interest by analyzing a feature quantity such as the color, pixel value gradient, shape, or size in the image through image processing.

The region-of-interest information acquiring unit42(an example of a feature quantity calculating unit) acquires region-of-interest information of the region of interest detected by the region-of-interest detecting unit41. The region-of-interest information can be, for example, information of coordinates of a contour of the region of interest in the image and a feature quantity of the region of interest. The coordinates information may be included in the feature quantity.

The coordinates calculating unit43calculates coordinates information indicating the position of the region of interest in the image detected by the region-of-interest detecting unit41. The coordinates calculating unit43calculates, for example, one or more pieces of coordinates information on the contour of a polygon or a circle that surrounds a region of interest. As the coordinates information, coordinates of vertexes of the polygon or coordinates of midpoints of sides of the polygon may be calculated, or coordinates of points at which a circumference of the circle is equally divided into a plurality of parts may be calculated.

The total-time measuring unit44measures, as a total time, a total detection time during which the region of interest is detected by the region-of-interest detecting unit41. The emphasis-degree setting unit45sets an emphasis degree to a relatively larger value as the total time is relatively longer. For the emphasis degree, a minimum, a maximum, and an increasing rate of the emphasis degree with respect to the total time are determined in advance. Thus, from the total time, the emphasis degree can be calculated. The emphasis-degree setting unit45calculates and sets the emphasis degree in accordance with the region of interest from the total time measured by the total-time measuring unit44.

If a plurality of regions of interest are detected by the region-of-interest detecting unit41, the total-time measuring unit44measures the total time of each of the regions of interest, and the emphasis-degree setting unit45sets the emphasis degree of the region of interest in accordance with the total time of the region of interest.

Note that the region of interest is emphasized at the emphasis degree set by the emphasis-degree setting unit45, as described later. The total-time measuring unit44may measure the total time during which the region of interest is emphasized.

The identical-region-of-interest determining unit46determines whether the region of interest included in a medical image in the moving image38is identical with a region of interest that is emphasized in a previous medical image, which precedes the above medical image in time series. The identical-region-of-interest determining unit46may also determine whether the region of interest included in a medical image in the moving image38is identical with a region of interest that is detected by the region-of-interest detecting unit41in a previous medical image, which precedes the above medical image in time series. This determination is performed by using the feature quantity of the region of interest acquired by the region-of-interest information acquiring unit42.

The display control unit48includes an image display control unit48A and a reporting information display control unit48B. The image display control unit48A outputs the moving image38acquired by the time-series image acquiring unit40to the display16and causes the display16to display the moving image38. That is, the display16sequentially displays a plurality of frame images38a.

Based on the coordinates information calculated by the coordinates calculating unit43, and based on the emphasis degree set by the emphasis-degree setting unit45, the reporting information display control unit48B outputs reporting information for reporting the region of interest to the display16. Herein, as the reporting information, a figure indicating the position of the region of interest is superposed at the position of the region of interest. Thus, at the position of the region of interest in the moving image38displayed on the display16, the figure in accordance with the total time of the region of interest is superposed. Thus, the region of interest is emphasized at the set emphasis degree by the superposed figure.

In this manner, the medical image processing apparatus14and the display16function as a diagnosis supporting apparatus.

Medical Image Processing Method: First Embodiment

A medical image processing method using the endoscope system9will be described. The medical image processing method is performed by the control unit47executing the program51stored in the storage unit49.

FIG.3is a flowchart illustrating an example of each process of the medical image processing method according to a first embodiment. The medical image processing method includes an image acquisition step (step S1), a region-of-interest detection step (step S2), an identical region-of-interest determining step (step S3), a total-time measuring step (step S4), an emphasis-degree setting step (step S5), and an emphasis processing step (step S6).

In step S1, the time-series image acquiring unit40acquires a frame image38ain a moving image38captured by the endoscope10. This frame image38ais set as a first medical image.

In step S2, the region-of-interest detecting unit41detects a region of interest from the first medical image acquired in step S1. The region of interest detected from the first medical image is set as a first region of interest.

In step S3, the region-of-interest information acquiring unit42acquires a feature quantity of the first region of interest from the first medical image. In addition, the region-of-interest information acquiring unit42reads, from the region-of-interest storage unit50, a feature quantity of a second region of interest emphasized (detected) in a second medical image, which is a frame image38aone frame before the first medical image in the moving image38.

Furthermore, the identical-region-of-interest determining unit46determines whether the first region of interest is identical with the second region of interest from a similarity degree between the feature quantity of the first region of interest and the feature quantity of the second region of interest.

In step S4, the total-time measuring unit44measures a total time of the first region of interest. The total time is a time that is started to be measured from the timing the region of interest is emphasized. The total time may also be a time that is started to be measured from the timing the region of interest is detected. The total time may be counted as the number of frames of a time-series image from the timing the region of interest is emphasized or the timing the region of interest is detected.

If it is determined in step S3that the first region of interest is not identical with the second region of interest, the total-time measuring unit44determines that the first region of interest is a newly detected region of interest and newly starts to measure the total time.

On the other hand, if it is determined in step S3that the first region of interest is identical with the second region of interest, the total-time measuring unit44measures the total time of the first region of interest by taking over the total time of the second region of interest. That is, the total-time measuring unit44reads the total time of the second region of interest from the region-of-interest storage unit50and adds, to the read total time, the time that equals to one frame from the second medical image to the first medical image.

In step S5, the emphasis-degree setting unit45sets an emphasis degree of the first region of interest.

If it is determined in step S3that the first region of interest is not identical with the second region of interest, since measurement of the total time has only newly started, the emphasis-degree setting unit45sets the emphasis degree of the first region of interest to a predetermined minimum.

On the other hand, if it is determined in step S3that the first region of interest is identical with the second region of interest, the emphasis-degree setting unit45sets the emphasis degree of the first region of interest in accordance with the total time that is measured by taking over the total time of the second region of interest.

Note that the emphasis-degree setting unit45may read an emphasis degree of the second region of interest from the region-of-interest storage unit50and may calculate the emphasis degree of the first region of interest from the read emphasis degree of the second region of interest, an elapsed time from the second medical image to the first medical image, and a predetermined increasing rate of the emphasis degree.

Lastly, in step S6, the display control unit48(an example of an emphasis processing unit) emphasizes the first region of interest included in the first medical image at the set emphasis degree. Herein, the image display control unit48A outputs the first medical image to the display16and causes the display16to display the first medical image. In addition, based on coordinates information calculated by the coordinates calculating unit43, and based on the emphasis degree set by the emphasis-degree setting unit45, the reporting information display control unit48B outputs reporting information for reporting the first region of interest to the display16. Thus, a figure in accordance with the total time of the first region of interest is superposed at the position of the first region of interest in the first medical image. Accordingly, the first region of interest is emphasized at the set emphasis degree by using the superposed figure.

In addition, in step S6, the control unit47stores the feature quantity, the total time, and the emphasis degree of the first region of interest in the region-of-interest storage unit50in association with the first region of interest.

In the above manner, according to the medical image processing method, the region of interest in the medical image can be appropriately reported.

Emphasis Processing

FIG.4illustrates an example of emphasis processing performed by the display control unit48. F4A illustrated inFIG.4illustrates a frame image38ain which a region of interest R1is detected. In addition, F4B illustrated inFIG.4illustrates a frame image38ain which a frame-shapedfigure E1that surrounds the region of interest R1is superposed on the frame image38aillustrated in F4A at the position of the region of interest R1. In this manner, by superposing the figure at the position of the region of interest R1, the display control unit48can emphasize the region of interest R1.

The shape of the frame-shaped figure to be superposed on the image is not limited to a rectangle but may also be a circle, an ellipse, or a polygon. In addition, the figure to be superposed on the image may also be an arrow indicating the region of interest or the like.

In addition,FIG.5illustrates another example of emphasis processing. F5A illustrated inFIG.5illustrates a frame image38ain which a region of interest R2is detected. In addition, F5B illustrated inFIG.5illustrates a frame image38ain which afigure E11is superposed on the frame image38aillustrated in F5A at the position of the region of interest R2. Thefigure E11has substantially the same shape as the region of interest R2, and brightness thereof as a pixel value is different from that of the region of interest R2. In the above manner, the display control unit48may superpose the figure for which the pixel value is changed at the position of the region of interest.

Processing for making the pixel value of the figure to be superposed at the position of the region of interest differ from the pixel value of the region of interest is not limited to processing for changing the brightness but may also be hue changing processing, saturation changing processing, contrast processing, negative/positive reversing processing, filtering processing, frequency processing, and the like. These kinds of processing are preferably processing by which the region of interest becomes more outstanding than a region outside the region of interest.

Increasing Rate of Emphasis Degree

FIG.6is a graph illustrating an example of a relationship between a total time of a region of interest and an emphasis degree to be set, in which the horizontal axis represents time, and the vertical axis represents emphasis degree.FIG.6illustrates five examples in F6A to F6E.

In F6A to F6E, a time t1is a timing at which the region of interest is detected. As illustrated in F6A to F6E, the emphasis degree is set to a minimum at the time t1and is set to a maximum at a time t2, which is a timing after the time t1. In addition, the emphasis degree is set to a relatively larger value as the total time from the time t1until the time t2is relatively longer. Note that the frame image38ais displayed as it is if the emphasis degree is 0.

The emphasis degree may be increased in proportion to the total time as illustrated in F6A or may be increased exponentially with respect to the total time as illustrated in F6B. Alternatively, the total time may be increased in a stepwise manner as illustrated in F6C. That is, the increasing rate of the emphasis degree to the total time (hereinafter referred to as increasing rate) is not limited to a fixed value but may be a value obtained by a function with respect to the total time.

The minimum, the maximum, and the increasing rate of the emphasis degree are stored in the storage unit49as predetermined fixed values. In addition, the minimum, the maximum, and the increasing rate of the emphasis degree may be changed as appropriate by using the feature quantity obtained from an image. Alternatively, a user may set desired values.

After the emphasis degree reaches the maximum, as illustrated in F6A to F6C, the maximum may be maintained as it is. Until a fixed time elapses, by placing emphasis by setting the emphasis degree at the maximum, a user is likely to recognize the region of interest.

Alternatively, as illustrated in F6D, after the emphasis degree reaches the maximum, the emphasis degree may be repeatedly increased again from the minimum. In this case, a blinking portion where the emphasis degree is repeatedly increased is likely to attract attention, and thus, a user is prevented from missing the region of interest.

Further alternatively, as illustrated in F6E, after the emphasis degree reaches the maximum, the emphasis processing may end, that is, the emphasis degree may be set to 0. In this case, the emphasis processing can be prevented from being performed too long, which may interrupt observation of the region of interest by a user.

These manners after the emphasis degree reaches the maximum may be selected as appropriate in accordance with the region of interest. Alternatively, a user may set a desired manner

Change of Reporting Information

FIG.7illustrates an example of change of a figure in accordance with an emphasis degree. F7A illustrated inFIG.7illustrates a frame image38aon which afigure E21is superposed. Thefigure E21is a frame-shaped figure that surrounds a region of interest R3and has a minimum emphasis degree. In addition, F7B illustrated inFIG.7illustrates a frame image38aon which afigure E22is superposed. Thefigure E22is a frame-shaped figure that surrounds the region of interest R3and has a maximum emphasis degree. The frame image38aillustrated in F7B is a frame image subsequent to the frame image38aillustrated in F7A in time series.

The figure E21and the figure E22are figures having the same line thickness and color and have different transmittances. Herein, by setting the transmittance of the figure to a relatively lower transmittance as the total time is relatively longer, the emphasis-degree setting unit45sets the emphasis degree to a relatively larger value as the total time is relatively longer. Thus, the transmittance of the figure E22is relatively lower than the transmittance of the figure E21.

FIG.8illustrates another example of change of a figure in accordance with an emphasis degree. F8A illustrated inFIG.8illustrates a frame image38aon which afigure E31is superposed. Thefigure E31is a frame-shaped figure that surrounds a region of interest R4and has a minimum emphasis degree. In addition, F8B illustrated inFIG.8illustrates a frame image38aon which afigure E32is superposed. Thefigure E32is a frame-shaped figure that surrounds the region of interest R4and has a maximum emphasis degree. The frame image38aillustrated in F8B is a frame image subsequent to the frame image38aillustrated in F8A in time series.

The figure E31and the figure E32are figures having the same transmittance and color and have different line thicknesses. Herein, by setting the line thickness of the figure to be relatively thicker as the total time is relatively longer, the emphasis-degree setting unit45sets the emphasis degree to a relatively larger value as the total time is relatively longer. Thus, the line of the figure E32is relatively thicker than the line of the figure E31.

FIG.9illustrates another example of change of a figure in accordance with an emphasis degree. F9A illustrated inFIG.9illustrates a frame image38aon which afigure E41is superposed. Thefigure E41is a frame-shaped figure that surrounds a region of interest R5and has a minimum emphasis degree. In addition, F9B illustrated inFIG.9illustrates a frame image38aon which afigure E42is superposed. Thefigure E42is a frame-shaped figure that surrounds the region of interest R5and has a maximum emphasis degree. The frame image38aillustrated in F9B is a frame image subsequent to the frame image38aillustrated in F9A in time series.

The figure E41and the figure E42are figures having the same transmittance and line thickness and have different colors (color temperatures). Herein, by setting the color of the figure to a color of a higher color intensity based on a given color intensity index as the total time is relatively longer, the emphasis-degree setting unit45sets the emphasis degree to a relatively larger value as the total time is relatively longer. Thus, the color of the figure E42has a relatively higher color intensity based on a given color intensity index than the color of the figure E41.

F9C illustrated inFIG.9illustrates transition from the color of thefigure E41to the color of thefigure E42. The intermediate color between the color of thefigure E41and the color of thefigure E42may be calculated by changing the equal ratio, equal difference, or ratio of the colors.

The emphasis-degree setting unit45may set the emphasis degree to a relatively larger value as the total time is relatively longer by acquiring the color of the frame image38aand setting the color of the figure to a color having a relatively higher contrast to the color of the frame image38aas the total time is relatively longer. The color of the frame image38adiffers depending on the color of the inside of the subject and the wavelength of illumination light supplied from the light source apparatus11. Thus, in accordance with the acquired color of the frame image38a, the color of the figure E41and the color of the figure E42are determined as appropriate.

Alternatively, instead of the color of the figure, at least one of the brightness or the saturation of the figure may be changed. The change preferably makes the region of interest seem more outstanding as the total time is longer. In this manner, by changing at least one of the color, the brightness, or the saturation of the figure, emphasis processing can be performed by taking into account the color difference from the periphery, the light source, and the like.

In addition, at least one of the size or the shape of the figure may be changed. Also in this case, the change preferably makes the region of interest seem more outstanding as the total time is longer.

Total Time

The total time measured by the total-time measuring unit44indicates, for example, the time during which a region of interest is continuously detected from the initial detection timing.

FIG.10is a drawing for explaining the total time.FIG.10illustrates examples of frame images38aat time t−1, time t, time t+1, time t+2, and time t+3. Herein, no region of interest is detected in the frame image38aat time t−1. In addition, an identical region of interest R6is detected in the frame images38aat time t, time t+1, and time t+2. In accordance with this, afigure E51, afigure E52, and afigure E53having a frame shape that surrounds the region of interest R6are superposed on the frame images38aat time t, time t+1, and time t+2, respectively.

In the examples illustrated inFIG.10, the total time of the region of interest R6in the frame image38aat time t is 0. The total time of the region of interest R6in the frame image38aat time t+1 is (t+1)−t=1, and the total time of the region of interest R6in the frame image38aat time t+2 is (t+2)−t=2.

Thus, the emphasis degree (minimum) in accordance with the total time0is set for the figure E51, the emphasis degree in accordance with the total time1is set for the figure E52, and the emphasis degree in accordance with the total time2is set for the figure E53.

In addition, as illustrated inFIG.10, no region of interest is detected in the frame image38aat time t+3. Thus, at time t+3, the total time of the region of interest R6is reset to 0, and emphasis processing is reset.

Note that the total time and the emphasis processing are not limited to being reset immediately upon stopping of detection of the region of interest.

In the moving image38captured by the endoscope10, in some cases, the region of interest is hidden from folds of intestines or the like and is not captured in the image at a certain timing. If the region of interest that has been hidden is found again, the newly found region of interest may be determined as another region of interest, and the emphasis degree may be reset, in which case a user may not pay attention to the region of interest.

In addition, the endoscope10is difficult to operate, and it is difficult to make the region of interest remain constantly in the observation image. Thus, if support is provided by recognizing the region of interest as a different region of interest every time the region of interest becomes out of the screen, a user's thought may be interrupted.

Furthermore, if the observation window (not illustrated) on the distal end surface of the distal end part27is soiled, for example, the region-of-interest detecting unit41cannot detect the region of interest in some cases.

Thus, even if the detection is stopped, information on the region of interest is held for a certain time Tk, and, upon detection of the identical region of interest later, the emphasis degree may be taken over.

FIG.11is a drawing for explaining the total time.FIG.11illustrates examples of frame images38aat time t, time t+1, time t+2, and time t+3. As illustrated inFIG.11, a region of interest R7is detected in the frame image38aat time t. In accordance with this, afigure E61having a frame shape that surrounds the region of interest R7is superposed on the frame image38aat time t. At time t, the total time of the region of interest R7is Tc. Thus, thefigure E61is a figure having an emphasis degree in accordance with the total time Tc. The control unit47stores the feature quantity, the total time, and the emphasis degree of the region of interest R7are stored in the region-of-interest storage unit50in association with the region of interest R7.

Subsequently, as illustrated inFIG.11, no region of interest is detected in the frame images38aat time t+1 and time t+2.

When the time further elapses, as illustrated inFIG.11, the region-of-interest detecting unit41detects a region of interest in the frame image38aat time t+3. Herein, an elapsed time Tp=(t+3)−t=3 from the time t at the last detection of the region of interest R7until time t+3 is less than the certain time Tk.

The identical-region-of-interest determining unit46compares the feature quantity of the region of interest R7stored in the region-of-interest storage unit50and the feature quantity of the region of interest detected in the frame image38aat time t+3 and determines whether the region of interest is identical. Herein, it is determined that the region of interest is identical.

For the region of interest R7detected in the frame image38aat time t+3, the emphasis-degree setting unit45takes over the total time and the emphasis degree of the region of interest R7stored in the region-of-interest storage unit50and adds, to the total time, the elapsed time Tp from time t until time t+3. That is, as the total time, time measurement starts at the timing of emphasis of the region of interest, and the measured time includes the time during which the region of interest is not emphasized.

Thus, the total time of the region of interest R7is Tc+Tp, and the emphasis degree is an emphasis degree in accordance with the total time Tc+Tp. As illustrated inFIG.11, afigure E62is superposed on the frame image38aat time t+3. Thefigure E62is a frame-shaped figure that surrounds the region of interest R7and has an emphasis degree in accordance with Tc+Tp.

If the time during which no region of interest is detected exceeds the certain time Tk, the feature quantity, the total time, and the emphasis degree of the region of interest stored in the region-of-interest storage unit50are reset (deleted). The certain time Tk during which information on the region of interest is continuously held may be a predetermined fixed value or may be determined from the amount of unevenness between the detected region of interest and its periphery, the color difference between the detected region of interest and its periphery, or the like. Alternatively, a user may set a desired value.

Herein, the elapsed time Tp added to the total time Tc that is taken over includes the time during which detection of the region of interest is stopped. However, only the time during which the region of interest is actually detected may be added. That is, time measurement may start at the timing of emphasis of the region of interest, and the total time may be a time measured excluding the time during which the region of interest is not emphasized. For example, the total time at time t+3 may be Tc+1 excluding time t+1 and time t+2 during which the region of interest R7is not detected. In this case, the time at the last detection of the region of interest R7is stored in the region-of-interest storage unit50, and thus, the time during which the region of interest R7is not detected can be recognized.

In addition, herein, upon detection of a region of interest in the frame image38aat time t+3, it is determined whether the region of interest is identical with the region of interest R7in the frame image38aat the last-detection time t. If it is determined that the region of interest is not identical with the region of interest R7in this determination, it may be determined whether the region of interest is identical with a third region of interest, which is a region of interest in a frame image38a(an example of a third medical image) preceding (e.g., at time t−1) the frame image38aat time tin time series.

As a result, if it is determined that the region of interest is identical, the emphasis-degree setting unit45may take over the total time and the emphasis degree of the third region of interest stored in the region-of-interest storage unit50for the region of interest detected in the frame image38aat time t+3, and may add, to the total time, the elapsed time Tp from time t−1 until time t+3, for example.

Emphasis on Plurality of Regions of Interest

In a case in which a plurality of regions of interest are detected in a moving image that is being observed, if a user pays too much attention to one region of interest, the user may divert attention from the existence of another region of interest. In particular, in a case in which the regions of interest are away from each other or each region of interest exists at an end of the screen, for example, the user is likely to divert attention from a region of interest, which may result in missing of the region of interest. In addition, in a case in which the endoscope10is operated quickly, even if a plurality of regions of interest are detected, a region of interest may become out of the screen before its emphasis degree becomes high, and the user may not pay attention to it. Thus, in a case in which a plurality of regions of interest are detected, for example, their emphasis degrees may be increased, or increasing rates of the emphasis degrees may be increased, so as to avoid this problem.

FIG.12illustrates an example of change of figures when a plurality of regions of interest are detected in a single medical image. F12A illustrated inFIG.12illustrates a frame image38aon which afigure E71that surrounds a region of interest R8and afigure E81that surrounds a region of interest R9are superposed. In addition, F12B illustrated inFIG.12illustrates a frame image38aon which afigure E72that surrounds the region of interest R8and afigure E82that surrounds the region of interest R9are superposed. The frame image38aillustrated in F12B is a frame image subsequent to the frame image38aillustrated in F12A in time series.

As in the above case, the emphasis degree of the figure E72is set to a relatively larger value than that of the figure E71. In addition, the emphasis degree of the figure E82is set to a relatively larger value than that of the figure E81. Furthermore, herein, the minimum, the maximum, and the increasing rate of the emphasis degree are set to larger values as the number of regions of interest increases.

FIG.13is a graph illustrating an example of change of emphasis degrees when two regions of interest are detected, in which the horizontal axis represents time and the vertical axis represents emphasis degree. As illustrated inFIG.13, a first region of interest is emphasized by using a minimum mina, a maximum maxα, and an increasing rate Rα1. In addition, a second region of interest is emphasized by using a minimum minβ, a maximum maxβ, and an increasing rate Rβ. Herein, minα, minβ, maxα, maxβ, Rα1, and R3have relationships minα<minβ, maxα<maxβ, and Rα1<Rβ. These minimums, maximums, and increasing rates are stored in advance in the storage unit49.

The emphasis degrees of the regions of interest illustrated inFIG.12change as illustrated inFIG.13. Herein, the first region of interest R8is detected at time t1, and the region of interest R9is detected at time t2subsequent to time t1. In this case, for the region of interest R8, the emphasis-degree setting unit45sets the emphasis degree being the minimum mina at time t1and then sets the emphasis degree that is a value increased at the increasing rate Rα1with respect to the total time.

Subsequently, the second region of interest R9is detected at time t2. For the region of interest R9, the emphasis-degree setting unit45sets the emphasis degree being the minimum minβ at time t2and then sets the emphasis degree that is a value increased at the increasing rate Rβ with respect to the total time. In addition, from time t2, the emphasis-degree setting unit45sets the emphasis degree of the region of interest R8to a value increased at the increasing rate Rα2. The increasing rate Rα2has a relationship Rα1<Rα2<Rβ.

When the time further elapses, at time t3, the emphasis-degree setting unit45sets the emphasis degree of the region of interest R9to the maximum maxβ. Subsequently, emphasis on the region of interest R9ends. Note that emphasis at the emphasis degree being the maximum maxβ may be maintained. In addition, at time t4, the emphasis-degree setting unit45sets the emphasis degree of the region of interest R8to the maximum maxα. Subsequently, as in the case for the region of interest R9, emphasis on the region of interest R8ends. Note that emphasis at the emphasis degree being the maximum maxα may be maintained.

Herein, the minimum, the maximum, and the increasing rate of the emphasis degree are set to larger values as the number of regions of interest increases. However, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree may be set to a larger value. In this manner, by setting at least one of the minimum or the maximum of the emphasis degree to a larger value as the number of regions of interest increases and increasing the emphasis degree, the user can pay attention to the regions of interest, and the user can be prevented from missing the regions of interest. In addition, by setting the increasing rate of the emphasis degree to a larger value as the number of regions of interest increases and increasing the emphasis degree at an earlier stage, the user can pay attention to the regions of interest, and the user can be prevented from missing the regions of interest.

Setting of Emphasis Degree in Accordance with Position in Image

In order to reduce a load on a patient, an endoscopic inspection needs to be performed quickly, without taking time. In addition, operation of the endoscope10is complex. Thus, the insertion part20may be moved too quickly in some cases.

In a case in which the insertion part20is moved quickly, when a region of interest is detected at a portion where a user does not pay attention, the user may miss the region of interest. In particular, in a case in which the region of interest is detected at an end portion of the image, the user is likely to miss the region of interest.

In order to prevent such missing, the minimum, the maximum, and the increasing rate may be set in accordance with the distance from the center of the image.

FIG.14is a drawing for explaining a human visual performance. A region A1illustrated inFIG.14is a region including a focus point with the highest eyesight. A region A2is a region where information can be received instantly with only an eye movement from the focus point. A region A3is a region where information can be received without effort with an eye movement and a head movement from the focus point.

As the human visual performance, the eyesight is highest at the focus point and suddenly decreases to a vague view as the distance from the focus point increases. Typically, since a user often focuses on the center of an image, it is preferable to increase a supporting function at an end portion of the image. The above trend differs in an upper portion, a lower portion, a left portion, and a right portion, and thus, the increasing rate may be changed in accordance with the direction. For example, support is more needed in the upper direction than in the other directions, and thus, at least one of the minimum, the maximum, or the increasing rate is preferably made relatively larger.

Setting of Emphasis Degree in Accordance with Brightness

The minimum, the maximum, and the increasing rate of the emphasis degree may be set based on luminance of the region of interest or a difference between the luminance of the region of interest and luminance of a region outside the region of interest. As the region of interest is darker, or as the difference in brightness from the periphery is smaller, the user is more likely to miss the region of interest. Thus, in this case, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively large value.

Conversely, as the region of interest is brighter, or as the difference in brightness from the periphery is larger, the user is more unlikely to miss the region of interest. Thus, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively small value in order to prioritize the visibility of the image.

FIG.15is a graph illustrating an example of change of the emphasis degree in accordance with brightness of the region of interest, in which the horizontal axis represents time and the vertical axis represents emphasis degree. In a case in which the brightness of the region of interest is normal, the minimum of the emphasis degree is min0, and the maximum thereof is max0. In addition, as illustrated inFIG.15, an increasing rate Ri0is Ri0=(max0−min0)/(t3−t1).

In a case in which the brightness of the region of interest is relatively dark, the minimum of the emphasis degree is min1, and the maximum thereof is max1. In addition, as illustrated inFIG.15, an increasing rate Ri1is Ri1=(max1−min1)/(t2−t1). Herein, min0, min1, max0, max1, Ri0, and Ri1have relationships min0<min1, max0<max1, and Ri0<Ri1.

In a case in which the brightness of the region of interest is relatively bright, the minimum of the emphasis degree is min2, and the maximum thereof is max2. In addition, as illustrated inFIG.15, an increasing rate Ri2is Ri2=(max2−min2)/(t4−t1). Herein, min0, min2, max0, max2, Ri0, and Ri2have a relationship min0>min2, max0>max2, and Ri0>Ri2. These minimums, maximums, and increasing rates are stored in advance in the storage unit49.

The brightness of the region of interest can be acquired from the luminance of the region of interest. Herein, change of the emphasis degree in accordance with the luminance of the region of interest is illustrated. However, the same applies to a case in which the difference between the luminance of the region of interest and the luminance of the region outside the region of interest is used.

Setting of Emphasis Degree in Accordance with Color Information

As in a case of setting the emphasis degree in accordance with the brightness, the minimum, the maximum, and the increasing rate of the emphasis degree may be set based on color information of the region of interest or a difference between the color information of the region of interest and color information of a region outside the region of interest.

As the saturation of the region of interest is lower, as the color difference from the periphery is smaller, or as the color difference from the color of a figure to be used as reporting information is smaller, the user is more likely to miss the region of interest. Thus, as in the case of brightness, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively large value. For example, as illustrated inFIG.15, the minimum of the emphasis degree is min1, the maximum thereof is max1, and the increasing rate thereof is Ri1.

Conversely, as the saturation of the region of interest is higher, as the color difference from the periphery is larger, or as the color difference from the color of a figure is larger, the user is more unlikely to miss the region of interest. Thus, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively small value in order to prioritize the visibility of the image. For example, as illustrated inFIG.15, the minimum of the emphasis degree is min2, the maximum thereof is max2, and the increasing rate thereof is Ri2.

The color information is not limited to the RGB (Red, Green, Blue) color space. For example, the color information may also be the device-independent Lab color space of the L*a*b* color system, the YMCK (Cyan, Magenta, Yellow, blacK) color space of a color system suitable for printing processing, the HLS (Hue-Luminance-Saturation) color space, the so-called YCbCr (Y: luminance, CbCr: color difference) color space formed of luminance and color difference, or the like.

The color difference may also be determined whether the color difference falls within the allowable range (JIS standard or typically used by various industrial associations) distinguishable by humans.

Setting of Emphasis Degree in Accordance with Movement Information

As in cases of brightness and color information, the minimum, the maximum, and the increasing rate of the emphasis degree may be set based on movement information such as a movement amount, a movement direction, or the like of the region of interest.

As the movement amount of the region of interest is larger, it is more necessary for the user to be conscious about detection at an earlier stage. In addition, in a case in which the movement direction of the region of interest is toward an end portion of the image, the user is likely to miss the region of interest. Thus, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively large value.

Conversely, in a case in which the movement amount of the region of interest is small, it is unnecessary to be conscious at an early stage. In addition, in a case in which the movement direction of the region of interest is toward the center of the image, the user is unlikely to miss the region of interest. Thus, at least one of the minimum, the maximum, or the increasing rate of the emphasis degree is set to a relatively small value.

Region of Interest that is not Identical but Exists on the Periphery

In some cases, even if a region of interest is identical, the shape thereof in a captured image may differ depending on the insertion angle of the insertion part20, for example. In addition, in some cases, the region-of-interest detecting unit41cannot detect a region of interest as an identical region of interest with an accuracy of 100%. For such a reason, although a region of interest is identical, if it is not determined that the region of interest is an identical region of interest, emphasis processing with the minimum emphasis degree is continuously performed in each frame image. Such a situation interrupts a user's attention, which is not preferable.

Thus, even if it is not determined that a detected region of interest is identical with a previously detected region of interest, in a case in which the detected region of interest exists in a certain area, the total time and the emphasis degree of the previously detected region of interest may be taken over. Even if the shape or the like differs, reaction of the region-of-interest detecting unit41indicates a high likelihood of the region of interest on the periphery, and the user is made to focus it, which is useful in preventing the user from missing the region of interest.

FIG.16is a drawing for explaining a region of interest that is not identical but exists on the periphery.FIG.16illustrates examples of frame images38aat time t−1, time t, and time t+1. These frame images are continuous in the moving image38.

In the frame image38aat time t−1, a region of interest R10is detected. A figure E91at an emphasis degree in accordance with the total time of the region of interest R10is superposed at the position of the region of interest R10on the frame image38a. The feature quantity, the total time, and the emphasis degree of the region of interest R10are stored in the region-of-interest storage unit50in association with the region of interest R10.

Herein, for the region of interest R10, the region-of-interest detecting unit41sets a certain area B1centered at the region of interest R10. Although the frame shape indicating the area B1is illustrated in each frame38afor explanation inFIG.16, the frame shape indicating the area B1is not displayed on the display16.

Subsequently, upon the frame image38aat time t being input to the region-of-interest detecting unit41, the region-of-interest detecting unit41detects a region of interest R11from the frame image38a. In addition, the region-of-interest information acquiring unit42acquires the feature quantity of the region of interest R11.

Subsequently, the identical-region-of-interest determining unit46compares the feature quantity of the region of interest R10stored in the region-of-interest storage unit50and the feature quantity of the region of interest R11calculated by the region-of-interest information acquiring unit42with each other, and determines whether the region of interest R10and the region of interest R11are identical regions of interest. Herein, the feature quantities are different, and the identical-region-of-interest determining unit46determines that the regions of interest are not identical.

Upon determination that the regions of interest are not identical, the identical-region-of-interest determining unit46further determines whether the position of the region of interest R11in the frame image38ais a position within the area B1set for the region of interest R10. Herein, the identical-region-of-interest determining unit46determines that the position of the region of interest R11is a position within the area B1.

Upon determination that the position of the region of interest R11is a position within the area B1, the total-time measuring unit44measures the total time of the region of interest R11by taking over the total time of the region of interest R10stored in the region-of-interest storage unit50.

In addition, in accordance with the total time measured by taking over the total time of the region of interest R10, the emphasis-degree setting unit45sets the emphasis degree of the region of interest R11. The emphasis-degree setting unit45may alternatively calculate and set the emphasis degree of the region of interest R11from the emphasis degree of the region of interest R10stored in the region-of-interest storage unit50, the elapsed time from time t−1 until time t, and the predetermined increasing rate of the emphasis degree.

As a result, as illustrated inFIG.16, on the frame image38aat time t, a frame-shapedfigure E92that emphasizes the region of interest R11is superposed at the position of the region of interest R11. Thefigure E92is a figure having a relatively larger emphasis degree than thefigure E91for the time that equals to one frame. In this manner, the region of interest R11can take over the emphasis degree of the region of interest R10.

In addition, the control unit47stores the feature quantity, the total time, and the emphasis degree of the region of interest R11in the region-of-interest storage unit50in association with the region of interest R11.

Subsequently, upon the frame image38aat time t+1 being input to the region-of-interest detecting unit41, the region-of-interest detecting unit41detects a region of interest R12from the frame image38a. In addition, the region-of-interest information acquiring unit42calculates the feature quantity of the region of interest R12.

Subsequently, the identical-region-of-interest determining unit46compares the feature quantity of the region of interest R11stored in the region-of-interest storage unit50and the feature quantity of the region of interest R12calculated by the region-of-interest information acquiring unit42with each other, and determines whether the region of interest R11and the region of interest R12are identical regions of interest. Herein, the feature quantities are different, and the identical-region-of-interest determining unit46determines that the regions of interest are not identical.

The identical-region-of-interest determining unit46further determines whether the position of the region of interest R12in the frame image38ais a position within the area B1set for the region of interest R10. Herein, the identical-region-of-interest determining unit46determines that the position of the region of interest R12is a position within the area B1.

Upon determination that the position of the region of interest R12is a position within the area B1, the total-time measuring unit44measures the total time of the region of interest R12by taking over the total time of the region of interest R11stored in the region-of-interest storage unit50.

In addition, in accordance with the total time measured by taking over the total time of the region of interest R11, the emphasis-degree setting unit45sets the emphasis degree of the region of interest R12.

As a result, as illustrated inFIG.16, on the frame image38aat time t+1, a frame-shapedfigure E93that emphasizes the region of interest R12is superposed at the position of the region of interest R12. Thefigure E93is a figure having a relatively larger emphasis degree than thefigure E92for the time that equals to one frame. In this manner, the region of interest R12can take over the emphasis degree of the region of interest R11.

Note that, if it is determined that the position of the region of interest R12is not a position within the area B1, the region of interest R12is treated as a newly detected region of interest. That is, the total-time measuring unit44newly starts to measure the total time of the region of interest R12, and the emphasis-degree setting unit45sets the emphasis degree of the region of interest R12to the predetermined minimum.

Herein, the identical-region-of-interest determining unit46determines whether the position of the region of interest R12in the frame image38ais a position within the area B1set for the initially detected region of interest R10. However, a new area may be set for the region of interest R11, and it may be determined whether the region of interest R12is at a position within the area set for the region of interest R11.

In addition, the identical-region-of-interest determining unit46sets the certain area B1centered at the region of interest R10. However, the area used for determination of the identical-region-of-interest determining unit46in order to take over the emphasis degree may be a preset fixed area.

In the above manner, even if it is not determined that a detected region of interest is identical with a previously detected region of interest, in a case in which the detected region of interest exists in a certain area, by taking over the emphasis degree of the previously detected region of interest, the region of interest can be emphasized at an appropriate emphasis degree.

Case in which Plurality of Display Units Exist

Although an example of displaying the moving image38on a single display unit has been described above, the medical image processing apparatus14may include a plurality of display units. Some users do not wish the reporting information to be superposed on an image that is being observed. Thus, an image on which the reporting information is superposed may be displayed on a display unit that is different from a display unit that displays an image that is being observed.

FIG.17is a block diagram illustrating an electric configuration of the medical image processing apparatus14including a plurality of display units. The display16includes a first display unit16A and a second display unit16B. The first display unit16A and the second display unit16B may be two different monitors or may be different regions of one monitor.

The image display control unit48A outputs a moving image38acquired by the time-series image acquiring unit40to at least one of the first display unit16A or the second display unit16B. In addition, the reporting information display control unit48B outputs reporting information for reporting a region of interest to at least one of the first display unit16A or the second display unit16B.

Each ofFIG.18toFIG.20is a drawing for explaining display content on the plurality of display units. Upper portions ofFIG.18toFIG.20illustrate examples of display content at time t, time t+1, time t+2, and time t+3 on the first display unit16A, and lower portions ofFIG.18toFIG.20illustrate examples of display content at time t, time t+1, time t+2, and time t+3 on the second display unit16B.

As illustrated inFIG.18toFIG.20, frame images38aat time t, time t+1, time t+2, and time t+3, which are continuous frame images in the moving image38, are displayed on the first display unit16A at time t, time t+1, time t+2, and time t+3, respectively. The frame images38aat time t, time t+1, and time t+2 include an identical region of interest R13. In addition, the frame image38aat time t+3 does not include a region of interest.

In the example illustrated inFIG.18, as in the first display unit16A, frame images38aat time t, time t+1, time t+2, and time t+3 are displayed on the second display unit16B, at time t, time t+1, time t+2, and time t+3, respectively.

In addition, on the frame images38aat time t, time t+1, and time t+2 displayed on the second display unit16B, a figure E101, a figure E102, and a figure E103are respectively superposed at the position of the region of interest R13. The figure E101, the figure E102, and the figure E103have emphasis degrees in accordance with total times of the region of interest R13. The figure E101, the figure E102, and the figure E103are frame-shaped figures that surround the position of the region of interest R13and have different line transmittances.

By displaying the moving image38in this manner, a user can observe the image on the first display unit16A and can check reports of a region of interest on the second display unit16B as necessary.

In addition, in the example illustrated inFIG.19, frame images38aare not displayed on the second display unit16B, and only figures that are reporting information are displayed. The second display unit16B displays thefigure E101, thefigure E102, and thefigure E103having emphasis degrees in accordance with total times of the region of interest R13at time t, time t+1, and time t+2, respectively, at the position corresponding to the position of the region of interest R13in each frame image38a. In addition, no figure is displayed at time t+3.

In addition, also in the example illustrated inFIG.20, only figures that are reporting information are displayed on the second display unit16B. The second display unit16B displays afigure E111, afigure E112, and afigure E113having emphasis degrees in accordance with total times of the region of interest R13at time t, time t+1, and time t+2, respectively, at the position corresponding to the position of the region of interest R13in each frame image38a. Thefigure E111, thefigure E112, and thefigure E113are figures having substantially the same shape as the region of interest R13and different transmittances. In addition, no figure is displayed at time t+3.

Also in a case of such display, a user can observe the image on the first display unit16A and can check reports of a region of interest on the second display unit16B as necessary.

Medical Image Processing Method: Second Embodiment

Another example of a medical image processing method according to a second embodiment will be described.FIG.21is a flowchart illustrating an example of each process of the medical image processing method according to the second embodiment.

In step S11, the time-series image acquiring unit40acquires, as data D1, a frame image38ain a moving image38captured by the endoscope10. The region-of-interest detecting unit41detects a region of interest from the data D1.

In step S12, the control unit47determines whether the region-of-interest detecting unit41has detected a region of interest.

If the region-of-interest detecting unit41has not detected a region of interest, the process proceeds to step S13. In step S13, the control unit47further determines whether a continuous non-detection time during which no region of interest is detected is less than a threshold time Th. Although the continuous non-detection time is determined herein, the number of continuous non-detection frames may alternatively be determined. Further alternatively, a total non-detection time, or a total number of continuous non-detection frames may be determined.

If the continuous non-detection time is greater than or equal to the threshold time Th, the process proceeds to step S14, and the control unit47resets the region-of-interest storage unit50, which is a database. That is, the feature quantity, the total time, and the emphasis degree of the region of interest stored in the region-of-interest storage unit50are deleted. By deleting the feature quantity, the total time, and the emphasis degree of the region of interest, if the next region of interest is detected, it may be determined that the region of interest is a newly detected region of interest.

If the continuous non-detection time is less than the threshold time Th, the process proceeds to step S15, and the control unit47updates the region-of-interest storage unit50, which is a database. That is, the feature quantity, the total time, and the emphasis degree of the region of interest stored in the region-of-interest storage unit50are updated to most recent states. The continuous non-detection time is also updated. By updating the region-of-interest storage unit50in the above manner, even if detection of the region of interest is stopped, in a case in which the identical region of interest is detected again, the emphasis degree and the like can be taken over. In addition, determination of the continuous non-detection time can be performed appropriately.

When resetting the region-of-interest storage unit50in step S14or updating the region-of-interest storage unit50in step S15ends, the process in this flowchart ends.

On the other hand, if it is determined in step S12that the region-of-interest detecting unit41has detected a region of interest, the process proceeds to step S16.

In step S16, the identical-region-of-interest determining unit46determines whether the region of interest detected by the region-of-interest detecting unit41is identical with a region of interest detected in a past frame image. This determination is performed by comparing the feature quantity of the previous region of interest stored in the region-of-interest storage unit50and the feature quantity of the region of interest detected in step S11, the feature quantity being calculated by the region-of-interest information acquiring unit42, with each other.

If it is determined in step S16that the regions of interest are identical, the process proceeds to step S17. If it is determined in step S16that the regions of interest are not identical, the process proceeds to step S18.

In step S18, the identical-region-of-interest determining unit46further determines whether the position of the region of interest detected in step S11in the image is a position on the periphery of the previously detected region of interest. The position on the periphery is a position within a certain area centered at a previously detected region of interest, for example.

If it is determined in step S18that the region of interest exists at the position on the periphery of the previously detected region of interest, the process proceeds to step S17.

In step S17, the total-time measuring unit44measures the total time of the region of interest detected in step S11by taking over the total time of the region of interest stored in the region-of-interest storage unit50. Thus, the emphasis degree in accordance with the total time can be taken over. Subsequently, the process proceeds to step S19.

In addition, also if it is determined in step S18that the region of interest does not exist at the position on the periphery of the previously detected region of interest, the process proceeds to step S19.

In step S19, the region-of-interest information acquiring unit42acquires, as data D2, a past frame image38aof the data D1of the moving image38, and extracts, from the data D2, feature quantities of the region of interest and of a region outside the region of interest, such as luminance, color, or movement information. The feature quantities may be extracted in all frames, at certain intervals, at timings intended by a user, or the like. If the feature quantities are not extracted, the process may proceed to step S20from step S17.

Note that the region-of-interest information acquiring unit42may acquire, as the data D2, a future frame image38aof the data D1of the moving image38. Although delay occurs if the future frame image38ais acquired, if movement information is extracted as a feature quantity, by referring to the past frame image38aand the future frame image38a, the movement information can be extracted with a high accuracy.

In this regard, as the frame image38ato be referred to when acquiring the movement information, not only the frame image38aimmediately before or immediately after the current frame image38a, but also all of any given number of frame images38acentered at the current frame image38aor a frame image38aselected therefrom may be used. This is because the region of interest is not always detected continuously in time series. In addition, regarding the past frame image38a, targets may be only frame images38athat take over the emphasis degree.

In step S20, the emphasis-degree setting unit45sets the emphasis degree of the region of interest detected in step S11. The emphasis-degree setting unit45may set the emphasis degree that is taken over in step S17as it is or may change the emphasis degree that is taken over in accordance with the feature quantity extracted in step S19and set the emphasis degree.

In step S21, the feature quantity, the total time, and the emphasis degree of the region of interest are updated in the region-of-interest storage unit50, which is a database.

Subsequently, in step S22, the image display control unit48A outputs the frame image38athat is the data D1to the display16. In addition, based on the emphasis degree set by the emphasis-degree setting unit45, the reporting information display control unit48B outputs reporting information for reporting the region of interest to the display16.

Thus, the display16displays the frame image38ain which the region of interest is emphasized.

In the above manner, in a case in which the detected region of interest is identical with the previous region of interest, by taking over the emphasis degree, the region of interest can be emphasized at the emphasis degree of a larger value as the emphasized total time is longer. In addition, even in a case in which the detected region of interest is not identical with the previous region of interest, if the position of the detected region of interest exists on the periphery of the position of the previous region of interest, as in a case in which the regions of interest are identical, by taking over the emphasis degree, the region of interest can be emphasized at the emphasis degree of a larger value as the emphasized total time is longer. In addition, even in a case in which no region of interest is detected, if the non-detection time is less than the threshold time Th, by updating the database, if the identical region of interest is detected again before the threshold time Th elapses, the emphasis degree or the like of the previous region of interest can be taken over.

In the above manner, the region of interest in a medical image can be reported appropriately.

Although an endoscopic image is described as an example of the medical image, the medical image processing apparatus according to this embodiment can be applied to, not only the endoscopic image, but also a time-series medical image such as a capsule endoscopic image or an ultrasound image.

Miscellaneous

It is also possible to constitute a non-transitory recording medium such as a compact disk-read only memory (CD-ROM) storing the program51causing a computer to execute the above-described medical image processing method.

Although the endoscope processor apparatus12and the medical image processing apparatus14are described as different apparatuses from each other in the above embodiments, the endoscope processor apparatus12and the medical image processing apparatus14may also be constituted as an integrated apparatus, and the functions of the medical image processing apparatus14may be provided in the endoscope processor apparatus12.

In addition, a hardware structure of a processing unit that performs various processes of the endoscope processor apparatus12and the medical image processing apparatus14is any of the following various processors. Various processors include a central processing unit (CPU) that is a general-purpose processor functioning as various processing units by executing software (programs), a graphics processing unit (GPU) that is a processor specialized in image processing, a programmable logic device (PLD) that is a processor in which the circuit configuration is changeable after manufacture, such as field programmable gate array (FPGA), a dedicated electric circuit that is a processor having a circuit configuration that is specially designed to execute specific processing, such as an application specific integrated circuit (ASIC), and the like.

One processing unit may be constituted by one of these various processors, or may be constituted by two or more processors of the same type or different types (e.g., a combination of a plurality of FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, a plurality of processing units may be constituted by one processor. As a first example for constituting a plurality of processing units as one processor, one or more CPUs and software may be combined to constitute one processor, and this processor may function as a plurality of processing units, as typified by a computer such as a server or a client. As a second example, a processor may be used that implements the functions of the entire system including a plurality of processing units as one integrated circuit (IC) chip, as typified by a system on chip (SoC) or the like. In this manner, various processing units are constituted by one or more of various processors as a hardware structure.

Furthermore, the hardware structure of these various processors is more specifically electric circuitry obtained by combining circuit elements such as semiconductor elements.

The technical scope of the present invention is not limited to the scope described in the above embodiments, and configurations or the like in each embodiment may be combined as appropriate between the embodiments without departing from the spirit of the present invention.

REFERENCE SIGNS LIST

9endoscope system10endoscope11light source apparatus12endoscope processor apparatus13display apparatus14medical image processing apparatus15operating unit16display16A first display16B second display20insertion part21handheld operating unit22universal cord25soft part26bending part27distal end part28imaging element29bending operation knob30air/water supply button31suction button32still image pick-up command unit33treatment tool introduction port35light guide36signal cable37aconnector37bconnector38moving image38aframe image39still image40time-series image acquiring unit41region-of-interest detecting unit42region-of-interest information acquiring unit43coordinates calculating unit44total-time measuring unit45emphasis-degree setting unit46identical-region-of-interest determining unit47control unit48display control unit48A image display control unit48B reporting information display control unit49storage unit50region-of-interest storage unit51programD1, D2dataE1to E113figureR1to R13region of interestS1to S22step of medical image processing method