Patent ID: 12213643

DESCRIPTION OF EMBODIMENTS

In the following, modes for carrying out the present disclosure (hereinafter, embodiment) will be described with reference to the drawings. Note that the present disclosure is not limited to the embodiments described in the following. Furthermore, the same reference sign is assigned to the same parts in the drawings.

[1. Schematic Configuration of a Learning System]

FIG.1is a view illustrating a configuration of a learning system1according to the present embodiment.

The learning system1is a system that generates a learning model used to estimate a fluorescent area (singular area) where fluorescence is emitted when near-infrared excitation light is emitted to an observation target. As illustrated inFIG.1, this learning system1includes a training image generation device2and a learning device3. Then, these training image generation device2and learning device3perform wired or wireless communication via a network NE (FIG.1).

Note that although only one training image generation device2is illustrated inFIG.1, the number of training image generation devices2is not limited to one and may be plural.

[2. Configuration of the Training Image Generation Device]

First, the configuration of the training image generation device2will be described.

FIG.2is a view illustrating the configuration of the training image generation device2.

The training image generation device2is a device that generates a training image used for machine learning in the learning device3. As illustrated inFIG.2, this training image generation device2includes an insertion unit21, a light source device22, a light guide23, a camera head24, a first transmission cable25, a display device26, a second transmission cable27, a control device28, and a third transmission cable29.

The insertion unit21is a rigid endoscope. That is, the insertion unit21has an elongated shape that is entirely rigid or that is partially flexible and partially rigid, and is inserted into a living body. In this insertion unit21, an optical system that includes one or a plurality of lenses and that collects light from the inside of the living body (subject) is provided.

One end of the light guide23is connected to the light source device22, and light emitted to the inside of the living body is supplied therefrom to the one end of the light guide23under the control of the control device28. As illustrated inFIG.2, this light source device22includes a first light source221and a second light source222.

The first light source221emits normal light in a first wavelength band. In the present embodiment, the first light source221includes a light emitting diode (LED) that emits white light.

The second light source222emits excitation light in a second wavelength band different from the first wavelength band. In the present embodiment, a semiconductor laser that emits near-infrared excitation light in a near-infrared wavelength band is included. The near-infrared excitation light is excitation light that excites a fluorescent substance such as indocyanine green. Furthermore, when excited by the near-infrared excitation light, the fluorescent substance such as indocyanine green emits fluorescence having a central wavelength on a longer wavelength side of a central wavelength of the wavelength band of the near-infrared excitation light. Note that the wavelength band of the near-infrared excitation light and the wavelength band of the fluorescence may be set in such a manner as to partially overlap or may be set in such a manner as not to overlap at all.

Then, between alternately repeated first and second periods, the first light source221is driven in the light source device22in the first period under the control of the control device28. That is, the light source device22emits the normal light (white light) in the first period. Furthermore, the second light source222is driven in the light source device22in the second period under the control of the control device28. That is, the light source device22emits the near-infrared excitation light in the second period.

Note that although the light source device22is configured separately from the control device28in the present embodiment, this is not a limitation, and a configuration of being provided inside the control device28may be employed.

One end of the light guide23is detachably connected to the light source device22, and the other end thereof is detachably connected to the insertion unit21. Then, the light guide23transmits the light (normal light or near-infrared excitation light) supplied from the light source device22from the one end to the other end, and supplies the light to the insertion unit21. In a case where the normal light (white light) is emitted to the inside of the living body, the normal light reflected inside the living body is collected by the optical system in the insertion unit21. Note that the normal light collected by the optical system in the insertion unit21is referred to as a first subject image in the following, for convenience of description. In addition, in a case where the near-infrared excitation light is emitted to the inside of the living body, the near-infrared excitation light reflected inside the living body, and fluorescence emitted from the excited fluorescent substance such as indocyanine green accumulated at a lesion in the living body are collected by the optical system in the insertion unit21. Note that fluorescence transmitted through an excitation light cut-off filter242a(described later) after the collection of the near-infrared excitation light and the fluorescence by the optical system in the insertion unit21is referred to as a second subject image in the following, for convenience of description.

The camera head24is detachably connected to a proximal end (eyepiece211(FIG.2)) of the insertion unit21. Then, under the control of the control device28, the camera head24captures the first subject image (normal light) and the second subject image (fluorescence), and outputs an image signal (RAW signal) acquired by the imaging. The image signal is, for example, an image signal of 4K or higher.

Note that a detailed configuration of the camera head24will be described in “2-1. Configuration of the camera head” described later.

One end of the first transmission cable25is detachably connected to the control device28via a connector CN1(FIG.2), and the other end thereof is detachably connected to the camera head24via a connector CN2(FIG.2). Then, the first transmission cable25transmits the image signal and the like output from the camera head24to the control device28, and also transmits a control signal, synchronization signal, clock, electric power, and the like output from the control device28to the camera head24.

Note that in the transmission of the image signal and the like from the camera head24to the control device28through the first transmission cable25, the image signal and the like may be transmitted as an optical signal or may be transmitted as an electric signal. The same applies to transmission of the control signal, synchronization signal, and clock from the control device28to the camera head24through the first transmission cable25.

The display device26includes a display using liquid crystal, organic electro luminescence (EL), or the like, and displays an image based on a video signal from the control device28under the control of the control device28.

One end of the second transmission cable27is detachably connected to the display device26, and the other end thereof is detachably connected to the control device28. Then, the second transmission cable27transmits the video signal processed by the control device28to the display device26.

The control device28includes a central processing unit (CPU), a field-programmable gate array (FPGA), or the like, and comprehensively controls operations of the light source device22, the camera head24, and the display device26.

Note that a detailed configuration of the control device28will be described in “2-2. Configuration of the control device” described later.

One end of the third transmission cable29is detachably connected to the light source device22, and the other end thereof is detachably connected to the control device28. Then, the third transmission cable29transmits the control signal from the control device28to the light source device22.

[2-1. Configuration of the Camera Head]

Next, the configuration of the camera head24will be described.

FIG.3is a view illustrating configurations of the camera head24and the control device28.

Note that the connectors CN1and CN2between the control device28and the camera head24and the first transmission cable25, connectors between the control device28and the display device26and the second transmission cable27, and connectors between the control device28and the light source device22and the third transmission cable29are omitted inFIG.3, for convenience of description.

As illustrated inFIG.3, the camera head24includes a lens unit241, an imaging unit242, and a first communication unit243.

The lens unit241includes one or a plurality of lenses, and forms the first subject image (normal light) and the second subject image (fluorescence) on an imaging surface of the imaging unit242(imaging element242b).

The imaging unit242captures the inside of the living body under the control of the control device28. As illustrated inFIG.3, this imaging unit242includes an excitation light cut-off filter242a, an imaging element242b, and a signal processing unit242c.

The excitation light cut-off filter242ais provided between the lens unit241and the imaging element242b, and includes a band-stop filter that removes a specific wavelength band. That is, the excitation light cut-off filter242ais arranged on an optical path of the near-infrared excitation light, which is reflected inside the living body, from the inside of the living body to the imaging element242b. Note that the wavelength band that is cut (removed) by the excitation light cut-off filter242ais referred to as a cut band, a wavelength band that is on a short wavelength side of the cut band and passes through the excitation light cut-off filter242ais referred to as a short wave-side transmission band, and a wavelength band that is on a long wavelength side of the cut band and passes through the excitation light cut-off filter242ais referred to as a long wave-side transmission band in the following, for convenience of description.

Here, the cut band includes at least a part of the wavelength band of the near-infrared excitation light. In the present embodiment, the cut band includes the entire wavelength band of the near-infrared excitation light. In addition, the long wave-side transmission band includes the entire wavelength band of the fluorescence. Furthermore, the short wave-side transmission band includes the entire wavelength band of the normal light (white light).

That is, the excitation light cut-off filter242atransmits the first subject image (normal light (white light)) from the lens unit241toward the imaging element242b. On the other hand, with respect to the near-infrared excitation light and the fluorescence from the lens unit241toward the imaging element242b, the excitation light cut-off filter242acuts (removes) the near-infrared excitation light and transmits the fluorescence (second subject image).

The imaging element242bincludes a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like that receives light transmitted through the excitation light cut-off filter242aand that performs conversion thereof into an electric signal (analog signal).

Here, the imaging surface (light receiving surface) of the imaging element242bis provided with a color filter242d(FIG.3) in which three filter groups grouped according to wavelength bands of transmitted light (red (R), green (G), and blue (B)) are arrayed in a predetermined format (such as Bayer array).

Specifically, the color filter242dincludes an R filter group that mainly transmits light in an R wavelength band, a B filter group that mainly transmits light in a B wavelength band, a first G filter group (arrayed in the same column as the R filter group) that mainly transmits light in the G wavelength band, and a second G filter group (arrayed in the same column as the B filter group) that mainly transmits the light in the G wavelength band. Note that the first and second G filter groups are collectively referred to as a G filter group in the following, for convenience of description.

Here, each of the R, G, and B filter groups also transmits the fluorescence. Also, the imaging element242bhas sensitivity not only to the light in the R, G, and B wavelength bands but also to light in the wavelength band of the fluorescence.

Then, the imaging element242bperforms imaging in every first and second periods that are alternately repeated in synchronization with light emission timing of the light source device22under the control of the control device28. Hereinafter, for convenience of description, an image generated by capturing the first subject image (normal light) by the imaging element242bin the first period is referred to as a first training image, and an image generated by capturing the second subject image (fluorescence) by the imaging element242bin the second period is referred to as a second training image. In addition, the first and second training images are collectively referred to as a training image.

The signal processing unit242cperforms signal processing on the training image (analog signal) generated by the imaging element242band outputs the training image (RAW signal (digital signal)).

The first communication unit243functions as a transmitter that transmits the training image (RAW signal (digital signal)) output from the imaging unit242to the control device28through the first transmission cable25. This first communication unit243includes, for example, a high-speed serial interface that performs communication of the training image at a transmission rate equal to or higher than 1 Gbps with the control device28through the first transmission cable25.

[2-2. Configuration of the Control Device]

Next, the configuration of the control device28will be described with reference toFIG.3.

As illustrated inFIG.3, the control device28includes a second communication unit281, a memory282, an image generation unit283, a control unit284, an input unit285, an output unit286, a storage unit287, and a third communication unit288.

The second communication unit281functions as a receiver that receives the training image (RAW signal (digital signal)) output from the camera head24(first communication unit243) through the first transmission cable25. This second communication unit281includes, for example, a high-speed serial interface that performs communication of the training image with the first communication unit243at the transmission rate equal to or higher than 1 Gbps.

The memory282includes, for example, a dynamic random access memory (DRAM) or the like. This memory282can temporarily store a plurality of frames of training images sequentially output from the camera head24(first communication unit243).

The image generation unit283processes the training images (RAW signals (digital signals)) sequentially output from the camera head24(first communication unit243) and received by the second communication unit281under the control of the control unit284. As illustrated inFIG.3, this image generation unit283includes a memory controller283a, a first image processing unit283b, a second image processing unit283c, and a display controller283d.

The memory controller283acontrols writing and reading of the training images to and from the memory282. More specifically, the memory controller283asequentially writes, in the memory282, the training images (first and second training images) sequentially output from the camera head24(first communication unit243) and received by the second communication unit281. In addition, the memory controller283areads the first training image from the memory282at certain timing, and causes the first image processing unit283bto input the read first training image. Furthermore, the memory controller283areads the second training image from the memory282at certain timing, and causes the second image processing unit283cto input the read second training image.

The first image processing unit283bexecutes first image processing on the input first training image (RAW signal (digital signal)).

Examples of the first image processing include optical black subtraction processing, white balance adjustment processing, demosaic processing, color correction processing, gamma correction processing, and YC processing of converting an RGB signal (first training image) into a luminance signal and a color difference signal (Y and CB/CRsignals).

The second image processing unit283cexecutes second image processing different from the first image processing on the input second training image (RAW signal (digital signal)).

Examples of the second image processing include processing of generating only a luminance signal (Y signal) from the input second training image (RAW signal (digital signal)).

The display controller283dgenerates a video signal for a display which signal is to display at least one of the first training image on which the first image processing is executed by the first image processing unit283band the second training image on which the second image processing is executed by the second image processing unit283c. Then, the display controller283doutputs the video signal to the display device26through the second transmission cable27.

The control unit284includes, for example, a CPU, an FPGA, or the like, and controls the operations of the light source device22, the camera head24, and the display device26and controls the operation of the entire control device28by outputting a control signal through the first to third transmission cables25,27, and29.

Note that a part of the function of the control unit284will be described in “3. Operation of the training image generation device” described later.

The input unit285includes operation devices such as a mouse, a keyboard, and a touch panel, and receives user operation by a user such as a doctor. Then, the input unit285outputs an operation signal corresponding to the user operation to the control unit284.

The output unit286includes a speaker, a printer, or the like, and outputs various kinds of information.

The storage unit287stores a program executed by the control unit284, information necessary for processing by the control unit284, and the like.

The third communication unit288transmits/receives information to/from the learning device3via the network NE under the control of the control unit284.

[3. Operation of the Training Image Generation Device]

Next, an operation of the above-described training image generation device2will be described.

FIG.4is a flowchart illustrating the operation of the training image generation device2.FIG.5andFIG.6are views for describing the operation of the training image generation device2. Specifically,FIG.5is a view illustrating a first training image WLI of one frame after execution of the first image processing.FIG.6is a view illustrating a second training image IR of one frame after execution of the second image processing. Note that the second training image IR illustrated inFIG.6is expressed in a gray scale, and intensity (corresponding to luminance value) of a component of the captured fluorescence is higher (luminance value is larger) as approaching black.

First, the control unit284executes time-division driving of the first and second light sources221and222(Step S1A). Specifically, in Step S1A, based on the synchronization signal, the control unit284causes the first light source221to emit the normal light (white light) in the first period and causes the second light source222to emit the near-infrared excitation light in the second period in the alternately repeated first and second periods.

After Step S1A, the control unit284causes the imaging element242bto capture the first and second subject images in the first and second periods respectively in synchronization with the light emission timing of the first and second light sources221and222based on the synchronization signal (Steps S1B to S1D). That is, in a case of the first period (Step S1B: Yes), in other words, in a case where the normal light (white light) is emitted to the inside of the living body, the imaging element242bcaptures the first subject image (normal light) and generates the first training image (Step S1C). On the other hand, in a case of the second period (Step S1B: No), in other words, in a case where the near-infrared excitation light is emitted to the inside of the living body, the imaging element242bcaptures the second subject image (fluorescence) and generates the second training image (Step S1D).

After Step S1C and S1D, the memory controller283acontrols writing and reading of the training images to and from the memory282based on the synchronization signal (Step S1E).

After Step S1E, the first and second image processing units283band283cexecute the following processing (Step S1F).

That is, the first image processing unit283bsequentially executes the first image processing on first training images sequentially read from the memory282by the memory controller283a. Then, the first image processing unit283boutputs the first training image WLI illustrated inFIG.5, for example. On the other hand, the second image processing unit283csequentially executes the second image processing on second training images sequentially read from the memory282by the memory controller283a. Then, the second image processing unit283coutputs the second training image IR illustrated inFIG.6, for example. Note that, since being images captured by the same imaging element242b, the first and second training images have the same image size as can be seen from comparison between the first and second training images WLI and IR illustrated inFIG.5andFIG.6. That is, the same pixel positions in the first and second training images are pixels acquired by imaging of the same position of the same subject.

After Step S1F, the control unit284controls an operation of the third communication unit288, and sequentially transmits, to the learning device3, training images in which the first and second training images respectively output from the first and second image processing units283band283cin Step S1F are paired (Step S1G).

Then, the control unit284returns to Step S1A.

[3. Configuration of the Learning Device]

Next, the configuration of the learning device3will be described.

FIG.7is a view illustrating the configuration of the learning device3.

The learning device3is, for example, a server device, and is a portion that generates a learning model by using the training images generated by the training image generation device2. As illustrated inFIG.7, this learning device3includes a communication unit31, a control unit32, and a storage unit33.

The communication unit31transmits/receives information to/from the training image generation device2(third communication unit288) via the network NE under the control of the control unit32.

The control unit32includes, for example, a CPU, an FPGA, or the like, and controls an operation of the entire learning device3. This control unit32includes a training image acquisition unit321, a singular area specification unit322, a first feature data extraction unit323, and a singular-corresponding area learning unit324.

Note that functions of the training image acquisition unit321, the singular area specification unit322, the first feature data extraction unit323, and the singular-corresponding area learning unit324will be described in “4. Operation of the learning device” described later.

The storage unit33stores a program executed by the control unit32, information necessary for processing by the control unit32, information generated by the processing, and the like.

[4. Operation of the Learning Device]

Next, the operation of the learning device3described above will be described.

FIG.8is a flowchart illustrating the operation of the learning device3.

First, via the communication unit31, the training image acquisition unit321sequentially acquires training images (first and second training images) transmitted from the training image generation device2(third communication unit288) (Step S2A).

After Step S2A, the singular area specification unit322specifies a fluorescent area (singular area) in the second training image (Step S2B).

Specifically, the singular area specification unit322specifies, as the fluorescent area, an area in which a pixel level is equal to or higher than a specific threshold in the second training image.

Here, examples of the pixel level include a luminance value corresponding to a Y signal (luminance signal) and an RGB value (pixel value). In the present embodiment, the luminance value is employed as the pixel level. That is, in the second training image IR illustrated inFIG.6, an area Ar in which the luminance value is equal to or larger than the specific threshold is specified as the fluorescent area. Furthermore, in the present embodiment, the singular area specification unit322specifies, as the fluorescent area Ar, each of a first fluorescent area (first singular area) Ar1in which the luminance value is within a first range (gray portion illustrated inFIG.6) and a second fluorescent area (second singular area) Ar2in which the luminance value is within a second range higher than the first range (black portion illustrated inFIG.6).

After Step S2B, the first feature data extraction unit323extracts feature data of each of a fluorescence-corresponding area (singular-corresponding area) and a non-corresponding area in the first training image paired with the second training image in which the fluorescent area is specified in Step S2B (Step S2C).

Here, in the first training image, the fluorescence-corresponding area is an area at a pixel position corresponding to the fluorescent area of the second training image (the same pixel position as the fluorescent area). Also, the non-corresponding area is an area other than the fluorescence-corresponding area in the first training image. In the present embodiment, the fluorescence-corresponding area includes, in the first training image, a first fluorescence-corresponding area at a pixel position corresponding to the first fluorescent area Ar1of the second training image and a second fluorescence-corresponding area at a pixel position corresponding to the second fluorescent area Ar2of the second training image. That is, the first feature data extraction unit323extracts feature data of each of the first and second fluorescence-corresponding areas and the non-corresponding area.

In addition, as the feature data of the first fluorescence-corresponding area, the following extraction methods (1) to (3) can be exemplified.

(1) Feature data is extracted for each pixel included in the first fluorescence-corresponding area.

(2) With a plurality of pixels included in the first fluorescence-corresponding area as one group, feature data is extracted for each group.

(3) Feature data of the entire first fluorescence-corresponding area is extracted.

Note that extraction methods for the feature data of the second fluorescence-corresponding area and the feature data of the non-corresponding area are also similar to the above.

Furthermore, examples of the feature data include feature data related to a resolution, edge, color, brightness, noise, contrast, histogram, and the like.

After Step S2C, the singular-corresponding area learning unit324generates a learning model by performing machine learning on the first and second fluorescence-corresponding areas based on the feature data of the first fluorescence-corresponding area, the feature data of the second fluorescence-corresponding area, and the feature data of the non-corresponding area (Step S2D). That is, by using the learning model, it is possible to determine whether an area having the feature data is any of the first and second fluorescence-corresponding areas and the non-corresponding area from the feature data.

Here, examples of the machine learning include machine learning using a convolutional neural network (deep learning). That is, in the machine learning, as the number of training images in which the first and second training images are paired increases, it becomes possible to generate a learning model capable of more accurately determining the first and second fluorescence-corresponding areas and the non-corresponding area.

In the present embodiment, when generating the learning model by performing the machine learning on the first and second fluorescence-corresponding areas, the singular-corresponding area learning unit324sets a weight of feature data related to a blue color component to be lower than a weight of feature data related to the other red and green color components. For example, the feature data related to the blue color component is not used for the machine learning.

Then, the singular-corresponding area learning unit324stores the generated learning model in the storage unit33.

[5. Configuration of the Medical Observation Device]

Next, a medical observation device4that estimates a fluorescent area by using the learning model generated by the learning device3will be described.

FIG.9is a view illustrating the configuration of the medical observation device4.FIG.10is a view illustrating configurations of a camera head44and a control device48.

As illustrated inFIG.9orFIG.10, the medical observation device4has a configuration substantially similar to that of the training image generation device2. Note that the same reference sign is assigned to a configuration, which is the same as that of the training image generation device2, in the medical observation device4.

In the following, among configurations of the medical observation device4, configurations different from those of the training image generation device2will be mainly described.

In the medical observation device4, the light source device42has a configuration corresponding to the light source device22in the training image generation device2. As illustrated inFIG.9orFIG.10, this light source device42includes only a first light source221. That is, unlike the light source device22, the light source device42does not include a second light source222.

Then, in the light source device42, the first light source221is driven and emits only the normal light (white light) under the control of the control device48.

Note that although the light source device42is configured separately from the control device48in the present embodiment, this is not a limitation, and a configuration provided inside the control device48may be employed.

In the medical observation device4, the camera head44has a configuration corresponding to the camera head24in the training image generation device2. As illustrated inFIG.10, since the light source device42does not include the second light source222, this camera head44has a configuration in which the excitation light cut-off filter242ais omitted from the camera head24.

Then, the camera head44(imaging element242b) performs imaging in a specific frame period under the control of the control device48. Hereinafter, for distinction from the first training image, an image generated by capturing the first subject image (normal light) by the camera head44(imaging element242b) will be referred as a captured image.

The control device48corresponds to the medical image processing device according to the present disclosure. In the medical observation device4, this control device48has a configuration corresponding to the control device28in the training image generation device2. In this control device48, as illustrated inFIG.10, an image generation unit483and a control unit484are employed with respect to the control device28instead of the image generation unit283and the control unit284.

The image generation unit483processes captured images (RAW signals (digital signals)) sequentially output from the camera head44(first communication unit243) and received by the second communication unit281under the control of the control unit484. As illustrated inFIG.10, this image generation unit483includes a memory controller483a, an image processing unit483b, a second feature data extraction unit483c, a singular-corresponding area specification unit483d, and a display controller483e.

Note that functions of the memory controller483a, the image processing unit483b, the second feature data extraction unit483c, the singular-corresponding area specification unit483d, and the display controller483ewill be described in “6. Operation of the medical observation device” described later.

The control unit484includes, for example, a CPU, an FPGA, or the like, and controls operations of the light source device42, the camera head44, and the display device26and controls an operation of the entire control device48by outputting a control signal through the first to third transmission cables25,27, and29.

Note that a part of the function of the control unit484will be described in “6. Operation of the medical observation device” described later.

[6. Operation of the Medical Observation Device]

Next, an operation of the medical observation device4described above will be described.

FIG.11is a flowchart illustrating the operation of the medical observation device4.FIG.12is a view for describing the operation of the medical observation device4. Specifically,FIG.12is a view illustrating a display image WLI′ generated in Step S3G. Here, inFIG.12, for convenience of description, it is assumed that a captured image that is the same as the first training image WLI illustrated inFIG.5is generated in Step S3B.

Note that it is assumed that the control unit484controls an operation of the third communication unit288, receives a learning model from the learning device3, and stores the learning model in the storage unit287before executing the operation of the medical observation device4described in the following.

First, the control unit484drives the light source device42(first light source221) (Step S3A). As a result, the normal light (white light) is emitted to the inside of the living body (observation target).

After Step S3A, the control unit484generates a captured image by causing the imaging element242bto capture the first subject image (normal light) in a specific frame period (Step S3B).

After Step S3B, the memory controller483acontrols writing and reading of the captured image to and from the memory282(Step S3C). Specifically, the memory controller483asequentially writes, into the memory282, captured images sequentially output from the camera head44(first communication unit243) and received by the second communication unit281. In addition, the memory controller483areads the captured images from the memory282at timing of specification, and causes the image processing unit483bto input the read captured images.

After Step S3C, the image processing unit483bsequentially executes the above-described first image processing on each of the captured images sequentially read from the memory282by the memory controller483a(Step S3D).

After Step S3D, with respect to the captured images sequentially output from the image processing unit483bin Step S3D, the second feature data extraction unit483cextracts feature data of each area of the captured images (Step S3E).

Here, as the feature data, the following extraction methods (4) and (5) can be exemplified.

(4) Feature data is extracted for each pixel included in a captured image.

(5) With a plurality of pixels included in a captured image as one group (area), feature data is extracted for each group.

In addition, the feature data extracted by the second feature data extraction unit483cis the same kind of feature data as the feature data extracted by the first feature data extraction unit323.

After Step S3E, based on the feature data extracted in Step S3E, the singular-corresponding area specification unit483dspecifies the first and second fluorescence-corresponding areas in the captured images by using the learning model stored in the storage unit287(Step S3F).

After Step S3F, the display controller483egenerates a display image in which the first and second fluorescence-corresponding areas specified in Step S3F are displayed in a manner of being distinguished from the other area in the captured images (Step S3G). For example, in a case where it is assumed that a captured image that is the same as the first training image WLI illustrated inFIG.5is generated in Step S3B, a display image WLI′ in which first and second fluorescence-corresponding areas Art′ and Ar2′ (fluorescence-corresponding areas Ar′) are distinguished from the other area is generated in the captured image, as illustrated inFIG.12. As a method of the distinction, for example, application of a single color (such as green) to the fluorescence-corresponding areas Ar′ in the captured image can be exemplified. In addition, as described above, the second fluorescent area Ar2is an area having the luminance value larger than that of the first fluorescent area Ar1. Thus, it is preferable that a color darker than that of the first fluorescence-corresponding area Ar1′ (expressed in gray inFIG.12, for convenience of description) is applied to the second fluorescence-corresponding area Ar2′ (expressed in black inFIG.12, for convenience of description).

Then, the display controller483egenerates a video signal corresponding to the display image WLI′, and outputs the video signal to the display device26through the second transmission cable27. As a result, the display device26displays the display image WLI′.

According to the present embodiment described above, the following effects are acquired.

The learning device3according to the present embodiment uses the training images in which the first and second training images are paired, and generates the learning model by performing, based on the feature data of the fluorescence-corresponding area at the pixel position corresponding to the fluorescent area in the second training image in the first training image, machine learning on the fluorescence-corresponding area.

Then, the control device48according to the present embodiment acquires the captured image acquired by capturing the first subject image (normal light) from the inside of the living body (observation target), and specifies the fluorescence-corresponding area in the captured image by using the above-described learning model based on the feature data of each area in the captured image.

That is, since the fluorescent area can be estimated by utilization of the learning model, it is not necessary to administer a fluorescent substance such as indocyanine green into the living body. Thus, convenience can be improved.

In addition, in the medical observation device4, the second light source222and the excitation light cut-off filter242acan be omitted since it is not necessary to emit near-infrared excitation light to the inside of the living body. Thus, the configuration can be simplified and downsized.

In addition, in the learning device3according to the present embodiment, the fluorescent area is divided into two stages of the first and second fluorescence areas according to levels of the luminance values. The same applies to a fluorescence-corresponding area corresponding to the fluorescent area. That is, the fluorescence-corresponding area specified by the control device48is also divided into the first and second fluorescence-corresponding areas. For this reason, the user such as a doctor can easily recognize a portion in which the intensity of the fluorescent component is estimated to be high and a portion in which the intensity of the fluorescent component is estimated to be low from the display image.

Incidentally, when a lesion such as cancer is specified by image recognition, it is not necessary to consider the feature data of the blue color component.

In the learning device3according to the present embodiment, when the learning model is generated by machine learning on the fluorescence-corresponding area, the weight of the feature data related to the blue color component is made lower than the weight of the feature data related to other color components. For example, the feature data related to the blue color component is not used for the machine learning. Thus, a processing load can be reduced since machine learning can be performed without consideration of unnecessary feature data.

Other Embodiments

Although modes for carrying out the present disclosure have been described above, the present disclosure is not limited only to the above-described embodiment.

Although the fluorescent area is divided into two stages of the first and second fluorescence areas according to the levels of the luminance values in the above-described embodiment, this is not a limitation. There may be only one fluorescent area, or the fluorescent area may be divided into three stages or more according to the levels of the luminance values. The same applies to the fluorescence-corresponding area corresponding to the fluorescent area.

In the above-described embodiment, the light in the first wavelength band is the normal light (white light), and the light in the second wavelength band is the near-infrared excitation light. However, this is not a limitation. As long as the first wavelength band and the second wavelength band are different, other light may be employed. At this time, the first and second wavelength bands may be partially overlapping bands, or may be bands that do not overlap at all.

For example, narrowband light used in so-called narrow band imaging (NBI) may be employed as the light in the second wavelength band. At this time, the light in the first wavelength band may be the normal light (white light) or other light.

Incidentally, photo dynamic diagnosis (PDD) that is one of cancer diagnosis methods for detecting a cancer cell is conventionally known.

In the photo dynamic diagnosis, for example, a photosensitive substance such as a 5-aminolevulinic acid (hereinafter, referred to as 5-ALA) is used. The 5-ALA is a natural amino acid originally included in living bodies of animals and plants. This 5-ALA is taken into cells after administration into a body, and is biosynthesized into protoporphyrin in mitochondria. The protoporphyrin is excessively accumulated in the cancer cell. In addition, the protoporphyrin that is excessively accumulated in the cancer cell has photoactivity. Thus, when being excited by excitation light (such as blue visible light in a wavelength band of 375 nm to 445 nm), the protoporphyrin emits fluorescence (such as red fluorescence in a wavelength band of 600 nm to 740 nm). As described above, a cancer diagnostic method of causing the cancer cell to fluorescently emit light by using the photosensitive substance is called the photo dynamic diagnosis.

Then, in the above-described embodiment, the excitation light that excites the protoporphyrin (such as blue visible light in the wavelength band of 375 nm to 445 nm) may be employed as the light in the second wavelength band. At this time, the light in the first wavelength band may be the normal light (white light) or other light.

Although the first and second training images are generated by the single imaging element242bin the above-described embodiment, this is not a limitation. For example, a configuration in which a first subject image and a second subject image are separated and respectively captured by two imaging elements and first and second training images are respectively generated by the two imaging elements may be employed. At this time, the learning device3needs to recognize a correspondence relationship of pixels between the first and second training images.

Although the training image generation device2and the learning device3are communicably connected to each other via the network NE in the above-described embodiment, this is not a limitation. The training image generation device2and the learning device3may be configured as one device.

Although the medical image processing device according to the present disclosure is mounted on the medical observation device4in which the insertion unit21includes a rigid endoscope in the above-described embodiment, this is not a limitation. For example, the medical image processing device according to the present disclosure may be mounted on a medical observation device in which an insertion unit21includes a flexible endoscope. In addition, the medical image processing device according to the present disclosure may be mounted on a medical observation device such as a surgical microscope that enlarges and observes a predetermined visual field area inside a subject (inside a living body) or a subject surface (living body surface) (see, for example, Japanese Patent Application Laid-open No. 2016-42981).

In the above-described embodiment, a part of the configuration of the camera head44or a part of the configuration of the control device48may be provided in, for example, the connector CN1or the connector CN2.

Note that the following configurations also belong to the technical scope of the present disclosure.

(1) A learning device including:a training image acquisition unit configured to acquire training images in which a first training image acquired by capturing light from a subject irradiated with light in a first wavelength band and a second training image acquired by capturing light from the subject irradiated with light in a second wavelength band different from the first wavelength band are paired;a singular area specification unit configured to specify a singular area in the second training image;a first feature data extraction unit configured to extract feature data of a singular-corresponding area at a pixel position, the pixel position corresponding to the singular area in the first training image; anda singular-corresponding area learning unit configured to generate a learning model by performing machine learning on the singular-corresponding area based on the feature data.

(2) The learning device according to (1), whereinthe subject emits fluorescence by being irradiated with excitation light in the second wavelength band,the second training image is acquired by capturing the fluorescence from the subject irradiated with the excitation light, andthe singular area specification unit is configured to specify, as the singular area, an area in which intensity of a component of the fluorescence is equal to or higher than a specific threshold in the second training image.

(3) The learning device according to (1) or (2), wherein the singular area specification unit is configured to specify, as the singular area, an area in which a pixel level is equal to or higher than a specific threshold in the second training image.

(4) The learning device according to (3), whereinthe singular area specification unit is configured to specify, as the singular area, each of a first singular area in which the pixel level is within a first range and a second singular area in which the pixel level is within a second range higher than the first range,the first feature data extraction unit is configured to extract feature data of each of a first singular-corresponding area, which is the singular-corresponding area in the first training image and is at a pixel position corresponding to the first singular area, and a second singular-corresponding area which is the singular-corresponding area in the first training image and is at a pixel position corresponding to the second singular area, andthe singular-corresponding area learning unit is configured to perform machine learning on each of the first singular-corresponding area and the second singular-corresponding area based on the feature data.

(5) The learning device according to any one of (1) to (4), whereinthe feature data includes feature data related to at least one of color and luminance, andthe singular-corresponding area learning unit is configured to make a weight of the feature data related to a blue color component lower than a weight of the feature data related to another color component when performing the machine learning on the singular-correspondence area and generating a learning model.

(6) The learning device according to any one of (1) to (5), whereinthe light in the first wavelength band is emitted in a first period of the first period and a second period that are alternately repeated, andthe light in the second wavelength band is emitted in the second period.

(7) A medical image processing device including:a captured image acquisition unit configured to acquire a captured image acquired by capturing light from an observation target irradiated with light in a first wavelength band;a second feature data extraction unit configured to extract feature data of each area of the captured image; anda singular-corresponding area specification unit configured to specify a singular-corresponding area in the captured image by using a learning model constructed by machine learning based on the feature data, whereinthe learning model is generated by the machine learning on the singular-corresponding area by utilization of training images, in which a first training image acquired by capturing light from a subject irradiated with the light in the first wavelength band and a second training image acquired by capturing light from the subject irradiated with light in a second wavelength band different from the first wavelength band are paired, based on the feature data of the singular-corresponding area at a pixel position, the pixel position corresponding to a singular area in the second training image, in the first training image.

(8) The medical image processing device according to (7), whereinthe subject emits fluorescence by being irradiated with excitation light in the second wavelength band,the second training image is acquired by capturing the fluorescence from the subject irradiated with the excitation light, andthe singular area is an area in which intensity of a component of the fluorescence is equal to or higher than a specific threshold in the second training image.

(9) The medical image processing device according to (7) or (8), wherein the singular area is an area in which a pixel level is equal to or larger than a specific threshold in the second training image.

(10) The medical image processing device according to (9), whereinthe singular area includes a first singular area in which the pixel level is within a first range, anda second singular area in which the pixel level is within a second range higher than the first range, andthe learning model is generated by machine learning on each of a first singular-corresponding area, which is the singular-corresponding area in the first training image and is at a pixel position corresponding to the first singular area, and a second singular-corresponding area, which is the singular-corresponding area in the first training image and is at a pixel position corresponding to the second singular area, based on the feature data of the first singular-corresponding area and the second singular-corresponding area.

(11) The medical image processing device according to any one of (7) to (10), whereinthe feature data includes feature data related to at least one of color and luminance, andthe learning model is generated by machine learning on the singular-corresponding area in a state in which a weight of the feature data related to a blue color component is made lower than a weight of the feature data related to another color component.

(12) The medical image processing device according to any one of (7) to (11), whereinthe light in the first wavelength band is emitted in a first period of the first period and a second period that are alternately repeated, andthe light in the second wavelength band is emitted in the second period.

(13) The medical image processing device according to any one of (7) to (12), further including a display controller configured to generate a display image in which the singular-corresponding area is displayed in a manner of being distinguished from another area in the captured image.

REFERENCE SIGNS LIST

1LEARNING SYSTEM2TRAINING IMAGE GENERATION DEVICE3LEARNING DEVICE4MEDICAL OBSERVATION DEVICE21INSERTION UNIT22,42LIGHT SOURCE DEVICE23LIGHT GUIDE24,44CAMERA HEAD25FIRST TRANSMISSION CABLE26DISPLAY DEVICE27SECOND TRANSMISSION CABLE28,48CONTROL DEVICE29THIRD TRANSMISSION CABLE31COMMUNICATION UNIT32CONTROL UNIT33STORAGE UNIT211EYEPIECE221FIRST LIGHT SOURCE222SECOND LIGHT SOURCE241LENS UNIT242IMAGING UNIT242aEXCITATION LIGHT CUT-OFF FILTER242bIMAGING ELEMENT242cSIGNAL PROCESSING UNIT242dCOLOR FILTER243FIRST COMMUNICATION UNIT281SECOND COMMUNICATION UNIT282MEMORY283IMAGE GENERATION UNIT283aMEMORY CONTROLLER283bFIRST IMAGE PROCESSING UNIT283cSECOND IMAGE PROCESSING UNIT283dDISPLAY CONTROLLER284,484CONTROL UNIT285INPUT UNIT286OUTPUT UNIT287STORAGE UNIT288THIRD COMMUNICATION UNIT321TRAINING IMAGE ACQUISITION UNIT322SINGULAR AREA SPECIFICATION UNIT323FIRST FEATURE DATA EXTRACTION UNIT324SINGULAR-CORRESPONDING AREA LEARNING UNIT483IMAGE GENERATION UNIT483aMEMORY CONTROLLER483bIMAGE PROCESSING UNIT483cSECOND FEATURE DATA EXTRACTION UNIT483dSINGULAR-CORRESPONDING AREA SPECIFICATION UNIT483eDISPLAY CONTROLLERAr FLUORESCENT AREAAr′ FLUORESCENCE-CORRESPONDING AREAAr1FIRST FLUORESCENT AREAAr1′ FIRST FLUORESCENCE-CORRESPONDING AREAAr2SECOND FLUORESCENT AREAAr2′ SECOND FLUORESCENCE-CORRESPONDING AREACN1, CN2CONNECTORIR SECOND TRAINING IMAGENE NETWORKWLI FIRST TRAINING IMAGEWLI′ DISPLAY IMAGE