Patent Description:
A typical endoscope apparatus irradiates an observation target with illumination light emitted from a distal end of an insertion section of an endoscope and captures an image of the observation target by using an imaging element to acquire image information. It is known that the illumination light can be implemented as special light, as well as white light (normal light), having a different spectrum from white light (<CIT> and <CIT>).

The endoscope apparatus described in <CIT> has a probe portion at the distal end of the insertion section of the endoscope such that the probe portion is pressed against a surface of a living body to detect a feature value of the surface of the living body, and automatically switches an observation mode (illumination light) between a normal-light observation mode using white light and a special-light observation mode using special light in accordance with the detected feature value.

The endoscope apparatus described in <CIT> has a first illumination mode in which the amount of narrow-band light is increased compared to the amount of broadband light, a second illumination mode in which the amount of narrow-band light is substantially equal to the amount of broadband light, and a third illumination mode in which the amount of narrow-band light is decreased compared to the amount of broadband light, determines the type of an observation site, and automatically switches the illumination mode in accordance with the determined type of the observation site, thereby reducing the load on the operator.

In recent years, it has been known to support an examination by performing recognition such as detecting a lesion included in an image through image analysis or classifying lesions by type and by performing notification.

In image analysis for recognition, accurate automatic recognition is enabled by machine learning of images such as deep learning (for example, A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet classification with deep convolutional neural networks, in NIPS, <NUM>).

In the invention described in <CIT>, the probe portion at the distal end of the insertion section of the endoscope is pressed against a surface of a living body to dent the surface of the living body, and when the size of the dented region of the surface of the living body exceeds a threshold value, the observation mode is automatically switched to the special-light observation mode. The operator needs to press the probe portion against the surface of the living body (perform a palpation).

In the invention described in <CIT>, in accordance with the type of the observation site (for example, the esophagus, the cardia, or the stomach), automatic switching is performed among the first illumination mode using illumination light suitable for special-light observation of the esophagus, the second illumination mode using illumination light suitable for special-light observation of the cardia, and the third illumination mode using illumination light suitable for special-light observation of the stomach. The automatic switching is performed only for the observation of a plurality of observation sites of different types in a single endoscopic examination.

<CIT> discloses a living body observation system that switches from a deep blood vessel mode to a white light mode when the size of a bleeding region is less than a second predetermined value during treatment. If the size of the bleeding becomes more than a first predetermined value during the treatment, the system switches from the white light mode to the deep blood vessel mode.

The present invention has been made in view of such circumstances, and an object thereof is to provide an endoscope apparatus, an endoscope processor, and a method for operating the endoscope apparatus in which illumination light (observation mode) is automatically switched in response to detection of a detection target from an image to reduce the burden of an operator's switching operation.

In an aspect of the present invention, there is provided an endoscope apparatus as claimed in claim <NUM>.

According to the aspect of the present invention, in response to continuous detection of the detection target from images captured under the first illumination light in the first observation mode, the first observation mode is automatically switched to the second observation mode in which an image is captured under the second illumination light. Thus, it is possible to capture an image of the detection target under the second illumination light, which is suitable for detailed observation of the detection target, and to reduce the burden of the operator's operation of switching the observation mode.

When the amount of change is within the threshold value, it is considered that the image remains substantially stationary (the operator is gazing at the detection target). Thus, the observation mode is switched to the second observation mode, which is suitable for detailed observation of the detection target.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the specific region is an entire region of an image captured by the imaging unit.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the specific region is a center region of an image captured by the imaging unit.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the specific region is a region corresponding to the detection target, the region being calculated based on detection information from the detector.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the amount of change calculated by the amount-of-change calculation unit is an amount of change in a position of the specific region.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the amount of change calculated by the amount-of-change calculation unit is an amount of change in a size of the specific region.

In an endoscope apparatus according to still another aspect of the present invention, preferably, in response to an elapse of a certain period of time after the observation mode switching unit switches to the second observation mode, the observation mode switching unit switches to the first observation mode. This is because the observation of the detection target in the second observation mode is completed after a certain period of time has elapsed.

In an endoscope apparatus according to still another aspect of the present invention, preferably, after the observation mode switching unit switches to the second observation mode, the observation mode switching unit switches to the first observation mode in response to the continuous detection determination unit determining that the detection target is not continuously detected. This is because there is no detection target to be observed in the second observation mode.

In an endoscope apparatus according to still another aspect of the present invention, preferably, after the observation mode switching unit switches to the second observation mode, the observation mode switching unit switches to the first observation mode in response to the amount-of-change determination unit determining that the amount of change is larger than the threshold value. When the amount of change is larger than the threshold value, it is considered that the image changes and the operator is not gazing at the detection target. Thus, the observation mode is switched to the first observation mode.

In an endoscope apparatus according to still another aspect of the present invention, preferably, after the observation mode switching unit switches to the second observation mode, the observation mode switching unit switches to the first observation mode in response to a still image being captured.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the continuous detection determination unit determines that the detector continuously detects the detection target in response to the detector consecutively detecting the detection target within a certain time range longer than a detection interval of the detector.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the continuous detection determination unit determines that the detector continuously detects the detection target in response to the detector detecting the detection target at a rate greater than or equal to a threshold value within a certain time range longer than a detection interval of the detector.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the first observation mode is a normal-light observation mode in which normal light is used as the first illumination light, and the second observation mode is a special-light observation mode in which special light is used as the second illumination light.

In an endoscope apparatus according to still another aspect of the present invention, preferably, the first observation mode is a first special-light observation mode in which first special light is used as the first illumination light, and the second observation mode is a second special-light observation mode in which second special light different from the first special light is used as the second illumination light.

According to the present invention, in response to continuous detection of a detection target from images captured under first illumination light in a first observation mode, the observation mode is automatically switched to a second observation mode in which an image is captured under second illumination light. Thus, it is possible to capture an image of the detection target under the second illumination light, which is suitable for detailed observation of the detection target, and to reduce the burden of the operator's operation of switching the observation mode.

The following describes preferred embodiments of an endoscope apparatus, an endoscope processor, and a method for operating the endoscope apparatus according to the present invention with reference to the accompanying drawings.

<FIG> is a perspective view illustrating the external appearance of an endoscope apparatus <NUM> according to the present invention.

As illustrated in <FIG>, the endoscope apparatus <NUM> is constituted mainly by an endoscope (here, a flexible endoscope) <NUM> that captures an image of an observation target in a subject, a light source device (light source unit) <NUM>, an endoscope processor <NUM>, a display device <NUM> such as a liquid crystal monitor, and a detector <NUM>.

The light source device <NUM> supplies various types of observation light, including white light (first illumination light) for capturing a normal-light image and special light (second illumination light) having a different spectrum from white light, to the endoscope <NUM>.

The endoscope processor <NUM> has an image processing function for generating image data of a normal-light image, a special-light image, or an observation image to be used for display/recording based on an image signal obtained by the endoscope <NUM>, a function of controlling the light source device <NUM>, a function of causing the display device <NUM> to display the normal-light image or the observation image and a detection result obtained by the detector <NUM>, and so on.

As described in detail below, the detector <NUM> is a section that accepts an endoscopic image from the endoscope processor <NUM> and detects the position of a detection target (such as a lesion, a scar from an operation, or a scar after a treatment) from the endoscopic image, discriminates the type of the lesion, or performs other processing. In this example, the endoscope processor <NUM> and the light source device <NUM> are constructed separately and electrically connected to each other. Alternatively, the light source device <NUM> may be incorporated into the endoscope processor <NUM>. Likewise, the detector <NUM> may be incorporated into the endoscope processor <NUM>.

The display device <NUM> displays a normal-light image, a special-light image, or an observation image based on image data to be used for display that is input from the endoscope processor <NUM>, and a recognition result obtained by the detector <NUM>.

The endoscope <NUM> includes a flexible insertion section <NUM> to be inserted into a subject, a handheld operation section <NUM> coupled to a proximal end portion of the insertion section <NUM> and used to grasp the endoscope <NUM> and operate the insertion section <NUM>, and a universal cord <NUM> that connects the handheld operation section <NUM> to the light source device <NUM> and the endoscope processor <NUM>.

An insertion section distal end 16a at a distal end of the insertion section <NUM> incorporates an illumination lens <NUM>, an objective lens <NUM>, an imaging element <NUM>, and so on (see <FIG>). A bendable bending portion 16b is coupled to the rear end of the insertion section distal end 16a. A flexible pipe portion 16c having flexibility is coupled to the rear end of the bending portion 16b.

The handheld operation section <NUM> is provided with an angle knob <NUM>, an operation button <NUM>, a forceps inlet <NUM>, and so on. The angle knob <NUM> is rotated to adjust the bending direction and the amount of bending of the bending portion 16b. The operation button <NUM> is used for various operations such as air supply, water supply, and suction. The forceps inlet <NUM> communicates with a forceps channel in the insertion section <NUM>. The handheld operation section <NUM> is also provided with an endoscope operating unit <NUM> (see <FIG>) that performs various kinds of setting, and so on.

The universal cord <NUM> has installed therein an air/water supply channel, a signal cable, a light guide, and so on. The universal cord <NUM> has disposed in a distal end portion thereof a connector portion 25a to be connected to the light source device <NUM> and a connector portion 25b to be connected to the endoscope processor <NUM>. Accordingly, observation light is supplied from the light source device <NUM> to the endoscope <NUM> via the connector portion 25a, and an image signal obtained by the endoscope <NUM> is input to the endoscope processor <NUM> via the connector portion 25b.

The light source device <NUM> is provided with a light source operating unit 12a such as a power button, a turn-on button for turning on the light source, and a brightness adjustment button, and the endoscope processor <NUM> is provided with a processor operating unit 13a including a power button and an input unit for accepting input from a pointing device such as a mouse (not illustrated).

<FIG> is a block diagram illustrating an electric configuration of the endoscope apparatus <NUM>.

As illustrated in <FIG>, the endoscope <NUM> roughly has a light guide <NUM>, the illumination lens <NUM>, the objective lens <NUM>, the imaging element <NUM>, the endoscope operating unit <NUM>, an endoscope control unit <NUM>, and a ROM (Read Only Memory) <NUM>.

Examples of the light guide <NUM> include a large-diameter optical fiber and a bundle fiber. The light guide <NUM> has a light incident end that is inserted into the light source device <NUM> via the connector portion 25a, and a light emitting end that passes through the insertion section <NUM> and faces the illumination lens <NUM> disposed in the insertion section distal end 16a. Illumination light supplied from the light source device <NUM> to the light guide <NUM> is applied to the observation target via the illumination lens <NUM>. The illumination light reflected and/or scattered by the observation target is incident on the objective lens <NUM>.

The objective lens <NUM> forms an image of reflected light or scattered light of the incident illumination light (i.e., an optical image of the observation target) on an imaging surface of the imaging element <NUM>.

The imaging element <NUM> is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) imaging element and is positioned and fixed relatively to the objective lens <NUM> at a position on the back side of the objective lens <NUM>. On the imaging surface of the imaging element <NUM>, a plurality of pixels constituted by a plurality of photoelectric conversion elements (photodiodes) that perform photoelectric conversion of an optical image are arranged two-dimensionally. In this example, on the light incident surface side of the plurality of pixels of the imaging element <NUM>, red (R), green (G), and blue (B) color filters are arranged for the respective pixels, thereby forming an R pixel, a G pixel, and a B pixel. The filter arrangement of the RGB color filters is typically, but not limited to, a Bayer arrangement.

The imaging element <NUM> converts the optical image formed by the objective lens <NUM> into an electrical image signal and outputs the electrical image signal to the endoscope processor <NUM>.

When the imaging element <NUM> is a CMOS imaging element, an A/D (Analog/Digital) converter is incorporated, and a digital image signal is output from the imaging element <NUM> directly to the endoscope processor <NUM>. When the imaging element <NUM> is a CCD imaging element, an image signal output from the imaging element <NUM> is converted into a digital image signal by an A/D converter (not illustrated) or the like and is then output to the endoscope processor <NUM>.

The endoscope operating unit <NUM> has arranged thereon a still-image capturing button (not illustrated) and a mode switching button (not illustrated) for manually switching an observation mode, and a switching signal from the mode switching button is input to the endoscope control unit <NUM>. The mode switching button is an operating unit that switches the type of illumination light (observation mode) each time the mode switching button is pressed, and includes an "AUTO" mode for automatically switching the observation mode, as described below. The mode switching button may be disposed in the processor operating unit 13a of the endoscope processor <NUM>.

The endoscope control unit <NUM> sequentially executes various programs and data read out from the ROM <NUM> or the like in accordance with the operation performed using the endoscope operating unit <NUM>, and mainly controls driving of the imaging element <NUM>. For example, in the normal-light observation mode in which white light (normal light) is used as illumination light, the endoscope control unit <NUM> controls the imaging element <NUM> to read out signals of the R pixel, the G pixel, and the B pixel of the imaging element <NUM>. In the special-light observation mode in which special light having a different spectrum from white light is used as illumination light, when violet light is emitted from a V-LED 32a or blue light is emitted from a B-LED 32b as observation light to acquire a specific special-light image, the endoscope control unit <NUM> controls the imaging element <NUM> to read out signals of only the B pixel of the imaging element <NUM> having spectral sensitivity in the wavelength range of violet light or blue light or to read out any one color pixel or two color pixels among the three color pixels including the R pixel, the G pixel, and the B pixel.

Further, the endoscope control unit <NUM> communicates with a processor control unit <NUM> of the endoscope processor <NUM> and transmits to the endoscope processor <NUM> information on the operation performed by the endoscope operating unit <NUM>, identification information for identifying the type of the endoscope <NUM> stored in the ROM <NUM>, and the like.

The light source device <NUM> has a light source control unit <NUM> and a light source unit <NUM>. The light source control unit <NUM> controls the light source unit <NUM> and communicates with the processor control unit <NUM> of the endoscope processor <NUM> to exchange various kinds of information.

The light source unit <NUM> has, for example, a plurality of semiconductor light sources. In this embodiment, the light source unit <NUM> has LEDs of four colors, namely, the V-LED (Violet Light Emitting Diode) 32a, the B-LED (Blue Light Emitting Diode) 32b, a G-LED (Green Light Emitting Diode) 32c, and an R-LED (Red Light Emitting Diode) 32d. The V-LED 32a, the B-LED 32b, the G-LED 32c, and the R-LED 32d are semiconductor light sources that emit violet (V) light, blue (B) light, green (G) light, and red (R) light, which are observation light having a peak wavelength at, for example, <NUM>, <NUM>, <NUM>, and <NUM>, respectively.

The light source control unit <NUM> individually controls, for the respective LEDs, turning on or off of the four LEDs of the light source unit <NUM>, the amount of light emitted at the time of turning on, and the like in accordance with the observation mode such as the normal-light observation mode and the special-light observation mode. In the normal-light observation mode, the light source control unit <NUM> turns on all of the V-LED 32a, the B-LED 32b, the G-LED 32c, and the R-LED 32d. In the normal-light observation mode, therefore, white light including V light, B light, G light, and R light is used as illumination light.

In the special-light observation mode, on the other hand, the light source control unit <NUM> turns on any one light source or an appropriate combination of a plurality of light sources among the V-LED 32a, the B-LED 32b, the G-LED 32c, and the R-LED 32d. In a case where a plurality of light sources are turned on, special light in which the amounts of light (the ratio of the amounts of light) to be emitted from the respective light sources are controlled is used as illumination light. This makes it possible to capture images of a plurality of layers having different depths of a photographic subject.

In this example, in the first observation mode, white light (WL) for a normal-light image is emitted. In the second observation mode, special light for a special-light image (BLI (Blue Light Imaging or Blue LASER Imaging), LCI (Linked Color Imaging), or NBI (Narrow Band Imaging)) is emitted.

The illumination light for BLI is illumination light having a high proportion of V light with high absorbance for the superficial blood vessel whereas the proportion of G light with high absorbance for the middle blood vessel is reduced, and is suitable for generating an image (BLI) suitable for enhancing a blood vessel or a structure in the mucosal superficial layer of a photographic subject.

The illumination light for LCI is illumination light in which the proportion of V light is higher than that of observation light for WL and which is more suitable for capturing a fine change in color tone than the observation light for WL, and is suitable for generating an image (LCI) subjected to color enhancement processing to make a reddish color more red and a whitish color more white relative to the color near the mucous membrane by also using the signal of the R component.

The illumination light for NBI is suitable for generating an image (NBI) in which a fine change in the surface to be irradiated is enhanced by narrowing the range of the wavelengths of illumination light to be applied.

Light of colors emitted from the LEDs 32a to 32d is incident on the light guide <NUM>, which is inserted into the endoscope <NUM>, via an optical path coupling portion formed by a dichroic mirror, a lens, and the like and an aperture diaphragm mechanism (not illustrated).

As the observation light of the light source device <NUM>, light in various wavelength ranges according to an observation purpose is selected, such as white light (light in the white wavelength range or light in a plurality of wavelength ranges), light (special light) having a peak in one or a plurality of specific wavelength ranges, or a combination thereof.

A first example of the specific wavelength range is, for example, the blue range or the green range in the visible range. The wavelength range in the first example includes a wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>, and light in the first example has a peak wavelength in the wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>.

A second example of the specific wavelength range is, for example, the red range in the visible range. The wavelength range in the second example includes a wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>, and light in the second example has a peak wavelength in the wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>.

A third example of the specific wavelength range includes a wavelength range in which the absorption coefficient is different between oxyhemoglobin and reduced hemoglobin, and light in the third example has a peak wavelength in the wavelength range in which the absorption coefficient is different between oxyhemoglobin and reduced hemoglobin. The wavelength range in the third example includes a wavelength range of <NUM> ± <NUM>, <NUM> ± <NUM>, <NUM> ± <NUM>, or greater than or equal to <NUM> and less than or equal to <NUM>, and light in the third example has a peak wavelength in the wavelength range of <NUM> ± <NUM>, <NUM> ± <NUM>, <NUM> ± <NUM>, or greater than or equal to <NUM> and less than or equal to <NUM> described above.

A fourth example of the specific wavelength range is a wavelength range (<NUM> to <NUM>) of excitation light that is used for observation (fluorescence observation) of fluorescence emitted from a fluorescent substance in a living body and that excites the fluorescent substance.

A fifth example of the specific wavelength range is the wavelength range of infrared light. The wavelength range in the fifth example includes a wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>, and light in the fifth example has a peak wavelength in the wavelength range greater than or equal to <NUM> and less than or equal to <NUM> or greater than or equal to <NUM> and less than or equal to <NUM>.

The endoscope processor <NUM> has the processor operating unit 13a, the processor control unit <NUM>, a ROM <NUM>, a digital signal processor (DSP) <NUM>, an image processing unit <NUM>, a display control unit <NUM>, a storage unit <NUM>, and so on.

The processor operating unit 13a includes a power button, an input unit that accepts inputs such as a coordinate position pointed on the screen of the display device <NUM> by a mouse and a click (execution instruction), and so on.

The processor control unit <NUM> reads out a necessary program and data from the ROM <NUM> in accordance with the information on the operation performed by the processor operating unit 13a and information on the operation performed by the endoscope operating unit <NUM>, which is received via the endoscope control unit <NUM>, and sequentially processes the program and data to control the units of the endoscope processor <NUM> and control the light source device <NUM>. The processor control unit <NUM> may accept a necessary instruction input from any other external device such as a keyboard connected via an interface (not illustrated).

Under the control of the processor control unit <NUM>, the DSP <NUM> functioning as a form of image acquisition unit that acquires image data of each frame of a moving image output from the endoscope <NUM> (the imaging element <NUM>) performs various types of signal processing, such as defect correction processing, offset processing, white balance correction, gamma correction, and demosaicing, on image data of one frame of the moving image input from the endoscope <NUM> to generate image data for the frame.

The image processing unit <NUM> receives image data from the DSP <NUM> and performs image processing, such as color conversion processing, color enhancement processing, and structure enhancement processing, on the received image data as necessary to generate image data indicating an endoscopic image in which an observation target appears. The color conversion processing is processing for performing color conversion on image data by using <NUM> × <NUM> matrix processing, gradation transformation processing, three-dimensional look-up table processing, and so on. The color enhancement processing is processing for color enhancement for image data subjected to color conversion processing, for example, in a direction of making a difference in tint between a blood vessel and a mucous membrane. The structure enhancement processing is, for example, processing for enhancing a specific tissue or structure included in an observation target such as a blood vessel or a pit pattern and is performed on image data after color enhancement processing.

The image data of each frame of the moving image processed by the image processing unit <NUM> is recorded in the storage unit <NUM> as a still image or a moving image instructed to be captured when an instruction is given to capture a still image or a moving image.

The display control unit <NUM> generates display data for displaying a normal-light image or a special-light image on the display device <NUM> on the basis of the image data input from the image processing unit <NUM>, outputs the generated display data to the display device <NUM>, and causes the display device <NUM> to display a display image (such as a moving image captured by the endoscope <NUM>).

Further, the display control unit <NUM> causes the display device <NUM> to display a recognition result input from the detector <NUM> via the image processing unit <NUM> or a recognition result input from the detector <NUM>.

When a region of interest is detected by the detector <NUM>, the display control unit <NUM> displays an index indicating the region of interest so as to be superimposed on an image displayed on the display device <NUM>. Examples of the index include highlighting such as changing the color of the region of interest in the display image, displaying a marker, and displaying a bounding box.

Further, the display control unit <NUM> can display, based on the detection result of the detection target by the detector <NUM>, information indicating the presence or absence of the detection target so as not to overlap with the image displayed on the display device <NUM>. The information indicating the presence or absence of the detection target may be, for example, such that the color of the frame of the endoscopic image is changed between when the detection target is detected and when the detection target is not detected, or such that the text "the detection target is present!" is displayed in a display region different from the endoscopic image.

When the detector <NUM> performs discrimination for a lesion, the display control unit <NUM> causes the display device <NUM> to display the discrimination result. Examples of the method for displaying the discrimination result include displaying text indicating the discrimination result in a display image on the display device <NUM>. The text may not necessarily be displayed in the display image, and may be displayed in any way so long as the correspondence relationship with the display image can be identified.

Next, the detector <NUM> according to the present invention will be described.

The detector <NUM> is a section that detects a detection target such as a lesion from images sequentially captured by an imaging unit (the endoscope <NUM>), and sequentially accepts images subjected to image processing by the endoscope processor <NUM>. In this example, in the first observation mode, a normal-light image (WL image) is accepted as an image for detection. In the second observation mode, a special-light image (BLI, CLI, or NBI) is accepted as an image for detection.

<FIG> is a schematic diagram illustrating a typical example configuration of a convolutional neural network (CNN), which is one of the learning models constituting the detector <NUM>.

A CNN <NUM> is, for example, a learning model for detecting the position of a detection target (such as a lesion, a scar from an operation, or a scar after a treatment) appearing in an endoscopic image or discriminating the type of the lesion. The CNN <NUM> has a multiple-layer structure and holds a plurality of weight parameters. The weight parameters are set to optimum values, thereby allowing the CNN <NUM> to become a learned model and function as a detector.

As illustrated in <FIG>, the CNN <NUM> includes an input layer 15A, an intermediate layer 15B having a plurality of convolution layers and a plurality of pooling layers, and an output layer 15C, and each layer has a structure in which a plurality of "nodes" are coupled using "edges".

In this example, the CNN <NUM> is a learning model that performs segmentation for recognizing the position of the detection target appearing in the endoscopic image. The learning model to which a fully convolutional network (FCN: Fully Convolutional Network), which is a type of CNN, is applied to the CNN <NUM>. Examples of the FCN includes: one that determines the position of the detection target appearing in the endoscopic image on a pixel-by-pixel basis or determines the presence or absence of the detection target in units of several pixels; one that outputs the values of the coordinates of the center of the detection target, the values of the coordinates of four corners of a rectangular shape surrounding the detection target, and the like.

An image of one frame for detection is input to the input layer 15A.

The intermediate layer 15B is a portion that extracts a feature from an image input from the input layer 15A. Each of the convolution layers of the intermediate layer 15B performs filtering processing (performs a convolution operation using a filter) on an image or a nearby node in the preceding layer to acquire a "feature map". The pooling layers reduce (or enlarge) the feature maps output from the convolution layers to obtain new feature maps. The "convolution layer" plays a role of feature extraction such as edge extraction from an image, and the "pooling layer" plays a role of providing robustness so that the extracted features are not affected by parallel displacement or the like. The intermediate layer 15B does not necessarily include sets each including a convolution layer and a pooling layer, and may be configured such that convolution layers are consecutive, or may also include a normalization layer.

The output layer 15C is a portion that outputs a detection result obtained by detecting the position of the detection target appearing in the endoscopic image or classifying (discriminating) the type of the lesion on the basis of the features extracted by the intermediate layer 15B.

The CNN <NUM> is learned using a large number of sets each including an image set for learning and correct answer data for the image set, and filter coefficients or offset values to be applied to the respective convolution layers of the CNN <NUM> are set to optimum values by using data sets for learning. The correct answer data is preferably a discrimination result or a detection target designated by a doctor for the endoscopic image.

In this example, the CNN <NUM> is configured to recognize the position of the detection target appearing in the endoscopic image. However, the detector (CNN) is not limited to this, and may be configured to execute discrimination for the lesion and output a discrimination result. For example, the detector may classify the endoscopic image into three categories including "neoplastic", "non-neoplastic", and "other" and output three scores corresponding to "neoplastic", "non-neoplastic", and "other" (the total of the three scores is <NUM>%) as the discrimination result, or may output the classification result if the endoscopic image can be clearly classified from the three scores. In addition, a CNN that outputs such a discrimination result preferably has a fully connected layer as the last one layer or a plurality of layers of the intermediate layer instead of the fully convolutional network (FCN).

Furthermore, the detector <NUM> preferably uses a learning model learned using a normal-light image when a normal-light image is to be input, and applies a learning model learned using a special-light image when a special-light image is to be input.

<FIG> is a block diagram illustrating a main part of a first embodiment of an endoscope processor in an endoscope apparatus according to the present invention.

The endoscope processor <NUM> of the first embodiment illustrated in <FIG> includes a processor control unit <NUM>-<NUM>.

The processor control unit <NUM>-<NUM> is a section that performs overall control of the units of the endoscope processor <NUM>. The processor control unit <NUM>-<NUM> of the first embodiment further includes a continuous detection determination unit <NUM> and an observation mode switching unit <NUM>.

The continuous detection determination unit <NUM> receives a detection result from the detector <NUM> and determines whether the detector <NUM> continuously detects the detection target on the basis of the received detection result. The determination of whether the detection target has been detected from one frame (image) can be performed, for example, based on whether a pixel having the detection target is present in a case where the detector <NUM> outputs the result of the determination of the presence or absence of the detection target on a pixel-by-pixel basis or in units of several pixels, or can be performed based on whether the values of the coordinates are output in a case where the detector <NUM> outputs the values of the coordinates of the center of the detection target or the values of the coordinates of four corners of a rectangular shape surrounding the detection target.

The continuous detection determination unit <NUM> can determine that the detector <NUM> continuously detects the detection target when the detector <NUM> detects the detection target consecutively within a certain time range longer than the detection interval of the detector <NUM> (the period of one frame of the moving image or the period of a plurality of frames of the moving image).

Whether the detection target is continuously detected is considered to be determined by sequentially storing detection results of the detector <NUM> and referring to the current detection result and the most recent detection result.

Preferably, the continuous detection determination unit <NUM> determines that the detector <NUM> continuously detects the detection target not only when the detector <NUM> detects the detection target at each detection interval within a certain time range longer the detection interval of the detector <NUM> but also when the detector <NUM> detects the detection target at a rate equal to or higher than a threshold value.

For example, when the detector <NUM> detects the detection target in the period of one frame of the moving image (for each frame), it is considered to refer to the detection results for the most recent <NUM> frames (for one second) that are sequentially input. In the determination of continuous detection, it may be determined that the detection target is continuously detected not only when, for example, the detection target is detected in all of <NUM> frames in a case where the previous <NUM> frames are referred to but also when, for example, the detection target is detected in every other frame to every three frames.

The observation mode switching unit <NUM> is a section that switches between the first observation mode and the second observation mode. In a state where the first observation mode is used, the observation mode switching unit <NUM> automatically switches from the first observation mode to the second observation mode when the continuous detection determination unit <NUM> determines that the detection target is continuously detected.

In this example, the first observation mode is a normal-light observation mode in which WL (normal light) is emitted for observation, and the second observation mode is a special-light observation mode in which special light for a special-light image (BLI, LCI, or NBI) is emitted for observation.

Accordingly, the observation mode switching unit <NUM> outputs a command for switching from the first observation mode to the second observation mode or a command for switching from emission of WL to emission of special light to the light source device <NUM> to automatically switch from the first observation mode to the second observation mode.

When a certain period of time (for example, several seconds) has elapsed after the switching of the observation mode to the second observation mode, the observation mode switching unit <NUM> switches to the first observation mode. Alternatively, after switching to the second observation mode, the observation mode switching unit <NUM> may switch to the first observation mode when the continuous detection determination unit <NUM> determines that the detection target is not continuously detected.

<FIG> is a conceptual diagram illustrating automatic switching of the observation mode when a normal-light observation mode using WL is set as the first observation mode and a special-light observation mode using special light for BLI is set as the second observation mode.

As illustrated in <FIG>, when the detection target is continuously detected from normal-light images (WL images) sequentially captured in the normal-light observation mode, the observation mode switching unit <NUM> automatically switches from the normal-light observation mode to the special-light observation mode, and special-light images (BLI) are captured in the special-light observation mode. When a certain period of time has elapsed after the switching of the observation mode to the special-light observation mode, the observation mode is switched again to the normal-light observation mode by the observation mode switching unit <NUM>.

Accordingly, in a state where the normal-light observation mode is used, the observation mode is automatically switched from the normal-light observation mode to the special-light observation mode when the detection target is continuously detected from sequentially captured images. Thus, it is possible to capture an image of the detection target under special light suitable for detailed observation of the detection target, and to reduce the burden of the operator's operation of switching the observation mode.

<FIG> is a block diagram illustrating a main part of a second embodiment of an endoscope processor in an endoscope apparatus according to the present invention. In <FIG>, components common to those of the first embodiment illustrated in <FIG> are denoted by the same reference numerals, and detailed description thereof will be omitted.

The endoscope processor <NUM> of the second embodiment illustrated in <FIG> includes a processor control unit <NUM>-<NUM>.

The processor control unit <NUM>-<NUM> is different from the processor control unit <NUM>-<NUM> illustrated in <FIG> mainly in that an amount-of-change calculation unit <NUM> and an amount-of-change determination unit <NUM> are added.

The amount-of-change calculation unit <NUM> sequentially receives image data of the respective frames of the moving image processed by the image processing unit <NUM> and computes an amount of change in a specific region of images captured by the endoscope <NUM> on the basis of the sequentially received image data.

The specific region of the images can be the entire region of a captured image. The amount of change in the specific region can be the amount of change in the size or position of the consecutively captured images.

The amount-of-change determination unit <NUM> determines whether the amount of change calculated by the amount-of-change calculation unit <NUM> is within a threshold value.

For example, when observation is performed while the endoscope insertion section is pulled out from the body cavity, the image being observed is varying, whereas when the endoscope insertion section is temporarily stopped from being pulled out, the image being observed is stationary. Accordingly, the threshold value for the amount of change can be a value set as a criterion for determining whether the size or position of the specific region between consecutive images has changed with the movement of the endoscope insertion section.

In a state where the normal-light observation mode is used, the observation mode switching unit <NUM> automatically switches from the normal-light observation mode to the special-light observation mode when the continuous detection determination unit <NUM> determines that the detection target is continuously detected and the amount-of-change determination unit <NUM> determines that the amount of change in a specific region of the images is within the threshold value.

After switching to the special-light observation mode, the observation mode switching unit <NUM> switches from the special-light observation mode to the normal-light observation mode when the amount-of-change determination unit <NUM> determines that the amount of change in a specific region of the images is larger than the threshold value.

According to the second embodiment, in a state where the normal-light observation mode is used, when a detection target is continuously detected from sequentially captured images and the amount-of-change determination unit <NUM> determines that the amount of change in the specific region is small (when the amount of change is determined to be within the threshold value), it is considered that the endoscope insertion section remains substantially stationary (the operator is gazing at the detection target). Thus, the observation mode is automatically switched from the normal-light observation mode to the special-light observation mode.

In contrast, even when the detection target is continuously detected from sequentially captured images, if the amount-of-change determination unit <NUM> determines that the amount of change in specific region is large (if the amount of change is determined to be larger than the threshold value), it is considered that the endoscope insertion section is moving (for example, observation is being performed while the endoscope insertion section is pulled out from the body cavity). Thus, the observation mode is not switched from the normal-light observation mode to the special-light observation mode.

The specific region is not limited to the entire region of a captured image, and may be, for example, a center region of a captured image.

<FIG> is a block diagram illustrating a main part of a third embodiment of an endoscope processor in an endoscope apparatus according to the present invention. In <FIG>, components common to those of the second embodiment illustrated in <FIG> are denoted by the same reference numerals, and detailed description thereof will be omitted.

The endoscope processor <NUM> of the third embodiment illustrated in <FIG> includes a processor control unit <NUM>-<NUM>.

The processor control unit <NUM>-<NUM> is different from the processor control unit <NUM>-<NUM> illustrated in <FIG> mainly in the target for which the amount of change is determined by an amount-of-change calculation unit <NUM> and the content of determination of the amount of change by an amount-of-change determination unit <NUM>.

The amount-of-change calculation unit <NUM> sequentially receives detection information of the detection target detected by the detector <NUM> and computes an amount of change in the detection target on the basis of the sequentially received detection information of the detection target. That is, the amount-of-change calculation unit <NUM> sets the detection target calculated based on the detection information from the detector <NUM> as a specific region and computes an amount of change in the region (specific region) corresponding to the detection target.

The amount of change in the detection target computed by the amount-of-change calculation unit <NUM> can be the amount of change in the position of the detection target that is consecutively detected.

When the detection information obtained by the detector <NUM> is, for example, the center coordinates of the detection target (lesion area) (the center coordinates can be computed regardless of whether the detection result is the presence or absence of a lesion in units of pixels or the coordinates of a rectangular shape), the coordinates in the current frame and the coordinates in the preceding frame are compared, and the amount of change is computed.

The amount-of-change determination unit <NUM> determines whether the amount of change in the detection target computed by the amount-of-change calculation unit <NUM> is within a threshold value. For example, when the detector <NUM> continuously detects the detection target and the amount of change in the center coordinates of the detection target is within <NUM> pixels, the amount-of-change determination unit <NUM> can determine that the amount of change in the detection target is within the threshold value.

In a state where the normal-light observation mode is used, the observation mode switching unit <NUM> automatically switches from the normal-light observation mode to the special-light observation mode when the continuous detection determination unit <NUM> determines that the detection target is continuously detected and the amount-of-change determination unit <NUM> determines that the amount of change in the detection target is within the threshold value.

After switching to the special-light observation mode, the observation mode switching unit <NUM> switches from the special-light observation mode to the normal-light observation mode when the amount-of-change determination unit <NUM> determines that the amount of change in the detection target is larger than the threshold value.

The amount of change in the detection target is not limited to the amount of change in the position (center coordinates) of the detection target, and may be the amount of change in the size of the detection target. For example, when the detection result obtained by the detector <NUM> is, for example, the values of the coordinates of four corners of a rectangular shape surrounding the detection target (lesion area), the area of the rectangular shape in the current frame and the area of the rectangular shape in the preceding frame are compared, and the amount of change is computed. For example, when the detector <NUM> continuously detects the detection target and the change in area is within <NUM>% of that of the preceding frame, the amount of change in the detection target can be determined to be within the threshold value.

<FIG> is a diagram illustrating input images (frames) sequentially captured by the endoscope <NUM>, detection results of the detection target detected from the input images, amounts of change in the center coordinates of the detection target, and amounts of change in the size (area) of the detection target.

In <FIG>, a frame at time t<NUM> is the current frame, and frames at time t-<NUM> to time t-<NUM> are previous frames.

The detector <NUM> detects the detection target from an input frame. In <FIG>, the detection target is not detected in the frame at time t-<NUM>, and the detection target is detected in the frames at the other times.

In <FIG>, an amount of change in the center coordinates of the detection target between consecutive frames is indicated by a vector (arrow), and an amount of change in the area of the detection target between consecutive frames is indicated by crescent-shaped regions. The amount of change in the detection target between consecutive frames includes: an amount of change between a detection region in which the detection target exists in the subsequent frame but the detection target does not exist in the preceding frame; and a detection region in which the detection target does not exist in the subsequent frame but the detection target exists in the preceding frame.

The amount-of-change calculation unit <NUM> illustrated in <FIG> computes the amount of change in the center coordinates of the detection target indicated by the vector or computes the area of the crescent-shaped regions, which is the amount of change in the area of the detection target. The amount-of-change determination unit <NUM> determines whether the amount of change in the center coordinates of the detection target or the amount of change in the area of the detection target computed by the amount-of-change calculation unit <NUM> is within a threshold value.

Since the detection target is not detected in the frame at time t-<NUM> and the detection target is detected in the frames at the other times, the continuous detection determination unit <NUM> can determine that the detection target is continuously detected.

In the embodiments described above, the first observation mode is the normal-light observation mode, and the second observation mode is the special-light observation mode. However, the present invention is not limited to this. For example, the first observation mode may be a first special-light observation mode, and the second observation mode may be a second special-light observation mode.

<FIG> is a conceptual diagram illustrating automatic switching of the observation mode when a first special-light observation mode using special light for BLI is set as the first observation mode and a second special-light observation mode using special light for LCI is set as the second observation mode.

As illustrated in <FIG>, when the detection target is continuously detected from first special-light images (BLI) sequentially captured in the first special-light observation mode, the observation mode switching unit <NUM> automatically switches from the first special-light observation mode to the second special-light observation mode, and second special-light images (LCI) are captured in the second special-light observation mode. When a certain period of time has elapsed after the switching of the observation mode to the second special-light observation mode, the observation mode is switched again to the first special-light observation mode by the observation mode switching unit <NUM>.

Accordingly, in a state where the first special-light observation mode is used, the observation mode is automatically switched from the first special-light observation mode to the second special-light observation mode when the detection target is continuously detected from sequentially captured images. Thus, it is possible to capture an image of the detection target in LCI having a different feature from BLI, and to reduce the burden of the operator's operation of switching the observation mode.

<FIG> is a flowchart illustrating an embodiment of a method for operating an endoscope apparatus according to the present invention and illustrates the processing procedures of the respective units of the endoscope apparatus <NUM> illustrated in <FIG>.

In <FIG>, first, the observation mode is set to the first observation mode, and the light source device <NUM>, which is controlled by the processor control unit <NUM>, emits white light as first illumination light (step S10). The endoscope <NUM> sequentially captures images (WL images) of the photographic subject irradiated with white light (WL) (step S12).

The detector <NUM> detects the detection target from the WL images captured by the endoscope <NUM> (step S14).

The continuous detection determination unit <NUM> (<FIG>) determines whether the detection target is continuously detected by the detector <NUM> (step S16). If it is determined that the detection target is not continuously detected (in the case of "No"), the process returns to step S10, and the processing of steps S10 to S16 is repeatedly performed.

On the other hand, if it is determined that the detection target is continuously detected (in the case of "Yes"), a transition to step S18 occurs. That is, in a state where the first observation mode is used, if it is determined that the detection target is continuously detected, the observation mode switching unit <NUM> switches to the second observation mode in which second illumination light (special light) is emitted.

In step S18, special light is emitted from the light source device <NUM>. The endoscope <NUM> sequentially captures images (special-light images) of the photographic subject irradiated with special light (step S20).

The processor control unit <NUM> determines whether a certain period of time has elapsed after the switching of the observation mode to the second observation mode (step S22). If it is determined that the certain period of time has not elapsed (in the case of "No"), a transition to step S18 occurs, and special-light images are continuously captured.

On the other hand, if it is determined that the certain period of time has elapsed (in the case of "Yes"), a transition to step S10 occurs. Accordingly, the observation mode is returned from the second observation mode to the first observation mode, and WL images are captured again.

In this embodiment, when imaging in the second observation mode continues for a certain period of time or when the detection target is not continuously detected after the observation mode is automatically switched from the first observation mode to the second observation mode, the observation mode is switched again to the first observation mode. However, this is not limiting, and when a still image is captured and recorded in accordance with the operation of the still-image capturing button after the observation mode is automatically switched from the first observation mode to the second observation mode, the observation mode may be switched to the first observation mode.

Alternatively, when the observation mode is automatically switched from the first observation mode to the second observation mode, a still image may be automatically captured and recorded, and, after that, the observation mode may be switched to the first observation mode. This makes it possible to automatically switch the observation mode and to automatically capture a still image. The burden of the operator's operation of the endoscope can further be reduced. In this case, in the second observation mode, the display device <NUM> or the like preferably notifies that a still image has been captured.

In this embodiment, furthermore, observation modes corresponding to different types of illumination light (illumination light for WL, BLI, LCI, and NBI) have been described. However, the observation modes are not limited to those corresponding to these types of illumination light. It is possible to appropriately set which of the two or more types of observation modes is set as each of the first observation mode and the second observation mode.

In this embodiment, furthermore, the endoscope apparatus <NUM> including the endoscope <NUM>, the light source device <NUM>, the endoscope processor <NUM>, and the detector <NUM> has been described. However, the present invention is not limited to the endoscope apparatus <NUM>, and may be implemented as the endoscope processor <NUM> not including the endoscope <NUM> as an element. In this case, the endoscope processor <NUM>, the light source device <NUM>, and the detector <NUM> may be integrated or separated.

In addition, the different types of illumination light are not limited to light emitted from LEDs of four colors. For example, a blue laser diode that emits blue laser light having a center wavelength of <NUM>, and a bluish violet laser diode that emits bluish violet laser light having a center wavelength of <NUM> may be used as light-emitting sources, and a YAG (Yttrium Aluminum Garnet) based fluorescent body may be irradiated with laser light of the blue laser diode and laser light of the bluish violet laser diode to emit light. When the fluorescent body is irradiated with blue laser light, the fluorescent body is excited to emit broadband fluorescent light, and a portion of the blue laser light passes through the fluorescent body as it is. The bluish violet laser light is transmitted without exciting the fluorescent body. Accordingly, adjusting the intensities of the blue laser light and the bluish violet laser light makes it possible to emit illumination light for WL, illumination light for BLI, and illumination light for LCI. In addition, emitting only the bluish violet laser light makes it possible to emit observation light having a center wavelength of <NUM>.

Furthermore, the present invention is also applicable to an endoscope apparatus including an endoscope (imaging unit) including a monochrome imaging element having no color filter, instead of the imaging element <NUM>, which is a color imaging element. When a normal-light image or a special-light image, which is a color endoscopic image, is to be acquired using the monochrome imaging element, the subject is sequentially illuminated with illumination light of different colors, and an image is captured for each illumination light (images are captured in a frame sequential manner).

For example, illumination light of different colors (R light, G light, B light, or V light) is sequentially emitted from the light source unit <NUM>, thereby capturing an R image, a G image, a B image, or a V image of a color corresponding to the R light, the G light, the B light, or the V light in a frame sequential manner by using the monochrome imaging element.

The endoscope processor can generate an image (for example, a WL image, BLI, LCI, NBI, or the like) corresponding to the first observation mode or the second observation mode from one or a plurality of images (an R image, a G image, a B image, or a V image) captured in a frame sequential manner. The image according to the observation mode, such as a WL image, BLI, LCI, or NBI, can be generated by adjusting the combination ratio of a plurality of images captured in a frame sequential manner. In the present invention, also in a case where images are captured in a frame sequential manner to generate, in accordance with an observation mode, images corresponding to the observation mode, the images are included in images of the photographic subject irradiated with illumination light emitted from a light source unit in accordance with the observation mode.

In addition, the detector <NUM> is not limited to a CNN, and may be, for example, a machine learning model other than a CNN, such as DBN (Deep Belief Network) or SVM (Support Vector Machine). That is, the detector <NUM> may be any device capable of detecting a detection target from an image.

Further, the hardware structure of the endoscope processor <NUM> and/or the detector <NUM> is implemented as the following various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor executing software (program) to function as various control units, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration is changeable after manufacture, a dedicated electric circuit, which is a processor having a circuit configuration specifically designed to cause specific processing to be executed, such as an ASIC (Application Specific Integrated Circuit), and so on.

A single processing unit may be configured as one of the various processors or as a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). Alternatively, a plurality of control units may be configured as a single processor. Examples of the configuration of a plurality of control units as a single processor include, first, a form in which, as typified by a computer such as a client or a server, the single processor is configured as a combination of one or more CPUs and software and the processor functions as the plurality of control units. The examples include, second, a form in which, as typified by a system on chip (SoC) or the like, a processor is used in which the functions of the entire system including the plurality of control units are implemented as one IC (Integrated Circuit) chip. As described above, the various control units are configured by using one or more of the various processors described above as a hardware structure.

Claim 1:
An endoscope apparatus comprising:
a light source unit (<NUM>) configured to emit first illumination light and second illumination light respectively corresponding to a first observation mode and a second observation mode;
an imaging unit (<NUM>) configured to capture an image of a photographic subject irradiated with the first illumination light or the second illumination light;
a detector (<NUM>) configured to detect a detection target from images sequentially captured by the imaging unit (<NUM>);
a continuous detection determination unit (<NUM>) configured to determine whether the detector (<NUM>) continuously detects the detection target; and
an observation mode switching unit (<NUM>) configured to switch between the first observation mode and the second observation mode;
an amount-of-change calculation unit (<NUM>) configured to calculate an amount of change in a specific region of images captured by the imaging unit (<NUM>); and
an amount-of-change determination unit (<NUM>) configured to determine whether the amount of change calculated by the amount-of-change calculation unit (<NUM>) is within a threshold value,
wherein in a state where the first observation mode is used, the observation mode switching unit (<NUM>) is configured to automatically switch to the second observation mode in response to the continuous detection determination unit (<NUM>) determining that the detection target is continuously detected and the amount-of-change determination unit (<NUM>) determining that the amount of change is within the threshold value, the combination of both indicating that the endoscope insertion section remains substantially stationary.