Patent ID: 12201262

EXAMPLE EMBODIMENT

Example Embodiment

In the following, an example embodiment of the present invention will be described with reference toFIGS.1to10.

Apparatus Configuration

First, a configuration of a surgery assistance apparatus in the present example embodiment will be described with reference toFIG.1.FIG.1is a diagram illustrating one example of the surgery assistance apparatus.

A surgery assistance apparatus1in the present example embodiment, which is illustrated inFIG.1, is an apparatus for improving the accuracy of endoscopic surgery by using image processing and identifying a target part image. As illustrated inFIG.1, the surgery assistance apparatus1includes a feature amount calculation unit2, a similarity degree calculation unit3, and an identification unit4.

Of these units, the feature amount calculation unit2calculates, from a human-body internal image captured using an endoscope, a feature amount of a target part image corresponding to a target part. The similarity degree calculation unit3calculates a similarity degree of the feature amount between different ones of the human-body internal images. The identification unit4identifies the target part image in each of the different human-body internal images if the similarity degree is greater than or equal to a predetermined value.

In such a manner, in the present example embodiment, a target part image in different human-body internal images captured using an endoscope is identified if a similarity degree of a feature amount between the different human-body internal images is greater than or equal to a predetermined value. Thus, the accuracy of endoscopic surgery can be improved because a target part image can be tracked.

Specifically, an observer conventionally visually tracks a target part image, and thus there are cases where the observer cannot refind a target part image if the observer loses track of the target part image. In such cases, there is a risk of a target part which is likely to become cancerous and need surgery, etc., of being overlooked. However, because a target part image can be identified using the surgery assistance apparatus in the present example embodiment, cases where a target part is overlooked can be reduced, regardless of the observer's skill and the like, and thus the accuracy of surgery can be improved.

Cases where the observer loses track of a target part image that is captured using an endoscope can be reduced because a target part image can be identified even if the target part image moves out of the frame or the target part image undergoes changes (a change in the size of the target part, a change in the image-capturing range, rotation of the image, etc.) as the endoscope is inserted and extracted, for example. Accordingly, the accuracy of endoscopic surgery can be improved.

Also, cases where the observer loses track of a target part image can be reduced because a target part image can be identified even if the target part is frequently concealed by parts of the human body other than the target part (for example, concealment of the target by intestinal folds, intestinal mucus, etc.) or by water ejected from the endoscope, etc. Accordingly, the accuracy of endoscopic surgery can be improved.

Furthermore, cases where the observer loses track of a target part image can be reduced because a target part image can be identified even in the case of non-rigid objects (objects without definite shapes) such as internal parts of the human body and the target part. Accordingly, the accuracy of endoscopic surgery can be improved.

System Configuration

Next, the configuration of the surgery assistance apparatus1in the present example embodiment will be specifically described with reference toFIG.2.FIG.2is a diagram illustrating one example of the surgery assistance apparatus and a system including the surgery assistance apparatus.

As illustrated inFIG.2, the system in the present example embodiment includes the surgery assistance apparatus1for executing image processing, an endoscope20, and an output device21. The surgery assistance apparatus1includes an output information generation unit22in addition to the feature amount calculation unit2, the similarity degree calculation unit3, and the identification unit4.

The endoscope20transmits, to the surgery assistance apparatus1connected to the endoscope20, a human-body internal image in which the inside of a human body is captured. For example, the endoscope20includes an insertion unit that is inserted into the human body, an image-capturing unit that is provided on the distal end-side of the insertion unit, an operation unit for controlling bending of the insertion unit, the capturing of images by the image-capturing unit, etc., and a connection unit that connects the endoscope20and the surgery assistance apparatus1. In addition to the image-capturing unit, the endoscope20also includes an illumination unit, a nozzle (nozzles) used for feeding air and water and for suction, a forceps port, and the like on the distal end-side of the insertion unit.

The output device21acquires, from the output information generation unit22, output information converted into formats that can be output, and outputs images, sound, etc., generated on the basis of the output information. The output device21, for example, includes an image display device utilizing liquid crystals, organic electroluminescence (EL), or a cathode ray tube (CRT), and further includes a sound output device such as a speaker, and the like. Note that the output device21may also be a printing device such as a printer.

The feature amount calculation unit2acquires a plurality of human-body internal images captured in time series by the endoscope20, and extracts a feature amount of a target part image corresponding to a target part from the human-body internal images. Furthermore, the feature amount calculation unit2includes a feature extraction unit23, a detection result calculation unit24, and a heatmap calculation unit25that extract feature amounts.

The feature extraction unit23extracts, from a human-body internal image, feature extraction information (feature amount f; feature vector) indicating features of a target part image. Specifically, local binary patterns (LBP) is one method for extracting local features of an image, and the extraction method is disclosed for example in the following document: “T. Ojala, M. Pietikainen, and D. Harwood, ‘Performance evaluation of texture measures with classification based on Lullback discrimination of distributions,’ in the Proceedings of IEEE International Conference on Pattern Recognition, 1994.”

FIG.3is a diagram illustrating one example of human-body internal images and images indicating feature amounts. Human-body internal images31(31a,31b,and31c) illustrated inFIG.3are images in which the inside of a colon is captured in time series, and target part images33(33aand33c) illustrated inFIG.3are images in which a tumor is captured. Note that, inFIG.3, while the target part images33aand33care captured in the human-body internal images31aand31c,no target part image is captured in the human-body internal image31bbecause the target part moved out of the frame.

InFIG.3, the feature extraction unit23acquires the plurality of human-body internal images that have been captured in time series, in the order of31a,31b,and31c.Note that the human-body internal images may be acquired continuously in the order in which human-body internal images in a moving image are captured.

Next, for each of the acquired human-body internal images31, the feature extraction unit23performs extraction of feature extraction information of a target part image33corresponding to the target part. InFIG.3, images32(32a,32b,and32c) are illustrated as images indicating the feature extraction information extracted from the human-body internal images31(31a,31b, and31c). Note that, in the images32inFIG.3, features of the target part are indicated by regions that are white or of a similar color, and the rest is indicated by black regions.

The detection result calculation unit24calculates detection result information (feature amounts r; feature vectors) using the feature extraction information (feature amount f; feature vector) extracted from each of the human-body internal images31. Specifically, the detection result calculation unit24applies processing, such as sliding window, for detecting a target part image33to each of the images32corresponding to the feature extraction information, and calculates the detection result information for each of the human-body internal images31.

FIGS.4and5are diagrams for describing the calculation of the detection result information. With respect to the images32corresponding to the feature extraction information, the detection result calculation unit24shifts a window41(window region) in the direction of the white arrow illustrated inFIG.4(in a predetermined order), and calculates the detection result information each time the window41is shifted (each time the window41is moved by a predetermined distance), for example. In addition, the detection result calculation unit24changes the size of the window41, uses the window41′ having a different size and shifts the window41′ in the predetermined order, and calculates the detection result information each time the window41′ is shifted. Note that the predetermined order is not limited to that in the direction (horizontal direction) of the white arrow illustrated inFIG.4.

The detection result information includes, for example, region information indicating the position and size of a detection target part and confidence information indicating the probability of the region of the detection target part corresponding to the target part. The region information and the confidence information are calculated using features inside the windows41and41′ (in the following, the window41′ is also referred to as the window41). The region information, for example, includes position information indicating the position of a rectangle circumscribing the target part, and size information indicating the size of the circumscribing rectangle.

If the images32are regarded as two-dimensional coordinate systems, position information indicating a position of the window41can be indicated by coordinates inside the window41. Center coordinates (X, Y) as illustrated inFIG.4are one example of coordinates inside the window41. If the images32are regarded as two-dimensional coordinate systems, size information indicating the size of the window41can be indicated by the width (W) and height (H) of the window41as illustrated inFIG.4.

Similarly to the region information of the window41, the region information of the detection result information can be indicated by the center coordinates (Xr, Yr) of the rectangle42circumscribing the target part, and size information indicating the size of the rectangle42(the width (Wr) and height (Hr) of the rectangle), as illustrated inFIG.5. Note that the center coordinates of the rectangle42circumscribing the target part may be indicated by the relative position to the window41, i.e., (Xr′=Xr−X, Yr′=Yr−Y). The confidence information is information indicating the probability (conf) of the region of the detection target part corresponding to the target part, which is calculated using the features inside the window41.

Note that the detection result information may, for example, be expressed in a form such as: feature vector r=(Xr′, Yr′, Wr, Hr, conf). Furthermore, the region information of the detection result information need not have a rectangular shape. The shape may be circular, elliptical, or the like, for example.

The heatmap calculation unit25calculates heatmap information (feature amount h; feature vector) using the feature extraction information (feature amount f; feature vector) extracted from each of the human-body internal images31. Specifically, the heatmap calculation unit25calculates the heatmap information by applying a semantic segmentation technique, for example, to the images32corresponding to the feature extraction information.

FIG.6is a diagram for describing the calculation of the heatmap information. The heatmap calculation unit25applies semantic segmentation to the images32corresponding to the feature extraction information, and uses different values to express coordinates corresponding to the target part images33and coordinates outside the target part images33. That is, the heatmap calculation unit25calculates images51(51a,51b,and51c) for indicating the heatmap information. Note that, in the images51inFIG.6, the target part is indicated by white regions, and the outside of the target part is indicated by black regions. Note that the coordinates may be expressed using pixels.

Furthermore, the feature extraction unit23may calculate feature extraction information (feature amount f′: feature vector) and detection result information (r) for each window41. In addition, the heatmap calculation unit25may calculate heatmap information (feature amount h′; feature vector) for each window41.

Note that the feature amount calculation unit2stores the human-body internal images31and the above-described feature amounts f, r, and h in an associated state.FIG.7is a diagram illustrating one example of a data structure of feature amounts. InFIG.7, in information61aindicating feature amounts, feature extraction information (feature amount fa), detection result information (feature amounts r1a, r2a, r3a, and r4a), and heatmap information (feature amount ha) that correspond to the human-body internal image31aare associated with one another. In information61b,feature extraction information (feature amount fb), detection result information (feature amounts r1b, r2b, r3b, and r4b), and heatmap information (feature amount hb) that correspond to the human-body internal image31bare associated with one another. In information61c,feature extraction information (feature amount fc), detection result information (feature amounts r1c, r2c, r3c, and r4c), and heatmap information (feature amount hc) that correspond to the human-body internal image31care associated with one another.

The similarity degree calculation unit3calculates a similarity degree using the feature extraction information (feature amount f), the detection result information (feature information r), and the heatmap information (feature amount h) in different human-body internal images31. Specifically, in the calculation of a similarity degree between human-body internal images31, a distance between feature vectors (similarity degree) is calculated using feature vectors of the feature extraction information (feature amount f), feature vectors of the detection result information (feature information r), and/or feature vectors of the heatmap information (feature amount h). Alternatively, the similarity degree may be expressed using linear combinations. The similarity degree calculation unit3calculates the similarity degree according to the methods described in (1) to (5).(1) The similarity degree calculation unit3calculates a similarity degree of the detection result information (r), a similarity degree of the feature extraction information (f), or a similarity degree of the heatmap information (h).(2) The similarity degree calculation unit3calculates a similarity degree of the position information (X, Y) included in the region information of the detection result information in each window, a similarity degree of the size information (W, H) included in the region information, a similarity degree of the confidence information (conf), a similarity degree of the feature extraction information (f′), or a similarity degree of the heatmap information (h′).(3) The similarity degree calculation unit3combines two or more out of the degrees of similarity indicated in (1) and (2), and sets the total sum of the degrees of similarity as a similarity degree. That is, the total sum of two or more out of the similarity degree of the detection result information (r), the similarity degree of the feature extraction information (f), the similarity degree of the heatmap information (h), the similarity degree of the position information (X, Y) included in the region information, the similarity degree of the size information (W, H) included in the region information, the similarity degree of the confidence information (conf), the similarity degree of the feature extraction information (f′), and the similarity degree of the heatmap information (h′) is set as a similarity degree.(4) The similarity degree calculation unit3calculates the similarity degree of the confidence information (conf), and if the calculated similarity degree is greater than or equal to a predetermined confidence value that is set in advance, calculates a similarity degree of feature amounts between the different human-body internal images31(one or more of the degrees of similarity indicated in (1) and (2) described above). The predetermined confidence value is a determination value calculated through experimentation, simulation, machine learning, etc., and is stored in a storage unit provided in the surgery assistance apparatus or outside the surgery assistance apparatus. Alternatively, the maximum value of confidence information calculated from a plurality of human-body internal images31acquired in the past may be set as the predetermined confidence value. By adopting such a configuration, the calculation of a similarity degree becomes unnecessary in a case where the same target part image33is continuously captured in human-body internal images31.(5) The similarity degree calculation unit3calculates the similarity degree of the confidence information (conf), and if the calculated similarity degree is greater than or equal to the predetermined confidence value, calculates the similarity degree of the region information (X, Y, W, H). Next, the similarity degree calculation unit3calculates the similarity degree of the feature extraction information (f) if the similarity degree of the region information is greater than or equal to a predetermined region value. Then, the similarity degree calculation unit3calculates the similarity degree of the heatmap information (h) if the similarity degree of the feature extraction information is greater than or equal to a predetermined feature extraction value.

Note that each of the predetermined region value and the predetermined feature extraction value is a determination value calculated through experimentation, simulation, machine learning, etc., and is stored in the storage unit provided in the surgery assistance apparatus or outside the surgery assistance apparatus. By adopting such a configuration, the calculation of a similarity degree becomes unnecessary in a case where the same target part image33is continuously captured in human-body internal images31, and the accuracy of degrees of similarity between human-body internal images31can be improved.

Furthermore, in the calculation of the similarity degree of the region information in (5), the similarity degree may be calculated using either the position information (X, Y) or the size information (W, H) in the region information.

The identification unit4identifies a target part image in each of the different human-body internal images31if the similarity degree is greater than or equal to a predetermined value. Specifically, if the similarity degree calculated according to one of (1) to (5) is greater than or equal to the predetermined value, the identification unit4associates the target part images33in the human-body internal images31with one another and stores the target part images33in the storage unit. The identification unit4performs the identification according to the methods indicated in (1′) to (5′).(1′) The identification unit4identifies the target part images33if the similarity degree of the detection result information (r) is greater than or equal to a predetermined detection result value, the similarity degree of the heatmap information (h) is greater than or equal to a predetermined heatmap value, or the similarity degree of the feature extraction information (f) is greater than or equal to the predetermined feature extraction value. The predetermined heatmap value is a determination value calculated through experimentation, simulation, machine learning, etc., and is stored in the storage unit provided in the surgery assistance apparatus1or outside the surgery assistance apparatus1.(2′) The identification unit4identifies the target part images33if the similarity degree of the position information (X, Y) included in the region information, the similarity degree of the size information (W, H) included in the region information, the similarity degree of the confidence information (conf), the similarity degree of the feature extraction information (f′), and the similarity degree of the heatmap information (h′) are all greater than or equal to a predetermined value set for each similarity degree. Alternatively, the identification unit4may identify the target part images33if any of the degrees of similarity is greater than or equal to the predetermined value.(3′) The identification unit4sets the total sum of two or more out of the degrees of similarity indicated in (1) and (2) as the similarity degree, and identifies the target part images33if the similarity degree is greater than or equal to a predetermined value that is set for each combination.(4′) The identification unit4carries out the identification indicated in (1′), (2′), or (3′) if the similarity degree of the confidence information (conf) is greater than or equal to the predetermined confidence value.(5′) The identification unit4identifies that the target part images33in the different human-body internal images31are the same if the similarity degree of the heatmap information (h), which is indicated in (5), is greater than or equal to the predetermined heatmap value.

By adopting such configurations, the target part image33ccaptured in the latest human-body internal image31cand the target part image33ain the human-body internal image31acaptured in the past can be associated with one another even if the human-body internal image31b,in which a target part image is not captured, is present between the human-body internal image31cand the human-body internal image31a,that is, even if the observer loses track of the target part image33, for example.

The output information generation unit22generates output information indicating that target part images33have been identified if target part images33are identified during surgery, and transmits the generated output information to the output device21. The output device21acquires the output information, and then outputs, on the basis of the output information, at least one of a screen and sound indicating that target part images33have been identified.

FIG.8is a diagram illustrating one example of display in which target part images are identified. InFIG.8, coloring, etc., is applied to the portions of the identified target part images33in the human-body internal images31aand31c,and indicators that the identified target part images33correspond to the same tumor are also displayed. Note that no indicator regarding identification is displayed for the human-body internal image31bbecause the human-body internal image31bdoes not include an identified target part image. InFIG.8, the target part images33are colored (the shaded areas), and indicators that the target part images33correspond to a “tumor A” are displayed. Note that the display method is not limited to that illustrated inFIG.8.

Apparatus Operations

Next, the operations of the surgery assistance apparatus in the example embodiment of the present invention will be described with reference toFIG.9.FIG.9is a diagram illustrating one example of the operations of the surgery assistance apparatus.FIGS.2to8will be referred to as needed in the following description. Furthermore, in the present example embodiment, a surgery assistance method is implemented by causing the surgery assistance apparatus to operate. Accordingly, the following description of the operations of the surgery assistance apparatus is substituted for the description of the surgery assistance method in the present example embodiment.

In step A1, the feature amount calculation unit2acquires human-body internal images31that have been captured in time series by the endoscope20. Next, in step A2, the feature amount calculation unit2calculates feature amounts of target part images33corresponding to a target part from the human-body internal images31captured using the endoscope20. SeeFIG.3.

Specifically, the feature amount calculation unit2(feature extraction unit23) extracts, from the human-body internal images31, feature extraction information (feature amounts f; feature vectors) indicating features of the target part images33. Next, the feature amount calculation unit2(detection result calculation unit24) calculates detection result information (feature amounts r; feature vectors) using the feature extraction information (feature amount f; feature vector) extracted from each of the human-body internal images31. For example, processing, such as sliding window, for detecting a target part image33is applied to images32corresponding to the feature extraction information, and the detection result information is calculated for each of the human-body internal images31. SeeFIGS.4and5.

Alternatively, the feature amount calculation unit2(heatmap calculation unit25) calculates heatmap information (feature amount h; feature vector) using the feature extraction information (feature amount f; feature vector) extracted from each of the human-body internal images31. The heatmap information is calculated by applying semantic segmentation to the images32corresponding to the feature extraction information, for example. SeeFIG.6.

Note that the feature amount calculation unit2stores the human-body internal images31and the above-described feature amounts f, r, and h in an associated state. SeeFIG.7.

In step A3, the similarity degree calculation unit3calculates a similarity degree using the feature extraction information (feature amount f), the detection result information (feature information r), and/or the heatmap information (feature amount h) in the latest human-body internal image31and a human-body internal image31captured before the latest human-body internal image31. Specifically, in the calculation of the similarity degree between the human-body internal images31, a distance between feature vectors (similarity degree) is calculated using feature vectors of the feature extraction information (feature amount f), feature vectors of the detection result information (feature information r), and/or feature vectors of the heatmap information (feature amount h). Alternatively, the similarity degree may be expressed using linear combinations. The similarity degree calculation unit3calculates the similarity degree according to the methods described in (1) to (5).

For example, the similarity degree calculation unit3calculates a similarity degree between the latest human-body internal image31cand the human-body internal image31aor31bcaptured before the latest human-body internal image31c.

In step A4, the identification unit4identifies the target part image33in the latest human-body internal image31and the target part image33in the human-body internal image31captured before the latest human-body internal image31if the similarity degree is greater than or equal to a predetermined value. Specifically, if the similarity degree calculated according to one of (1) to (5) is greater than or equal to the predetermined value, the identification unit4associates the target part images33in the human-body internal images31with one another and stores the target part images33in the storage unit. The identification unit4performs the identification according to the methods indicated in (1′) to (5′).

In step A5, the output information generation unit22generates output information indicating that target part images33have been identified if target part images33are identified during surgery, and transmits the generated output information to the output device21.

In step A6, the output device21acquires the output information, and then outputs, on the basis of the output information, at least one of a screen and sound indicating that target part images33have been identified. SeeFIG.8.

In step A7, the surgery assistance apparatus1terminates the identification processing illustrated in steps A1to A7if an instruction to terminate the identification processing is acquired (Yes). The surgery assistance apparatus1moves on to the processing in step A1if the identification processing illustrated in steps A1to A7is to continue (No).

Effects of Example Embodiment

As described above, according to the present example embodiment, target part images33in different human-body internal images31captured using an endoscope20are identified if a similarity degree of feature amounts of the different human-body internal images31is greater than or equal to a predetermined value. Thus, because target part images33can be identified, cases where a target part is overlooked can be reduced, regardless of the observer's skill and the like, and thus the accuracy of surgery can be improved.

Cases where the observer loses track of a target part image that is captured using an endoscope can be reduced because a target part image can be identified even if the target part image moves out of the frame or changes as the endoscope is inserted and extracted, for example. Accordingly, the accuracy of endoscopic surgery can be improved.

Also, cases where the observer loses track of a target part image can be reduced because a target part image can be identified even if the target part is frequently concealed by parts of the human body other than the target part (for example, concealment of the target by intestinal folds, intestinal mucus, etc.) or by water ejected from the endoscope, etc. Accordingly, the accuracy of endoscopic surgery can be improved.

Furthermore, cases where the observer loses track of a target part image can be reduced because a target part image can be identified even in the case of non-rigid objects (objects without definite shapes) such as internal parts of the human body and the target part. Accordingly, the accuracy of endoscopic surgery can be improved.

Program

It suffices for the program in the example embodiment of the present invention to be a program that causes a computer to carry out steps A1to A7illustrated inFIG.9. By installing this program on a computer and executing the program, the surgery assistance apparatus and the surgery assistance method in the present example embodiment can be realized. In this case, the processor of the computer functions and performs processing as the feature amount calculation unit2, the similarity degree calculation unit3, the identification unit4, and the output information generation unit22.

Also, the program in the present example embodiment may be executed by a computer system formed from a plurality of computers. In this case, the computers may each function as one of the feature amount calculation unit2, the similarity degree calculation unit3, the identification unit4, and the output information generation unit22for example.

Physical Configuration

Here, a computer that realizes the surgery assistance apparatus by executing the program in the example embodiment will be described with reference toFIG.10.FIG.10is a diagram illustrating one example of a computer realizing the surgery assistance apparatus.

As illustrated inFIG.10, a computer110includes a CPU111, a main memory112, a storage device113, an input interface114, a display controller115, a data reader/writer116, and a communication interface117. These components are connected via a bus121so as to be capable of performing data communication with one another. Note that the computer110may include a graphics processing unit (GPU) or a field-programmable gate array (FPGA) in addition to the CPU111or in place of the CPU111.

The CPU111loads the program (codes) in the present example embodiment, which is stored in the storage device113, onto the main memory112, and performs various computations by executing these codes in a predetermined order. The main memory112is typically a volatile storage device such as a dynamic random access memory (DRAM). Furthermore, the program in the present example embodiment is provided in a state such that the program is stored in a computer readable recording medium120. Note that the program in the present example embodiment may also be a program that is distributed on the Internet, to which the computer110is connected via the communication interface117.

In addition, specific examples of the storage device113include semiconductor storage devices such as a flash memory, in addition to hard disk drives. The input interface114mediates data transmission between the CPU111and input equipment118such as a keyboard and a mouse. The display controller115is connected to a display device119, and controls the display performed by the display device119.

The data reader/writer116mediates data transmission between the CPU111and the recording medium120, and executes the reading of the program from the recording medium120and the writing of results of processing in the computer110to the recording medium120. The communication interface117mediates data transmission between the CPU111and other computers.

Furthermore, specific examples of the recording medium120include a general-purpose semiconductor storage device such as a CompactFlash (registered trademark, CF) card or a Secure Digital (SD) card, a magnetic recording medium such as a flexible disk, and an optical recording medium such as a compact disk read-only memory (CD-ROM).

Supplementary Note

In relation to the above example embodiment, the following Supplementary notes are further disclosed. While a part of or the entirety of the above-described example embodiment can be expressed by (Supplementary note 1) to (Supplementary note 18) described in the following, the present invention is not limited to the following description.

Supplementary Note 1

A surgery assistance apparatus including:

a feature amount calculation unit configured to calculate, from a human-body internal image captured using an endoscope, a feature amount of a target part image corresponding to a target part;

a similarity degree calculation unit configured to calculate a similarity degree of the feature amount between different ones of the human-body internal images; and

an identification unit configured to identify the target part image in each of the different human-body internal images if the similarity degree is greater than or equal to a predetermined value.

Supplementary Note 2

The surgery assistance apparatus according to Supplementary note 1, wherein

the feature amount includes feature extraction information indicating features of the target part image extracted from the human-body internal image, region information indicating the position and size of a window region with which a part or an entirety of an image corresponding to the target part is detected from the feature extraction information, confidence information indicating the probability of the image in the window region being an image corresponding to the target part, heatmap information calculated using the feature extraction information, or information that is a combination of two or more out of the feature extraction information, region information, confidence information, and heatmap information.

Supplementary Note 3

The surgery assistance apparatus according to Supplementary note 2, wherein

the similarity degree calculation unit calculates the similarity degree of the feature amount between the different human-body internal images if the confidence information is greater than or equal to a predetermined confidence value.

Supplementary Note 4

The surgery assistance apparatus according to Supplementary note 2 or 3, wherein

the similarity degree calculation unit calculates the similarity degree of the region information if the similarity degree of the confidence information is greater than or equal to a predetermined confidence value, calculates the similarity degree of the feature extraction information if the similarity degree of the region information is greater than or equal to a predetermined region value, and calculates the similarity degree of the heatmap information if the similarity degree of the feature extraction information is greater than or equal to a predetermined feature extraction value, and

the identification unit identifies that the target part images in the different human-body internal images are the same if the similarity degree of the heatmap information is greater than or equal to a predetermined heatmap value.

Supplementary Note 5

The surgery assistance apparatus according to any one of Supplementary notes 1 to 4, wherein

the target part is a tumor, and the target part image is an image in which the tumor is captured.

Supplementary Note 6

The surgery assistance apparatus according to any one of Supplementary notes 1 to 5, further including

an output device configured to, if the target part image is identified during surgery, output information indicating that the target part image has been identified to an observer during the surgery.

Supplementary Note 7

A surgery assistance method including:(a) a step of calculating, from a human-body internal image captured using an endoscope, a feature amount of a target part image corresponding to a target part;(b) a step of calculating a similarity degree of the feature amount between different ones of the human-body internal images; and(c) a step of identifying the target part image in each of the different human-body internal images if the similarity degree is greater than or equal to a predetermined value.

Supplementary Note 8

The surgery assistance method according to Supplementary note 7, wherein

the feature amount includes feature extraction information indicating features of the target part image extracted from the human-body internal image, region information indicating the position and size of a window region with which a part or an entirety of an image corresponding to the target part is detected from the feature extraction information, confidence information indicating the probability of the image in the window region being an image corresponding to the target part, heatmap information calculated using the feature extraction information, or information that is a combination of two or more out of the feature extraction information, region information, confidence information, and heatmap information.

Supplementary Note 9

The surgery assistance method according to Supplementary note 8, wherein

in the (b) step, the similarity degree of the feature amount between the different human-body internal images is calculated if the confidence information is greater than or equal to a predetermined confidence value.

Supplementary Note 10

The surgery assistance method according to Supplementary note 8 or 9, wherein

in the (b) step, the similarity degree of the region information is calculated if the similarity degree of the confidence information is greater than or equal to a predetermined confidence value, the similarity degree of the feature extraction information is calculated if the similarity degree of the region information is greater than or equal to a predetermined region value, and the similarity degree of the heatmap information is calculated if the similarity degree of the feature extraction information is greater than or equal to a predetermined feature extraction value, and

in the (c) step, the target part images in the different human-body internal images are identified as being the same if the similarity degree of the heatmap information is greater than or equal to a predetermined heatmap value.

Supplementary Note 11

The surgery assistance method according to any one of Supplementary notes 7 to 10, wherein

the target part is a tumor, and the target part image is an image in which the tumor is captured.

Supplementary Note 12

The surgery assistance method according to any one of Supplementary notes 7 to 11, further including

(d) a step of, if the target part image is identified during surgery, outputting information indicating that the target part image has been identified to an observer during the surgery.

Supplementary Note 13

A computer readable recording medium that includes recorded thereon a surgery assistance program that causes a computer to carry out:(a) a step of calculating, from a human-body internal image captured using an endoscope, a feature amount of a target part image corresponding to a target part;(b) a step of calculating a similarity degree of the feature amount between different ones of the human-body internal images; and(c) a step of identifying the target part image in each of the different human-body internal images if the similarity degree is greater than or equal to a predetermined value.

Supplementary Note 14

The computer readable recording medium that includes the surgery assistance program recorded thereon according to Supplementary note 13, wherein

the feature amount includes feature extraction information indicating features of the target part image extracted from the human-body internal image, region information indicating the position and size of a window region with which a part or an entirety of an image corresponding to the target part is detected from the feature extraction information, confidence information indicating the probability of the image in the window region being an image corresponding to the target part, heatmap information calculated using the feature extraction information, or information that is a combination of two or more out of the feature extraction information, region information, confidence information, and heatmap information.

Supplementary Note 15

The computer readable recording medium that includes the surgery assistance program recorded thereon according to Supplementary note 14, wherein

in the (b) step, the similarity degree of the feature amount between the different human-body internal images is calculated if the confidence information is greater than or equal to a predetermined confidence value.

Supplementary Note 16

The computer readable recording medium that includes the surgery assistance program recorded thereon according to Supplementary note 14 or 15, wherein

in the (b) step, the similarity degree of the region information is calculated if the similarity degree of the confidence information is greater than or equal to a predetermined confidence value, the similarity degree of the feature extraction information is calculated if the similarity degree of the region information is greater than or equal to a predetermined region value, and the similarity degree of the heatmap information is calculated if the similarity degree of the feature extraction information is greater than or equal to a predetermined feature extraction value, and

in the (c) step, the target part images in the different human-body internal images are identified as being the same if the similarity degree of the heatmap information is greater than or equal to a predetermined heatmap value.

Supplementary Note 17

The computer readable recording medium that includes the surgery assistance program recorded thereon according to any one of Supplementary notes 13 to 16, wherein

the target part is a tumor, and the target part image is an image in which the tumor is captured.

Supplementary Note 18

The computer readable recording medium that includes the surgery assistance program recorded thereon according to any one of Supplementary notes 13 to 17, wherein

the surgery assistance program further causes the computer to carry out

(d) a step of, if the target part image is identified during surgery, outputting information indicating that the target part image has been identified to an observer during the surgery.

INDUSTRIAL APPLICABILITY

In such a manner, according to the present invention, the accuracy of endoscopic surgery can be improved by identifying a target part image. The present invention is useful in fields in which endoscopic surgery is necessary.

REFERENCE SIGNS LIST

1Surgery assistance apparatus2Feature amount calculation unit3Similarity degree calculation unit4Identification unit20Endoscope21Output device22Output information generation unit23Feature extraction unit24Detection result calculation unit25Heatmap calculation unit110Computer111CPU112Main memory113Storage device114Input interface115Display controller116Data reader/writer117Communication interface118Input equipment119Display device120Recording medium121Bus