Patent Description:
A technology is available which performs, in order to detect a target from moving image data obtained by photographing a target, segmentation using a neural network (NNW) for each of frame images included in the moving image data.

As a first method, a technology is available in which a combined image (for example, an optical flow) representative of a motion of a target is inputted to one of two NNWs like a <NUM>-way network and segmentation of the target is performed using a segmentation network for a still picture of the other one of the two NNWs.

As a second method, a technology is available in which several preceding and succeeding frame images of moving image data are inputted together to an NNW to perform
segmentation of a target.

For example, a case is assumed in which moving image data is such moving image data that it includes much noise and indicates a small movement of a target like moving image data of an ultrasonography video or a surveillance video photographed by a surveillance camera and having comparatively-low picture quality. In the case where a target is detected from such moving image data as just described including a shape of the target, the first and second methods described above sometimes suffer from such inconveniences as described below.

The first method is suitable for segmentation of a target with movement like a running vehicle, in other words, a target whose position changes between image frames, because a combined image (for example, an optical flow) representative of a movement of the target is used as one of inputs. However, the first method is not suitable for detailed segmentation specified for a target region such as moving image data obtained by photographing a target whose change in position is comparatively small.

In the second method, it is difficult to perform training taking a frame image of a target for which segmentation is to be performed into consideration. Therefore, for example, even if a target does not appear in a target frame image, if the target appears in preceding or succeeding frame images of the target frame image, then there is the possibility that the NNW detects the target in the target frame image in error.

In this manner, it is considered that both of the first and second methods described above are low in robustness against noise of a frame image of moving image data in object detection of the frame image.

According to one aspect, it is an object of the present technology to improve the robustness against noise of a frame image of moving image data in object detection of the frame image.

<NPL>) describes the design, training, and evaluation of a deep neural network for removing noise from medical fluoroscopy videos. The method is said to be able to deliver a result quickly enough to be used in real-time scenarios, and this is said to be unlike other current standard techniques for video denoising. The authors say the described method is able to produce results of a similar quality to the existing industry-standard denoising techniques.

<CIT>describes video object segmentation using a neural network and the training of the neural network. The neural network both detects a target object in the current frame based on a reference frame and a reference mask that define the target object and propagates the segmentation mask of the target object for a previous frame to the current frame to generate a segmentation mask for the current frame. The neural network is pre-trained using synthetically generated static training images and is then fine-tuned using training videos.

<NPL>) describes a consideration that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. The authors explicitly learn this decomposed (factorised) representation of imaging data, focusing in particular on cardiac images. The authors propose Spatial Decomposition Network, SDNet, which factorises 2D medical images into spatial anatomical factors and non-spatial imaging factors. The authors say the paper demonstrate that this high-level representation is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis. The authors say their model can match the performance of fully supervised segmentation models, using only a fraction of the labelled images. The authors say that the factorised representation also benefits from supervision obtained either when auxiliary tasks are used to train the model in a multi-task setting (e.g. regressing to known cardiac indices), or when aggregating multi modal data from different sources (e.g. pooling together MRI and CT data). To explore the properties of the learned factorisation, the authors perform latent-space arithmetic and state that they can synthesise CT from MR and vice versa, by swapping the modality factors. The authors also say they demonstrate that the factor holding image specific information can be used to predict the input modality with high accuracy.

According to an aspect of the present invention, there is provided a training program as set out in Claim <NUM>.

According to another aspect of the present invention, there is provided a training method as set out in Claim <NUM>.

According to another aspect of the present invention, there is provided an information processing apparatus as set out in Claim <NUM>. data obtained by photographing a target and a plurality of annotation images each indicative of a region of the target in each of a plurality of frame images included in the moving image data; and executing a training process using the training data. The training process includes: detecting the target included in the plurality of frame images; inputting a combined image to an auto-encoder, the combined image being obtained by combining a plurality of partial images including the target and a plurality of peripheral region images of the target, the plurality of partial images and plurality of peripheral region images being detected in a given number of preceding and succeeding second frame images in a time series of the moving image data of a first frame image from among the plurality of frame images; inputting a partial image, in the plurality of partial images, corresponding to the first frame image to a neural network that performs a segmentation process for an image; and performing parameter update of the auto-encoder and the neural network, based on a difference between a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and a partial image of the annotation image indicative of a region of the target in the first frame image.

According to the one aspect, the robustness of a frame image of moving image data against noise in object detection of the frame image can be improved.

There is described herein a training program that causes a computer to execute a process includes: acquiring training data including moving image data obtained by photographing a target and a plurality of annotation images each indicative of a region of the target in each of a plurality of frame images included in the moving image data; and executing a training process using the training data. The training process includes: detecting the target included in the plurality of frame images; inputting a combined image to an auto-encoder, the combined image being obtained by combining a plurality of partial images including the target and a plurality of peripheral region images of the target, the plurality of partial images and plurality of peripheral region images being detected in a given number of preceding and succeeding second frame images in a time series of the moving image data of a first frame image from among the plurality of frame images; inputting a partial image, in the plurality of partial images, corresponding to the first frame image to a neural network that performs a segmentation process for an image; and performing parameter update of the auto-encoder and the neural network, based on a difference between a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and a partial image of the annotation image indicative of a region of the target in the first frame image.

In the following, an embodiment of the present technology is described with reference to the drawings. However, the embodiment described below is illustrative to the end, and there is no intention to eliminate various modifications and applications of the technology that are not specified in the following. It is to be noted that, unless otherwise specified, in the drawings referred to in the following description of the embodiment, same or like elements are denoted by like reference characters.

<FIG> is a view illustrating an example of a training process by an information processing apparatus <NUM> according to the embodiment. It is to be noted that, in the description with reference to <FIG>, "moving image data" signifies video data such as an echo image obtained by photographing a comparatively small target with respect to a size of a frame image and including noise, and "target" signifies a predetermined part of a photographing target of an echo image.

The information processing apparatus <NUM> acquires training data including moving image data obtained by photographing a target and multiple annotation images indicative of a region of the target in each of multiple frame images included in the moving image data. Then, the information processing apparatus <NUM> executes a training process using the training data.

For example, as depicted in <FIG>, the information processing apparatus <NUM> executes processes of (a) to (d) described below for each of multiple frame images in the training process. As an example, the processes of (a) to (d) may be repeatedly executed, changing while a first frame image that is a target (noticed) frame image in a time series order of the moving image data. In the following description, an entire frame image is sometimes referred to as "total image".

For example, as depicted in <FIG>, the information processing apparatus <NUM> may detect a target included in (t-n)th to (t+n)th total images 1a in a time series of moving image data from among multiple frame images. The symbol "t" is a variable indicative of the first frame image. n indicates an integer equal to or greater than one and is an example of a given number (predetermined number). Further, (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th frame images are an example of n preceding and n succeeding second frame images 1a of the first frame image 1a. It is to be noted that, in <FIG> and the following drawings, the (t-n)th to (t+n)th total images are denoted by "total images (t±n)" for simplified illustration.

It is to be noted that the information processing apparatus <NUM> may detect the target included in an annotation image 1b of the (t)th total image 1a included in the training data in addition to the foregoing.

The information processing apparatus <NUM> may detect the target from the total image 1a and the annotation image 1b, for example, by an object detection unit <NUM>. The object detection unit <NUM> may be a trained object detection model generated, for example, using a dataset of the training data for specifying a region of the target included in an input image, and may be an object detection NNW such as a YOLO as an example.

The object detection unit <NUM> may output target peripheral images 2a to 2c and a target peripheral annotation image 2d as a result of the detection of the target.

The target peripheral image 2a is multiple partial images including a target and a peripheral region of the target detected in n frame images preceding to the first frame image 1a, namely, in the (t-n)th to (t-<NUM>)th second frame images 1a.

The target peripheral image 2b is multiple partial images including a target and a peripheral region of the target detected in n frame images succeeding the first frame image 1a, namely, in the (t+<NUM>)th to (t+n)th second frame images 1a.

It is to be noted that, in <FIG> and the succeeding figures, for simplified illustration, the (t-n)th to (t-<NUM>)th target peripheral images 2a and (t+<NUM>)th to (t+n)th target peripheral images 2b are denoted by "target peripheral images (t-n)" and "target peripheral images (t+n)", respectively.

The target peripheral image 2c is a partial image including a target and a peripheral region of the target that are detected in the (t)th first frame image 1a.

The target peripheral annotation image 2d is a partial image including a target and a peripheral region of the target that are detected in the annotation image 1b, and is, for example, a partial image obtained by cutting out a partial region the same as that of the target peripheral image 2c from the annotation image 1b.

(b) The information processing apparatus <NUM> inputs a combined image obtained by combining the target peripheral images 2a and 2b to an auto-encoder <NUM>.

For example, the information processing apparatus <NUM> may combine n target peripheral images 2a and 2b by lining up them in a channel direction.

The auto-encoder <NUM> is an example of a support module <NUM>. For example, as the auto-encoder <NUM> is exemplified by an NNW in which the number of units in an intermediate layer is small in comparison with the number of units of each of an input layer and an output layer, such as an auto encoder.

(c) the information processing apparatus <NUM> inputs the target peripheral image 2c to a segmentation unit <NUM> that performs a segmentation process for an image.

The segmentation unit <NUM> is an example of a segmentation module <NUM>. Although, as the segmentation unit <NUM>, various NNWs for segmentation are available, in the embodiment, for example, the U-Net is used. It is to be noted that the segmentation unit <NUM> is not limited to the U-Net, and may be a different neural network that executes Semantic Segmentation or maybe a neural network that uses a segmentation method other than the Semantic Segmentation.

Each of the auto-encoder <NUM> and the segmentation unit <NUM> is an NNW that is a target to be trained in a training process.

(d) The information processing apparatus <NUM> performs parameter update of the auto-encoder <NUM> and the segmentation unit <NUM> on the basis of a difference between a combined output-image obtained by combining an output image from the auto-encoder <NUM> and an output image from the segmentation unit <NUM> and the target peripheral annotation image 2d.

The information processing apparatus <NUM> generates the combined output-image by adding the output image from the auto-encoder <NUM> and the output image from the segmentation unit <NUM> for each pixel, for example, by an adding unit <NUM>. The combined output-image is an example of a segmented image. Then, the information processing apparatus <NUM> may input the target peripheral annotation image 2d, for example, to the adding unit <NUM> and may train the auto-encoder <NUM> and the segmentation unit <NUM> by backward error propagation or the like on the basis of the difference between the combined output-image and the target peripheral annotation image 2d.

Consequently, the information processing apparatus <NUM> trains a support module <NUM> that outputs complementation information based on a context of preceding and succeeding images of the first frame image 1a on the basis of the target peripheral annotation image 2d. Further, the information processing apparatus <NUM> can train the segmentation module <NUM> on the basis of the target peripheral annotation image 2d.

Accordingly, in object detection of the frame image 1a of the moving image data, even if noise is included in the frame image 1a, a network for outputting a segmentation result focusing on the first frame image 1a can be constructed, considering the preceding and succeeding images of the first frame image 1a.

From the foregoing, with the information processing apparatus <NUM>, robustness against noise of the frame image 1a in object detection of the frame image 1a of the moving image data can be improved.

Further, the information processing apparatus <NUM> according to the embodiment includes a feature outputting unit <NUM> in the support module <NUM> as exemplified in <FIG>. The feature outputting unit <NUM> may be a trained model generated using a dataset of an image different from the frame image 1a obtained by photographing a target and may be a trained model for estimating a label to an input image.

As the feature outputting unit <NUM>, for example, VGG-Backbone is available. The VGG-Backbone may be, for example, an NNW equivalent to a trained NNW such as a VGG from which an output layer is removed. As an example, the VGG-Backbone may be an NNW including a convolution layer and a pooling layer with a fully connected layer as an outputting later removed from a VGG. It is to be noted that the VGG is an example of a trained NNW usable in the embodiment. The trained NNW to be utilized in the embodiment is not limited only to the VGG (or the VGG-Backbone).

For example, the information processing apparatus <NUM> depicted in <FIG> may input a feature relating to the entire first frame image 1a obtained by inputting the (t)th first frame image (total image) 1a to the feature outputting unit <NUM>, in other words, a context relating to the entire image, to an intermediate layer of the auto-encoder <NUM>.

It is to be noted that the intermediate layer of the auto-encoder <NUM> may be a bottleneck of the auto-encoder <NUM> and may be, as an example, a layer in which the size (vertical and horizontal sizes) of an image to be processed is in the minimum from among layers of the auto-encoder <NUM>.

Consequently, the auto-encoder <NUM> to which a combined image of the target peripheral images 2a and 2b is inputted can make use of the context of the entire image from the feature outputting unit <NUM> in addition to the context of preceding and succeeding images of the first frame image 1a. Accordingly, the accuracy of the output from the auto-encoder <NUM> can be enhanced.

In the following description, a case in which the training process and the estimation process by the information processing apparatus <NUM> are utilized for decision of presence or absence of defect in a site called interventricular septum of the heart in ultrasonographic image diagnosis is described as an example.

As exemplified in <FIG>, the interventricular septum included in an ultrasonographic image sometimes seems to have a defect by noise. Noise can appear at a position different for each frame image of moving image data. Accordingly, in order to decide presence or absence of a defect in the interventricular septum included in the first frame image, it is important to perform segmentation for a portion of the interventricular septum using information of preceding and succeeding second frame images of the first frame image as exemplified in <FIG>.

<FIG> is a block diagram depicting an example of a functional configuration of a server <NUM> according to the embodiment. The server <NUM> is an apparatus that trains an NNW group for estimating a target from a frame image and estimates a target using the NNW group, and is an example of the information processing apparatus <NUM> depicted in <FIG>.

In the description of the embodiment, it is assumed that the target of a segmentation target is an interventricular septum and the image for which segmentation is to be performed is an ultrasonographic image such as an echo image obtained by photographing the thoracic cage including the interventricular septum, for example, a fetus chest.

As depicted in <FIG>, the server <NUM> may illustratively include a memory unit <NUM>, an object detection unit <NUM>, a feature outputting unit <NUM>, an auto-encoder <NUM>, a segmentation unit <NUM>, an acquisition unit <NUM>, a training unit <NUM>, an execution unit <NUM> and an outputting unit <NUM>.

The memory unit <NUM> is an example of a storage region and stores various information to be used for training the auto-encoder <NUM> and the segmentation unit <NUM>, executing and outputting the estimation process using an NNW group and so forth. As depicted in <FIG>, the memory unit <NUM> may be illustratively capable of storing multiple pieces of model information 11a, training data 11b, input data 11c and output data 11d.

The target detection unit <NUM> is an example of the target detection unit <NUM> depicted in <FIG>, and, for example, the target detection unit <NUM> detects a target from each of multiple inputted images and outputs multiple partial images including the detected target and a peripheral image of the target. A partial region is generated, for example, by cutting out a region including a target and a peripheral region of the target in a rectangular shape from the inputted image.

The target detection unit <NUM> may be, for example, an object detection model generated using the training data 11b and trained in advance for specifying a region of the target included in an input image and may be an object detection NNW such as a YOLO as an example. For example, a manager or a utilizer of the server <NUM> may execute training of the target detection unit <NUM> in advance using the training data 11b.

The feature outputting unit <NUM> is an example of the feature outputting unit <NUM> depicted in <FIG>. For example, the feature outputting unit <NUM> may be a trained model generated using a dataset of an image different from an image obtained by photographing a target and may be a trained model for estimating a label for an input image.

As the feature outputting unit <NUM>, for example, a VGG-Backbone is available. As an example, the VGG-Backbone may be an NNW in which a fully connected layer as an outputting layer is removed from a VGG and which consequently includes a convolution layer and a pooling layer. It is to be noted that a VGG is an example of a trained NNW usable in the embodiment. A trained NNW usable in the embodiment is not limited only to a VGG (or a VGG-Backbone).

It is to be noted that, since the feature outputting unit <NUM> is generated using a dataset of an image different from an image of the training data 11b, the feature outputting unit <NUM> may be a model obtained by diverting or processing a trained NNW such as a VGG publicly opened on the Internet or the like.

The auto-encoder <NUM> is an example of the auto-encoder <NUM> depicted in <FIG>. As the auto-encoder <NUM>, for example, an NNW in which the number of units of an intermediate layer is smaller in comparison with those of the inputting layer and the outputting layer, as an example, an auto encoder, is available.

The feature outputting unit <NUM> and the auto-encoder <NUM> collectively serve as an example of the support module <NUM>.

The segmentation unit <NUM> is an example of the segmentation unit <NUM> and an example of the segmentation module <NUM> depicted in <FIG>. Although various segmentation NNWs are available as the segmentation unit <NUM>, in the embodiment, for example, it is assumed that a U-Net is used. It is to be noted that the segmentation unit <NUM> is not limited to the U-Net and may be a different neural network for executing Semantic Segmentation or may be a neural network using a segmentation method other than the Semantic Segmentation.

Each of the auto-encoder <NUM> and the segmentation unit <NUM> is an NNW of a target a target to be trained in the training process in the server <NUM>.

In the following description, the target detection unit <NUM>, the feature outputting unit <NUM>, the auto-encoder <NUM>, and the segmentation unit <NUM> are sometimes referred to as "NNWs" or "NNW group".

Information of a network structure, various parameters and so forth for implementing the NNWs <NUM> to <NUM> may be stored as model information 11a for each of the NNWs <NUM> to <NUM> in the memory unit <NUM>.

The acquisition unit <NUM> acquires information to be used for training and execution of the auto-encoder <NUM> and the segmentation unit <NUM>, for example, from a computer not depicted.

For example, the acquisition unit <NUM> may acquire and store the training data 11b to be used for training of the auto-encoder <NUM> and the segmentation unit <NUM> into the memory unit <NUM>.

The training data 11b may include moving image data obtained by photographing a target and multiple annotation images indicative of a region of the target in each of multiple frame images included in the moving image data.

For example, the training data 11b may include m (m: two or more, for example, higher than n, integer) image sets <NUM> as depicted in <FIG>. Each of the image sets <NUM> may include an image <NUM> obtained by photographing a target (that may be referred to as "training target") and an annotation image <NUM> as correct answer data of segmentation of the image <NUM>.

The image <NUM> is an example of a frame image and, for example, may be an echo image obtained by photographing the interventricular septum that is an example of a target as depicted in <FIG>. Each of the echo images in the multiple image sets <NUM> may be a frame cut out in a time series (for example, T = <NUM> to (m - <NUM>)) from a series of echo images. The image <NUM> may be referred to as "total image".

The annotation image <NUM> is an example of an annotation image and is an image obtained by masking a target (in an example of <FIG>, "interventricular septum") in the image <NUM> of the image set <NUM>, for example, as depicted in <FIG>. It is to be noted that "mask" signifies, for example, that a region of a mask target is displayed in a mode distinguishable from a region that is not the mask target, and, as an example, "mask" may signify that a region of a mask target is filled with a predetermined color (a predetermined color is set to pixels in a region of a mask target). In the example of <FIG>, a masked region, in other words, a region corresponding to the thorax, is indicated by painting in white while the other region, in other words, any region other than the thorax, is indicated by paining in black.

It is to be noted that the server <NUM> may perform training of the auto-encoder <NUM> and the segmentation unit <NUM> using multiple training data 11b, in other words, using a dataset for multiple f moving image data.

Further, the acquisition unit <NUM> may acquire and store input data 11c to be used in the estimation process by the NNW groups <NUM> to <NUM> into the memory unit <NUM>.

The input data 11c is an example of target data including target moving image data obtained by photographing an estimation target.

For example, as depicted in <FIG>, the input data 11c may include multiple images <NUM> obtained by photographing a target (that may be referred to as "estimation target").

The image <NUM> is an example of a target frame image and may be an echo image obtained by photographing the interventricular septum that is an example of the estimation target, for example, as depicted in <FIG>. Each of the echo images included in the input data 11c may be a frame cut out in a time series (for example, T = <NUM> to (m - <NUM>)) from a series of echo images. The image <NUM> may be referred to as "total image".

The training unit <NUM> is an example of a training execution unit and performs training of the auto-encoder <NUM> and the segmentation unit <NUM> using the training data 11b acquired by the acquisition unit <NUM>.

The execution unit <NUM> is an example of an estimation processing unit that executes an estimation process of a region of the estimation target for the input data 11c. The execution unit <NUM> performs the estimation process of segmentation of a target for the input data 11c using the trained auto-encoder <NUM> and the segmentation unit <NUM> that are trained by the training unit <NUM> and the input data 11c that is acquired by the acquisition unit <NUM>.

The outputting unit <NUM> may output (accumulate) a segmented image <NUM> that is to be described below and that is inputted from the execution unit <NUM> to (into) the memory unit <NUM>, and generate output data 11d on the basis of multiple accumulated segmented images <NUM>.

The output data 11d includes an image set including one or more segmented images <NUM>, in other words, one or more output images, and, for example, may be moving image data of a video including multiple frame images. In the embodiment, as exemplified in <FIG>, the output data 11d may be a video obtained by coupling the segmented images <NUM> in a time series as frame images.

It is to be noted that the outputting unit <NUM> may transmit the output data 11d, for example, to a computer not depicted.

Now, an example of operation of the server <NUM> configured in such a manner as described above is described.

<FIG> is a flow chart illustrating an example of operation of the training phase, and <FIG> is a view illustrating an example of operation of the training phase.

The NNW groups <NUM> to <NUM> may be coupled to each other by the configuration depicted in <FIG> in the server <NUM>. In order to perform training of the auto-encoder <NUM> and the segmentation unit <NUM> in the training phase, the training unit <NUM> may include a combining unit 17a and an adding unit 17b hereinafter described as exemplified in <FIG>.

As exemplified in <FIG>, the training unit <NUM> acquires (t-n)th to (t+n)th total images <NUM> and a (t)th annotation image <NUM> from time series images <NUM> included in training data 11b acquired by the acquisition unit <NUM> (step S1). In <FIG> and succeeding drawings, for simplified illustration, the (t-n)th to (t+n)th total images <NUM> are referred to as "total images (t±n). " It is to be noted that the value of n may be, as an example, "<NUM>" or the like.

The training unit <NUM> may generate total images <NUM> and an annotation image <NUM> by resizing the acquired (t-n)th to (t+n)th total images <NUM> and (t)th annotation image <NUM> into an input size to the target detection unit <NUM>. Further, the training unit <NUM> may generate a total image <NUM> having a size resized to the input size to the feature outputting unit <NUM> from the size of the acquired (t)th total image <NUM>.

The training unit <NUM> inputs the resized (t-n)th to (t+n) total images <NUM> to the target detection unit <NUM> (step S2: refer to reference character A of <FIG>).

Further, the training unit <NUM> inputs the resized (t)th total image <NUM> to the feature outputting unit <NUM> (step S3: refer to reference character B of <FIG>). The feature outing unit <NUM> extracts a feature (feature amount) of the inputted (t)th total image <NUM> and inputs the extracted feature to an intermediate layer of the auto-encoder <NUM> (step S4: refer to reference character C of <FIG>).

<FIG> is a view depicting an example of a configuration of the feature outputting unit <NUM> and the auto-encoder <NUM>. In the example depicted in <FIG>, a VGG-Backbone (VGG backbone) <NUM> is an example of the feature outputting unit <NUM> and an Auto Encoder (auto encoder) <NUM> is an example of the auto-encoder <NUM>.

As depicted in <FIG>, the VGG backbone <NUM> at least includes a layer <NUM> that outputs the feature extracted by the VGG backbone <NUM>. The auto encoder <NUM> includes multiple layers <NUM> to <NUM> such as a convolution layer. It is to be noted that, in the example of <FIG>, one layer <NUM> of the VGG backbone <NUM> is depicted while illustration of layers in preceding stages to the layer <NUM> is omitted.

Referring to <FIG>, (x, y, z) indicated in the blocks of the layer <NUM> and the layers <NUM> to <NUM> indicate a size of information to be utilized (processed) by the layers <NUM> to <NUM> to <NUM>. For example, the symbols "x" and "y" indicate sizes in the vertical and horizontal directions of an image, and the symbol "z" indicates a channel number.

The feature outputted from the layer <NUM> of the VGG backbone <NUM> may be coupled (concatenated) in a channel direction to the output of the layer <NUM> of the auto encoder <NUM> and may be inputted to the layer <NUM> (refer to step S4 of <FIG> and reference character C of <FIG>).

The layer <NUM> performs a process using information of (<NUM>, <NUM>, <NUM>) in which the output (<NUM>, <NUM>, <NUM>) of the layer <NUM> and the output (<NUM>, <NUM>, <NUM>) of the layer <NUM> are coupled to each other in the channel (z) direction. Further, the layer <NUM> performs a process in which the feature that is the output (<NUM>, <NUM>, <NUM>) of the layer <NUM> is taken into account, and outputs information of (<NUM>, <NUM>, <NUM>) whose size is equal to that of the opposing layer <NUM>.

In this manner, the layer <NUM> is an example of an intermediate layer of the auto-encoder <NUM>. The intermediate layer may be, as an example, a layer whose size (x, y) is in the minimum, or in other words, may be a bottleneck of the auto encoder <NUM>.

It is to be noted that the intermediate layer of the auto encoder <NUM> that serves as an outputting designation of a feature from the layer <NUM> is not limited to the example depicted in <FIG> but may be any of various layers between the input layer <NUM> and the output layer <NUM> among the layers of the auto encoder <NUM>.

Referring back to <FIG>, the object detection unit <NUM> detects a target from each of the (t-n)th to (t+n)th total images <NUM> inputted in step S2 (reference character A in <FIG>), and outputs target peripheral images 12a to 12c including the detected target and a peripheral region of the detected target.

For example, the training unit <NUM> inputs the (t)th target peripheral image 12c outputted from the object detection unit <NUM> to the segmentation unit <NUM> (step S5: refer to reference character D in <FIG>). The segmentation unit <NUM> inputs a segmentation image 15a obtained by segmentation (for example, masking) of the target on the basis of the (t)th target peripheral image 12c to the adding unit 17b (step S6: refer to reference character E in <FIG>).

Further, for example, the training unit <NUM> combines, by the combining unit 17a thereof, n (t-n)th to (t-<NUM>)th target peripheral images 12a and n (t+<NUM>)th to (t+n)th target peripheral images 12b outputted from the object detection unit <NUM> (refer to <FIG>).

It is to be noted that, in <FIG> and the succeeding figures, the (t-n)th to (t-<NUM>)th target peripheral images 12a are represented as "target peripheral images (t-n)" and the (t+<NUM>)th to (t+n)th target peripheral images 12b are represented as "target peripheral images (t+n)" for simplified indication.

The combining unit 17a lines up, for example, n images in a channel direction to output a combined image 12e. As an example, the combining unit 17a may output two combined images 12e including a combined image 12e in which the n (t-n)th to (t-<NUM>)th target peripheral images 12a are used and another combined image 12e in which the n (t+<NUM>)th to (t+n)th target peripheral images 12b are used. It is to be noted that the combining unit 17a may otherwise output one combined image 12e, using the (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th target peripheral images 12a and 12b (2n images).

Then, the training unit <NUM> inputs the combined image 12e outputted from the combining unit 17a to the auto-encoder <NUM> (step S7: refer to reference character F in <FIG>).

The auto-encoder <NUM> receives the combined images 12e of the (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th images as an input to the input layer and receives a feature inputted from the feature outputting unit <NUM> as an input to the intermediate layer thereof, and outputs an output image 14a from the output layer. In the example of <FIG>, the input layer may be the layer <NUM>; the intermediate layer may be the layer <NUM>; and the output layer may be the layer <NUM>.

The training unit <NUM> inputs the output image 14a outputted from the auto-encoder <NUM> to the adding unit 17b (step S8: refer to reference character G in <FIG>).

<FIG> is a view illustrating an example of a configuration and operation of the adding unit 17b. As depicted in <FIG>, the adding unit 17b may illustratively include processing functions of an addition processing unit <NUM>, a difference calculation unit <NUM> and a training processing unit <NUM>.

The addition processing unit <NUM> adds a segmentation image 15a outputted from the segmentation unit <NUM> and an output image 14a outputted from the auto-encoder <NUM> for each cell to generate a combined output-image 12f (step S9: refer to reference character H in <FIG>).

The difference calculation unit <NUM> calculates a difference <NUM> between the combined output-image 12f outputted from the addition processing unit <NUM> and a target peripheral annotation image 12d outputted from the object detection unit <NUM> and outputs the difference <NUM> to the training processing unit <NUM>. As the calculation method for a difference by the difference calculation unit <NUM>, various known methods such as, for example, a least squares method can be applied.

Here, the target peripheral annotation image 12d inputted to the difference calculation unit <NUM> is described. As depicted in <FIG>, the training unit <NUM> inputs a resized (t)th annotation image <NUM> to the object detection unit <NUM> (step S10: refer to reference character I in <FIG>).

It is to be noted that the inputting of the annotation image <NUM> to the object detection unit <NUM> (step S10) may be performed, for example, in parallel to the inputting of the (t-n)th to (t+n)th total images <NUM> to the object detection unit <NUM> and the feature outputting unit <NUM> (steps S2 and S3).

The object detection unit <NUM> outputs a target peripheral annotation image 12d obtained by cutting out, from the inputted (t)th annotation image <NUM>, a partial region same as that of the (t)th target peripheral image 12c.

For example, the training unit <NUM> inputs the (t)th target peripheral annotation image 12d outputted from the object detection unit <NUM> to the difference calculation unit <NUM> of the adding unit 17b (step S11: refer to reference character J in <FIG> and reference character K in <FIG>).

The training processing unit <NUM> performs training of the auto-encoder <NUM> and the segmentation unit <NUM> on the basis of the difference <NUM> calculated by the difference calculation unit <NUM> (step S12: refer to reference character L in <FIG>), and the processing ends therewith.

As the training method of the auto-encoder <NUM> and the segmentation unit <NUM> by the training processing unit <NUM>, various machine learning methods may be used. As an example, in a machine learning process, in order to reduce the difference <NUM>, namely, to reduce the value of an error function, a back propagation process of determining (updating) a parameter to be used in processes in a forward propagation direction by the auto-encoder <NUM> and the segmentation unit <NUM> may be executed. Then, in the machine learning process, an update process of updating a variable such as a weight may be executed on the basis of a result of the back propagation process.

The training unit <NUM> may repeatedly execute the machine learning process of the auto-encoder <NUM> and the segmentation unit <NUM>, for example, using multiple image sets <NUM> included in training data 11b until a number of iterations, accuracy, or the like reaches a threshold value. The auto-encoder <NUM> and the segmentation unit <NUM> for which the training is completed are examples of a trained model.

For example, the training unit <NUM> may execute the processes in steps S1 to S12 depicted in <FIG> using each of the multiple total images <NUM> in the training data 11b as a first frame image by changing the value of (t) corresponding to the frame number in the moving image data. It is to be noted that the total images <NUM> up to the (n-<NUM>)th total image <NUM> from the top and the end in the moving image data as the training data 11b may be excluded from a selection target for a first frame image.

<FIG> is a flow chart illustrating an example of operation of the estimation phase, and <FIG> and <FIG> are views illustrating an example of operation of the estimation phase.

The execution unit <NUM> may include a combination unit 18a and an adding unit 18b to be described below as exemplified in <FIG> in order to perform estimation of segmentation of an estimation target in the estimation phase. It is to be noted that the combination unit 18a may have a processing function similar to that of the combining unit 17a. Further, <FIG> is basically similar to <FIG> in terms of flows of data although it is different from <FIG> in reference character of the image <NUM>, in that the annotation image <NUM> is not inputted to the object detection unit <NUM>, in that the object detection unit <NUM> outputs cutout position information <NUM> of the target peripheral image 12c in place of the target peripheral annotation image 12d, and in the configuration of the adding unit 18b.

As exemplified in <FIG>, the execution unit <NUM> acquires the (t-n)th to (t+n)th total images <NUM> from among the time series images <NUM> included in the input data 11c acquired by the acquisition unit <NUM> (step S21: refer to <FIG>).

It is to be noted that the (t)th total image <NUM> is an example of a third frame image, and the (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th total images <NUM> are an example of a predetermined number of preceding and succeeding fourth frame images of the third frame image in the time series of target moving image data.

As depicted in <FIG>, the execution unit <NUM> may generate total images <NUM>', for example, by resizing the sizes of the acquired (t-n)th to (t+n)th total images <NUM> to the input size of the object detection unit <NUM>. Further, the execution unit <NUM> may generate a total image <NUM>" by resizing the size of the acquired (t)th image <NUM> to the input size of the feature outputting unit <NUM>.

The execution unit <NUM> inputs the resized (t-n)th to (t+n)th total images <NUM>' to the object detection unit <NUM> (step S22).

Further, the execution unit <NUM> inputs the resized (t)th total image <NUM>" to the feature outputting unit <NUM> (step S23). The feature outputting unit <NUM> extracts a feature of the inputted (t)th total image <NUM>" and inputs the extracted feature to the intermediate layer of the auto-encoder <NUM> (step S24).

The object detection unit <NUM> detects the estimation target from each of the (t-n)th to (t+n)th total images <NUM>' inputted in step S22. Then, the object detection unit <NUM> outputs the target peripheral images 12a to 12c (refer to <FIG>) including the detected estimation target and a peripheral region of the estimation target, in other words, multiple partial images.

For example, the execution unit <NUM> inputs the (t)th target peripheral image 12c outputted from the object detection unit <NUM> to the segmentation unit <NUM> trained with parameter update by the training unit <NUM> (step S25). The segmentation unit <NUM> inputs a segmentation image 15a (refer to <FIG>) obtained by segmenting the estimation target on the basis of the inputted target peripheral image 12c to the adding unit 18b (step S26).

Further, for example, the execution unit <NUM> combines, by the combination unit 18a thereof, the n (t-n)th to (t-<NUM>)th target peripheral images 12a and the n (t+<NUM>)th to (t+n)th target peripheral images 12b outputted from the object detection unit <NUM>.

The combination unit 18a may output a combined image 12e, for example, by lining up n images in the channel direction similarly to the combining unit 17a. It is to be noted that the combination unit 18a may output one combined image 12e using the (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th target peripheral images 12a and 12b (2n images).

Then, the execution unit <NUM> inputs the combined image 12e outputted from the combination unit 18a to the auto-encoder <NUM> trained already with parameter update by the training unit <NUM> (step S27).

The auto-encoder <NUM> receives the (t-n)th to (t-<NUM>)th and (t+<NUM>)th to (t+n)th combined images 12e as an input to the input layer thereof and receives the feature inputted from the feature outputting unit <NUM> as an input to the intermediate layer thereof, and outputs an output image 14a (refer to <FIG>) from the output layer thereof.

The execution unit <NUM> inputs the output image 14a outputted from the auto-encoder <NUM> to the adding unit 18b (step S28).

<FIG> is a view illustrating an example of a configuration and operation of the adding unit 18b. As depicted in <FIG>, the adding unit 18b may illustratively include processing functions of an addition processing unit <NUM> and a size restoration unit <NUM>.

The addition processing unit <NUM> adds the segmentation image 15a outputted from the segmentation unit <NUM> and the output image 14a outputted from the auto-encoder <NUM> for each pixel to generate a combined output-image <NUM> (refer to <FIG>) (step S29).

The size restoration unit <NUM> receives as inputs thereto the combined output-image <NUM> outputted from the addition processing unit <NUM> and the cutout position information <NUM> of the target peripheral image 12c outputted from the object detection unit <NUM>.

Here, the cutout position information <NUM> inputted to the size restoration unit <NUM> is described.

As depicted in <FIG>, when the (t)th total image <NUM>' is inputted in step S22, the object detection unit <NUM> outputs the cutout position information <NUM> of the target peripheral image 12c in the total image <NUM>' together with the target peripheral image 12c. The execution unit <NUM> inputs, for example, the cutout position information <NUM> outputted from the object detection unit <NUM> to the size restoration unit <NUM> of the adding unit 18b (step S30: refer to <FIG> and <FIG>).

The cutout position information <NUM> is an example of position information indicative of the position in the (t)th total image <NUM>' from which the (t)th target peripheral image 12c is cut out. As the cutout position information <NUM>, for example, coordinate information indicative of a cutout position (region) of the target peripheral image 12c in the total image <NUM>' or like information is available.

The size restoration unit <NUM> returns, on the basis of the combined output-image <NUM> and the cutout position information <NUM>, the size of the combined output-image <NUM> to the original size of the total image <NUM> to generate a segmented image <NUM> (step S31). The segmented image <NUM> is an example of an image that includes a region estimated as an estimation target in the total image <NUM>.

For example, the size restoration unit <NUM> may fit the combined output-image <NUM> into the original (t)th image <NUM> on the basis of the cutout coordinates indicated by the cutout position information <NUM> to perform restoration. For this purpose, for example, the (t)th total image <NUM> may be inputted in addition to the cutout position information <NUM> of the (t)th target peripheral image 12c.

The execution unit <NUM> may change, for example, the value of (t) corresponding to the frame number in the target moving image data to set each of the multiple total images <NUM> in the input data 11c as a third frame image to execute the processes in steps S21 to S31 depicted in <FIG>. It is to be noted that the images <NUM> up to the (n-<NUM>)th image <NUM> from the top and the end in the moving image data as the input data 11c may be excluded from a selection target for a third frame image.

The outputting unit <NUM> accumulates the segmented images <NUM> and outputs output data 11d in which the accumulated segmented images <NUM> are combined to the output data 11d (step S32), and the processing ends therewith. It is to be noted that, as the outputting destination of the output data 11d, for example, a computer or the like not depicted is available in addition to the memory unit <NUM>.

As above, the execution unit <NUM> and the outputting unit <NUM> are an example of an image outputting unit that outputs an image including a region estimated as an estimation target in a third frame image on the basis of the combined output-image <NUM> and the cutout position information <NUM>.

As above, with the server <NUM> according to the embodiment, segmentation of a target is performed by inputting the following three kinds of images <NUM> and 12a to 12c to the NNWs <NUM> to <NUM> different from one another and integrating outputs (results) from the NNWs <NUM> to <NUM>.

For example, the server <NUM> inputs an image 12c, in which a peripheral region of a target in a frame image of the target is enlarged, to the segmentation unit <NUM>. Further, the server <NUM> inputs images 12a and 12b in which a target peripheral region is enlarged in frame images preceding to and succeeding the frame image of the target to the auto-encoder <NUM>. Furthermore, the server <NUM> inputs the image <NUM> of the entire frame of the target to the feature outputting unit <NUM>.

Consequently, the auto-encoder <NUM> can output, based on the frame images preceding to and succeeding the frame image of the target, an output image 14a, from which an influence of noise of the object included in the frame image of the target has been decreased.

Accordingly, the robustness of the frame image against noise in object detection of a frame image of moving image data can be improved.

Further, for example, even in the case where at least p art of a target in moving image data whose picture quality is comparatively rough is hidden by noise, segmentation of the target including the region hidden by the noise can be performed precisely.

Furthermore, by providing context information of surroundings of portions cut out as the target peripheral images 12a and 12b, namely, of the total image, as an intermediate feature from the feature outputting unit <NUM> to the auto-encoder <NUM>, the auto-encoder <NUM> can utilize information of portions other than the cutout portion.

For example, in the output image 14a based only on the target peripheral images 12a and 12b, the direction of the target in the output image 14a does not sometimes coincide with the correct direction of the target in the total image. Therefore, by providing a feature of the total image from the feature outputting unit <NUM> to the auto-encoder <NUM>, the auto-encoder <NUM> can output the output image 14a that takes the direction of the target into consideration.

Further, from the server <NUM>, the output image 14a from the auto-encoder <NUM> and the segmentation image 15a from the segmentation unit <NUM> are outputted. Consequently, for example, a user of the server <NUM> can compare, in the estimation phase, the output image 14a and the segmentation image 15a with each other to decide in what point the output image 14a has been amended with respect to the segmentation image 15a solely from the segmentation unit <NUM>.

<FIG> is a block diagram depicting an example of a hardware (HW) configuration of a computer <NUM> that implements the functions of the information processing apparatus <NUM> and the server <NUM>. In the case where multiple computers are used as HW resources that implement the functions of the information processing apparatus <NUM> and the server <NUM>, each computer may have the HW configuration exemplified in <FIG>.

As depicted in <FIG>, the computer <NUM> may illustratively include a processor 20a, a memory 20b, a storage unit 20c, an interface (IF) unit 20d, an input/output (I/O) unit 20e and a reading unit 20f as a hardware configuration.

The processor 20a is an example of an arithmetic processing unit that performs various controls and arithmetic operations. The processor 20a may be coupled for mutual communication to the blocks in the computer <NUM> by a bus 20i. It is to be noted that the processor 20a may be a multiprocessor including multiple processors or may be a multicore processor having multiple processor cores or otherwise may be configured so as to have multiple multicore processors.

As the processor 20a, integrated circuits (ICs) such as, for example, a CPU, an MPU, a GPU, an APU, a DSP, an ASIC and an FPGA are available. It is to be noted that, as the processor 20a, a combination of two or more of such integrated circuits as mentioned above may be used.

For example, processing functions of at least part of the information processing apparatus <NUM>, the acquisition unit <NUM> of the server <NUM>, at least part of the training unit <NUM>, at least part of the execution unit <NUM> and the outputting unit <NUM> may be implemented by a CPU, an MPU or the like as the processor 20a. Further, processing functions of at least part of the information processing apparatus <NUM>, the NNWs <NUM> to <NUM> of the server <NUM>, at least part of the training unit <NUM> and at least part of the execution unit <NUM> may be implemented by an accelerator such as a GPU or an ASIC (for example, a TPC) within the processor 20a.

CPU is an abbreviation of Central Processing Unit, and MPU is an abbreviation of Micro Processing Unit. GPU is an abbreviation of Graphics Processing Unit, and APU is an abbreviation of Accelerated Processing Unit. DSP is an abbreviation of Digital Signal Processor, and ASIC is an abbreviation of Application Specific IC and FPGA is an abbreviation of Field-Programmable Gate Array. TPU is an abbreviation of Tensor Processing Unit.

The memory 20b is an example of HW that stores information of various data, programs and so forth. As the memory 20b, one or both of a volatile memory such as a dynamic random access memory (DRAM) and a nonvolatile memory such as a persistent memory (PM) are available.

The storage unit 20c is an en example of HW that stores information of various data, programs and so forth. As the storage unit 20c, various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD) and a nonvolatile memory are available. As the nonvolatile memory, for example, a flash memory, a storage class memory (SCM), a read only memory (ROM) and so forth are available.

Further, the storage unit 20c may store a program <NUM> (training program) that implements all or part of various functions of the computer <NUM>. For example, the processor 20a of the information processing apparatus <NUM> can implement functions as the information processing apparatus <NUM> exemplified in <FIG> and <FIG> by expanding the program <NUM> stored in the storage unit 20c on the memory 20b and executing the expanded program <NUM>. Further, the processor 20a of the server <NUM> can implement the functions as the server <NUM> exemplified in <FIG>, <FIG>, <FIG>, <FIG> and <FIG> by expanding the program <NUM> stored in the storage unit 20c on the memory 20b and executing the expanded program <NUM>.

It is to be noted that the storage region at least one of the memory 20b and the storage unit 20c has may be capable of storing the information 11a to 11d depicted in <FIG>. In other words, the memory unit <NUM> of <FIG> may be implemented by a storage region at least one of the memory 20b and the storage unit 20c has.

The IF unit 20d is an example of a communication IF that performs control and so forth of coupling to and communication with a network. For example, the IF unit 20d may include an adapter that complies with a local area network (LAN) such as the Ethernet (registered trademark) or optical communication such as the Fibre Channel (FC) or the like. The adapter may be compatible with a communication method for one of or both wireless and wired communication. For example, the server <NUM> may be coupled for mutual communication to a different apparatus through the IF unit 20d. For example, the program <NUM> may be downloaded from the network to the computer <NUM> through the communication IF and stored into the storage unit 20c.

The I/O unit 20e may include one of or both an inputting apparatus and an outputting apparatus. As the inputting apparatus, for example, a keyboard, a mouse, a touch panel and so forth are available. As the outputting apparatus, for example, a monitor, a projector, a printer and so forth are available.

The reading unit 20f is an example of a reader for reading out information of data and programs recorded on a recording medium <NUM>. The reading unit 20f may include a connection terminal or device to or into which the recording medium <NUM> can be connected or inserted. As the reading unit 20f, for example, an adapter that complies with Universal Serial Bus (USB) or the like, a drive device that accesses a recording disk, a card reader that accesses a flash memory such as an SD card and so forth are available. It is to be noted that the recording medium <NUM> has the program <NUM> stored therein and the reading unit 20f may read out the program <NUM> from the recording medium <NUM> and store the program <NUM> into the storage unit 20c.

As the recording medium <NUM>, illustratively a non-transitory computer-readable recording medium such as a magnetic/optical disk, a flash memory and so forth are available. As the magnetic/optical disk, illustratively a flexible disk, a compact disc (CD), a digital versatile disc (DVD), a Blu-ray (registered trademark) disk, a holographic versatile disc (HVD) and so forth are available. As the flash memory, illustratively a semiconductor memory such as a USB memory or an SD card is available.

The HW configuration of the computer <NUM> described above is exemplary. Accordingly, increase or decrease of HW in the computer <NUM> (for example, addition or deletion of an arbitrary block), division, integration in arbitrary combination, addition or deletion of a bus and so forth may be performed suitably. For example, in the information processing apparatus <NUM> and the server <NUM>, at least one of the I/O unit 20e and the reading unit 20f may be omitted.

The technology relating to the embodiment described above can be carried out in such a modified or altered form as described below.

For example, the processing functions <NUM> to <NUM> provided in the server <NUM> depicted in <FIG> may individually be merged or divided in arbitrary combinations.

It is to be noted that, although it is described in the description of the embodiment that the target and the image are an interventricular septum and an echo image, respectively, they are not restrictive. The technique according to the embodiment can be applied also to various objects and images as described below.

As the target, for example, in addition to a part of the human body, various objects in regard to which one or both of the size and the amount of movement of the target is comparatively small with respect to the total region of an image are available. Further, the target does not have to be an object that can be viewed with the naked eye, for example, like an object at least part of which is buried in the ground. As the image, various images obtained by photographing a region including a target are available. For example, as the image, various images are available including an ultrasonic image other than an echo image, a magnetic resonance image, an X-ray image, a detection image by a sensor that captures a temperature, electromagnetic waves or the like, and a captured image by an image sensor that captures visible light or invisible light.

Further, the server <NUM> depicted in <FIG> may be configured such that the various processing functions are implemented by multiple apparatus that cooperate with each other through a network. As an example, the acquisition unit <NUM> and the outputting unit <NUM> may be a Web server; the NNWs <NUM> to <NUM>, training unit <NUM> and execution unit <NUM> may be an application server; and the memory unit <NUM> may be a database (DB) server. In this case, the Web server, application server and DB server may cooperate with each other through a network to implement the processing functions as the server <NUM>.

Claim 1:
A program for training an auto-encoder (<NUM>) and a neural network (<NUM>) for object detection, the program causing a computer to execute a process comprising:
acquiring training data including moving image data being obtained by photographing a target and including a plurality of frame images and a plurality of annotation images each masking the target in a respective one of the plurality of frame images included in the moving image data; and
executing a training process using the training data, wherein
the training process comprises:
detecting the target in each of (t-n)th to (t+n)th frame images, where n is an integer equal to or greater than one, in the moving image data in time series, and obtaining first partial images (2a, 2b, 2c; 12a, 12b, 12c) by cutting out a first partial region from the corresponding frame image that includes the target and a peripheral region of the target;
obtaining a second partial image (2d; 12d) by cutting out a second partial region from a tth annotation image corresponding to a tth frame image, the second partial region being the same as the first partial region;
inputting a first combined image (12e) to the auto-encoder (<NUM>), the first combined image (12e) being obtained by lining up in a channel direction, the first partial images (2a, 2b; 12a, 12b) extracted from the (t-n)th to (t-<NUM>)th frame image being preceding of the tth frame image and (t+<NUM>)th to the (t+n)th frame image being subsequent to the tth frame image, and obtaining, from the auto-encoder (<NUM>), a first output-image (14a) serving as a result of processing performed on the first combined image;
inputting the first partial image (2c; 12c) extracted from the tth frame image to the neural network (<NUM>) that performs a segmentation process for an input image, and obtaining, from the neural network (<NUM>), a second output-image (15a) as a result of the segmentation process performed on the target included in the first partial image; and
performing parameter update of the auto-encoder (<NUM>) and the neural network (<NUM>), based on a difference between a first combined output-image-and the second partial image (2d; 12d), the first combined output-image being obtained by combining the first output-image from the auto-encoder (<NUM>) and the second output-image from the neural network (<NUM>).