Patent ID: 12205275

DESCRIPTION OF THE EMBODIMENTS

The above-described related art is not capable of selecting appropriate imaging conditions or inference models unless the lesion to be detected is specified. Accordingly, a plurality of different inference models are used in accordance with imaging conditions for each image, and there is a problem in that inference is time-consuming. There also is a problem with the above-described technology in that results obtained using different inference models with regard to a plurality of images are combined, and accordingly an increase in imaging conditions makes the amount of time required for inference longer as well.

Accordingly, the technology according to the present disclosure provides technology for performing inferencing regarding lesions in images, obtained by imaging a subject under a plurality of imaging conditions, more speedily.

Embodiments of the present disclosure will be described below with reference to the Figures. Components, members, and processing, illustrated in the Figures, which are the same or equivalent, are denoted by the same symbols and a repetitive description will be omitted as appropriate. Also, part of the components, members, and processing are omitted from illustration in the Figures.

First Embodiment

An information processing system according to a first embodiment will be described below. In the present embodiment, an inference model is selected that is appropriate for an image group obtained by imaging a subject under a plurality of imaging conditions, and inferencing is performed related to the lesion in the images. In the description below, a case when a brain tumor is detected as a lesion, from an image group including T1-weighted images, T2-weighted images, and diffusion-weighted images obtained by imaging the head of a subject using an MRI device, will be assumed, as one example.

FIG.1is a block diagram illustrating an example of a configuration of an information processing system1according to the present embodiment. The information processing system1includes an imaging device110, a data server120, and an information processing device130. The imaging device110is connected to the data server120, and data imaged by the imaging device110is saved in the data server120. The data server120is connected to the information processing device130. The information processing device130has an acquiring unit131, a selecting unit132, an inference unit133, a generating unit134, and an output unit135. The functions of the parts of the information processing device130are realized by a central processing unit (CPU) executing a program stored in a memory in the information processing device130. The information processing device130acquires data of an image group imaged by the data server120, and executes the processing described below using the acquired data.

In the present embodiment, the imaging device110is an MRI device that images the head of a subject under the imaging conditions of T1 weighting, T2 weighting, and diffusion weighting. The data server120saves an image group of T1-weighted images, T2-weighted images, and diffusion-weighted images received from the imaging device110. The data server120also holds a plurality of trained inference models that have learned image groups including T1-weighted images, T2-weighted images, and diffusion-weighted images in advance by a deep neural network, or the like. Note that it is sufficient for at least one type of images of T1-weighted images, T2-weighted images, and diffusion-weighted images to be included in the image group, and images imaged under other imaging conditions may be included. Also, images to be used for learning are not limited to images imaged by the imaging device110, and may be selected as appropriate. A system such as a picture archiving and communication system (PACS), or the like, that saves and manages images over a network, for example, may be used for the data server120.

The acquiring unit131acquires an image group including a plurality of T1-weighted images, a plurality of T2-weighted images, and a plurality of diffusion-weighted images from the data server120, in response to user instructions performed at the information processing device130. Note that the types of images that the acquiring unit131acquires are not limited to these three types, and images of at least one type may be acquired. The acquiring unit131corresponds to acquiring means that acquires an image group obtained by imaging a subject under a plurality of imaging conditions. In the following description, the plurality of T1-weighted images may be referred to simply as “T1-weighted images”. The same is true for the plurality of T2-weighted images and the plurality of diffusion-weighted images. “T1-weighted image” is the name of images imaged by executing a pulse sequence using first parameters, and “T2-weighted image” is the name of images imaged by executing a pulse sequence using second parameters. Typically, a longer repetition time (TR) is set for parameters for T1-weighted images as compared to parameters for T2-weighted images. In this way, imaging conditions are decided by parameters in the sequence. That is to say, imaging conditions can be made to differ depending on the difference in parameters among sequences.

The selecting unit132compares the image group acquired by the acquiring unit131(T1-weighted images, T2-weighted images, and diffusion-weighted images) with the input conditions of the inference models (types of images (imaging conditions) that are the object of input to the models) that the data server120holds. Input conditions of the inference models are each correlated with at least one imaging condition of the plurality of imaging conditions. For example, input conditions include conditions that at least one type of image of the T1-weighted images, T2-weighted images, and diffusion-weighted images is the object of input, such as conditions that only T1-weighted images are the object of input, conditions that T1 and T2-weighted images are the object of input, and so forth. This comparison identifies inference models regarding that satisfy the input conditions of at least one type of image out of the T1-weighted images, T2-weighted images, and diffusion-weighted images. The selecting unit132then selects, out of the inference models that satisfy the input conditions of the acquired image group, the inference model that is the highest in order of priority, which has been defined in advance. The standard for the order of priority of the inference model selected here is, for example, the height of inference precision in detecting lesions from images, and the higher the inference precision is, the higher the order of priority is. The selecting unit132may select not only the inference model that is the highest in order of priority, but also select inference models that are a predetermined number of places in height from the top of order of priority. Thus, the selecting unit132selects a plurality of models corresponding to a plurality of imaging conditions. Also, the standard for the order of priority may be employed as appropriate, such as the number of images for each imaging condition included in other image groups, regardless of the height of inference precision. The selecting unit132corresponds to selecting means that selects at least one inference model from a plurality of inference models, on the basis of at least one imaging condition out of the plurality of imaging conditions. Also, the standard for the order in priority corresponds to a predetermined standard for selecting the inference model with the highest order in priority.

The selecting unit132then selects, out of the image group acquired by the acquiring unit131, images that match the input conditions of the selected inference model, as a new image group. The selecting unit132outputs the selected inference model and the image group to the inference unit133.

For example, an assumption will be made that the data server120holds an inference model A, an inference model B, and an inference model C, which perform some sort of inference with regard to input images. The input conditions of the inference models A, B, and C here respectively are “T1-weighted images and T2-weighted images are object of input”, “T1-weighted images, T2-weighted images, and fluid-attenuated inversion recovery (FLAIR) images are object of input”, and “T1-weighted images are object of input”. Also, the order of priority of the inference models A, B, and C is inference model B, inference model A, and inference model C, in descending order. Further, an image group including T1-weighted images, T2-weighted images, and diffusion-weighted images is acquired by the acquiring unit131.

In this case, the selecting unit132selects the inference model A and the inference model C as inference models of which the acquired image group satisfies the input conditions. No FLAIR images are included in the image group, and accordingly the selecting unit132excludes the inference model B from the object of selection. The selecting unit132then selects the inference model A that has the highest order of priority. Further, the selecting unit132selects, from the image group acquired by the acquiring unit131, T1-weighted images and T2-weighted images that are objects of input matching the input conditions of the inference model A, as a new image group.

Now, the diffusion-weighted images included in the image group acquired by the acquiring unit131are ineligible for input of the inference model A, and accordingly the selecting unit132excludes diffusion-weighted images from the image group.

The inference unit133applies the selected inference model to the new image group made up of images matching the input conditions of the inference model selected by the selecting unit132, and performs detection of a brain tumor that is the lesion in the images. In a case when a plurality of inference models are selected by the selecting unit132, the inference unit133performs inferencing regarding the lesion for each of the images in the image group, using the plurality of inference models. The inference unit133corresponds to inference means that perform inferencing as to images of the image group.

The generating unit134generates superimposed images, in which indices indicating a region of the brain tumor detected by the inference unit133are superimposed on images in the image group selected by the selecting unit132. The generating unit134corresponds to generating means that generates images indicating the position of the lesion detected by inference in the images of the acquired image group.

The output unit135outputs superimposed images, in which indices of the brain tumor generated by the generating unit134are superimposed, to the data server120, a display unit of an information processing device omitted from illustration, an external display device, or the like.

Next, the processing that the information processing device130executes in the present embodiment will be described with reference to the flowchart inFIG.2. The CPU of the information processing device130starts the processing of the flowchart inFIG.2upon receiving an instruction from a user of the information processing device130, for example.

In step S21, the CPU of the information processing device130accepts a specification of imaging conditions for images from the user. The acquiring unit131then performs communication with the data server120and acquires an image group imaged under imaging conditions matching the imaging conditions that the user has specified, from the data server120. The acquiring unit131outputs the acquired image group to the selecting unit132.

In step S22, the selecting unit132first selects an inference model that satisfies the input conditions of the image group input from the acquiring unit131. Next, the selecting unit132selects the inference model of which the order of priority is the highest, in accordance with the order of priorities of the inference models stored in a memory of the information processing device130. The selecting unit132selects, from the image group input from the acquiring unit131, images matching the input conditions defined in the selected inference model as a new image group, and outputs the selected image group to the inference unit133.

Now, at the time of selecting the inference model satisfying the input conditions of the image group input from the acquiring unit131, the selecting unit132may identify the inference model to be selected by referencing header information, or the like, included in images of the input image group. Alternatively, the selecting unit132may use machine learning, or the like, to distinguish under what imaging conditions the input image group was imaged, and to identify the inference model to be selected on the basis of the results of distinguishing. In these cases, in step S21, the acquiring unit131acquires the image group from the data server120without performing processing of accepting a specification regarding imaging conditions from the user.

In step S23, the inference unit133detects the brain tumor in the images by inputting the image groups selected in step S22into the inference model selected in S22. In the present embodiment, the brain tumor in the images is detected by the inference unit133using a known segmentation technique on the images, such as a U-Net, or the like, as an example.

In step S24, the generating unit134generates superimposed images, in which indices indicating a region of the brain tumor detected by the inference unit133are superimposed on the images in the image group selected in step S22. The output unit135then outputs the superimposed images generated by the generating unit134to the data server120.

Thus, according to the information processing device of the present embodiment, an appropriate image group for input to an inference model can be selected from an image group imaged by an MRI device or the like, and inferencing of a legion within the images of the selected image group can be performed. Accordingly, inferencing regarding lesions can be speedily performed on an image group obtained by imaging the same subject under a plurality of imaging conditions.

Second Embodiment

Next, an information processing system according to a second embodiment will be described. Note that, in the following description, configurations that are the same as those in the first embodiment are denoted by the same symbols, and a detailed description will be omitted.

Phenomena that deteriorate the quality of inference, such as artifacts, for example, may occur in images imaged by an imaging device. As such, in the present embodiment, the information processing device selects, from an image group obtained by imaging the same subject under a plurality of imaging conditions, an image group of a quality appropriate for inference of a lesion. The information processing device then selects an inference model appropriate for the selected image group, and performs inferencing regarding the lesion in the images. Although artifacts are assumed in the following description as phenomena that deteriorate the quality of inference, the following processing may be applied to other phenomena that occur in images and that affect quality of inference, such as image distortion, contrast, and so forth.

An example of choosing an image group with appropriate quality on the basis of estimation values of artifacts occurring at the time of imaging images will be described in the present embodiment. Although a case of a motion artifact occurring at the time of imaging is assumed in the following description, the types of artifacts occurring in images are not limited to this. The present embodiment can also be applied in cases when artifacts occurring in images are ring artifacts, streak artifacts, shower artifacts, shading artifacts, beam hardening, and so forth, for example.

FIG.3is a block diagram illustrating an example of a configuration of an information processing system3according to the present embodiment. A description will be made here primarily regarding configurations that are different from the configurations of the first embodiment. An information processing device330includes the acquiring unit131, the inference unit133, the generating unit134, the output unit135, a selecting unit332, and a calculating unit336.

The selecting unit332compares the image group acquired by the acquiring unit131with input conditions of the inference models that the data server120holds, and selects an inference model for the image group acquired in accordance with an order of priority. The selecting unit332then selects, from the image group acquired by the acquiring unit131, images matching the input conditions of the selected inference model as a new image group. The selecting unit332outputs the selected image group to the calculating unit336.

The calculating unit336divides the images of the image group selected by the selecting unit332into a plurality of regions, and calculates, for each divided region, the probability of a motion artifact occurring, using a known technique that is based on image processing, for each imaging condition. The calculating unit336corresponds to calculating means that calculates the probability of occurrence of a phenomenon that would deteriorate inference in each region of the images.

The probability of a motion artifact occurring may be calculated by applying an inference model trained in advance, using machine learning, or the like, to the images. The calculating unit336outputs information indicating the probability of a motion artifact occurring for each divided region of the image group to the selecting unit332.

The selecting unit332compares the probability of occurrence that is input from the calculating unit336with a predetermined threshold value of probability of occurrence of a motion artifact defined in advance. The selecting unit332then identifies images having a region in which the probability of occurrence of an artifact exceeds the threshold value. Further, the selecting unit332excludes, from the image group output to the calculating unit336, images imaged under the same imaging conditions as the images identified here, and reselects an inference model with regard to the image group made up of the remaining images, by executing the above processing. The selecting unit332then outputs the image group made up of the images after exclusion and the reselected inference model to the inference unit133.

Next, the processing executed by the information processing device330according to the present embodiment will be described with reference to the flowchart inFIG.4. The CPU of the information processing device330starts the processing of the flowchart inFIG.4upon receiving an instruction from a user of the information processing device330, for example.

Step S21is the same as the processing in the first embodiment. The acquiring unit131outputs the image group acquired from the data server120to the selecting unit332.

Next, in step S31, the selecting unit332selects an inference model that satisfies the input conditions of the image group input from the acquiring unit131. The selecting unit332then selects the inference model of which the order of priority is the highest, in accordance with the order of priorities of the inference models stored in memory of the information processing device330. The selecting unit332further selects, from the image group input from the acquiring unit131, images matching the input conditions defined in the selected inference model as a new image group. In step S31, the selecting unit332outputs the selected image group to the calculating unit336.

In step S32, the calculating unit336divides each image of the image group selected in step S31by the selecting unit332into a plurality of regions, using a known technique. The calculating unit336then calculates the probability of occurrence of a motion artifact, using a known technique, for each divided region in each image.

In step S33, the selecting unit332determines whether or not the probability of occurrence of a motion artifact calculated in step S32exceeds the threshold value defined in advance. In a case when there is a region in which the probability of occurrence of a motion artifact exceeds the threshold value (Y in S33), the flow advances to step S34. Conversely, in a case when there is no region in which the probability of occurrence of a motion artifact exceeds the threshold value (N in S33), the flow advances to step S23.

In step S34, the selecting unit332excludes, from the image group selected in step S31, images imaged under the same imaging conditions as images having a region in which the probability of occurrence of a motion artifact exceeds the threshold value. The flow then advances to step S35.

In step S35, the selecting unit332executes processing that is the same as in step S31, and subjects the image group made up of the remaining images in step S34to processing, reselects the image group and inference model, and outputs the selected image group and inference model to the inference unit133.

Next, the processing in steps S23and S24is the same as the processing in the first embodiment. In a case when the flow has advanced from step S33to step S34(case of Y in S33), in step S23, the inference unit133inputs the image group reselected in step S35to the inference model reselected in step S35, and detects the brain tumor in each of the images of the image group. Also, in a case when the flow has advanced from step S33to step S23without passing through steps S34and S35(case of N in S33), in step S23, the inference unit133uses the inference model and the image group selected in step S31to detect the brain tumor in each of the images of the image group. Then, in step S24, the generating unit134generates superimposed images, in which indices indicating a region of the brain tumor detected by the inference unit133are superimposed on the images in the image group reselected in step S35. The output unit135then outputs the superimposed images generated by the generating unit134to the data server120.

Thus, according to the information processing device of the present embodiment, even if images of poor quality in which artifacts have occurred are included in the imaged image group, inferencing can be performed with such images excluded. Accordingly, the precision of inference regarding a lesion in images obtained by imaging the same subject under a plurality of imaging conditions can be improved.

Other Embodiments

The above-described embodiments are but specific examples of the present disclosure. The scope of the present disclosure is not limited to the configurations of the above-described embodiments, and various embodiments can be made within a scope not departing from the essence thereof.

Modifications of the above embodiments will be described below. The following modifications may be combined with each other and carried out, or may be combined as appropriate with the above embodiments and carried out. Note that, in the following description, configurations that are the same as those in the above embodiments are denoted by the same symbols, and a detailed description will be omitted.

First Modification

A case of detecting a brain tumor from T1-weighted images and T2-weighted images obtained by imaging the head of a subject using an MRI device has been described as an example in the above embodiments. Note that, as one modification, the imaging conditions when imaging the subject with an MRI device is not limited to T1 weighting, T2 weighting, or diffusion weighting. The imaging conditions may be proton-density weighted imaging, FLAIR, susceptibility-weighted imaging, magnetic resonance angiography (MRA), T2*-weighted imaging, or the like, and may be any combination of these imaging conditions.

Also, the imaging device110that images the subject is not limited an MRI device, and may be an X-ray CT device, ultrasonography device, or the like. Examples of imaging conditions for an X-ray CT device include lung window, mediastinal window, and so forth. Also, examples of imaging conditions for an ultrasonography device may be B-mode, doppler, elastography, and so forth. Further, a plurality of imaging devices may be used in tandem in the above information processing system. In this case, an image group obtained by imaging the same site of the same subject by the plurality of imaging devices is preferably used. Also, a singular imaging device that images a subject through a combination of a plurality of imaging devices, such as positron emission tomography (PET)-CT, may be used. Accordingly, in the above information processing system, the acquiring unit131can acquire an image group obtained by imaging by different modalities, acquire an image group obtained by imaging under a plurality of imaging conductions by the same modality, and so forth. In a case of imaging the subject by different modalities, image groups of various image types are obtained by imaging by these modalities. The term “image type” as used here indicates the type of modality that images the subject and generates an imaged image.

FIG.5illustrates a schematic configuration of an information processing system5that is an example of the present modification. Note that configurations that are the same as those in the above embodiments are denoted by the same symbols, and a detailed description will be omitted. In the information processing system5, the data server120stores imaged images generated by a first imaging device510, a second imaging device520, and a third imaging device530, which are each different modalities. For example, an arrangement may be made where the first imaging device510is an MRI device, the second imaging device520is a CT device, and the third imaging device530is an ultrasonography device. The information processing device330then executes the above processing as to an image group made up of a plurality of image types acquired by the data server120, thereby performing inferencing regarding lesions as to images in the image group.

In the present modification, the acquiring unit131acquires an image group made up of a plurality of image types obtained by imaging a subject by different modalities. Also, the selecting unit332uses image types as the input conditions of the inference models instead of the imaging conditions in the above embodiments, and selects at least one inference model from a plurality of inference models on the basis of at least one image type out of a plurality of image types. Note that, the selecting unit332may select an inference model corresponding to image types and imaging conditions. The inference unit133uses the selected inference model to perform inferencing regarding lesions as to the images of the image group that has been acquired.

The types of modalities used for imaging the subject differ depending on the usage environment of the information processing system. According to the present modification, even in a case when only images of a particular plurality of image types, such as MR images and CT images alone, for example, can be obtained, optimal inference results regarding lesions corresponding to image types can be obtained by the information processing device330.

Also, the information processing device330may perform inferencing regarding lesions as to an images of image group including imaged images obtained using a plurality of imaging conditions, and imaged images obtained by a plurality of image types. For example, a case is assumed where an image group includes T1-weighted images and T2-weighted images generated by an MRI device, and CT images generated by a CT device that is a separate modality. Thus, the information processing device330can obtain inference regarding lesions in the same way as above, with regard to an image group made up of a combination of a plurality of images obtained by difference sequences of a singular modality, and images obtained by another modality, as well.

Also, the imaging site of the subject is not limited to the head, and may be the chest or abdomen, or the like. The lesion that is the object of inference is not limited to brain tumors, and may be a stroke, a lung nodule, or the like, depending on the imaging site. Also, inferencing relating to lesions is not limited to detection of lesions, and may include malignancy differentiation, prognosis prediction, determination of whether or not a lesion is present in an image, and so forth. An inference model is created in accordance with the lesion that is the object of inference or the object of detection. Also, in the above description, image types and image conditions can be deemed to be the same input conditions, as input conditions for the inference model. That is to say, the types of modalities used for imaging may be included in the imaging conditions.

Second Modification

A description has been made in the above embodiments regarding a case when images acquired under the same imaging conditions as an image having a region regarding which the probability of occurrence of an artifact exceeds a threshold value are excluded from the image group acquired from the data server120. However, as one modification, an arrangement may be made in which the selecting unit332excludes only images having a region regarding which the probability of occurrence of an artifact exceeds the threshold value, i.e., only images including artifacts, from the acquired image group.

In this case, in step S34, the selecting unit332excludes only images having a region regarding which the probability of occurrence of an artifact exceeds the threshold value from the image group selected in step S31. In step S35, the selecting unit332reselects the inference model and the image group as to the image group made up of sets of remaining images, in the same way as described above. The inference unit133then inputs the image group reselected in step S35to the inference model reselected in step S35, and detects the brain tumor in the images of the image group.

Accordingly, images in which no artifacts have occurred can be input to the inference model without being excluded, and, accordingly, it is anticipated that inference results with even higher precision can be obtained.

Note that, in the present modification, when images having regions in which the probability of occurrence of an artifact exceeds the threshold value are excluded, and the number of images remaining that have been imaged under the same imaging conditions as the excluded image is low, there is a possibility that using the images imaged under these imaging conditions may also lower the inference precision. Accordingly, an arrangement may be made in which, in a case when the percentage of images having occurrence of artifacts in the images imaged under the same imaging conditions exceeds a predetermined percentage, the selecting unit332may exclude all images imaged under the same imaging conditions in the same way as in the above embodiments. An example of the percentage of images having occurrence of artifacts here is the proportion of the number of images having occurrence of artifacts as to the images imaged under the same imaging conditions. The number of images having an occurrence of artifacts in the images imaged under the same imaging conditions exceeding a predetermined number may be deemed as being the percentage of images having occurrence of artifacts as to the images imaged under the same imaging conditions exceeding the predetermined percentage.

Third Modification

Processing of excluding images having a region in which the probability of occurrence of an artifact exceeds the threshold value from the object of input to the inference model, regardless of position in the image, has been described in the above embodiments. However, as one modification, the selecting unit332may determine in step S34whether or not the region in which the probability of occurrence of an artifact exceeds the threshold value is included in a region of interest, defined in advance. Alternatively, the selecting unit332may determine whether or not there is a region in which the probability of occurrence of an artifact exceeds the threshold value for only this region of interest. The region of interest may be defined as appropriate, such as being specified by a user, being automatically extracted from an image acquired from the data server120, and so forth. In a case when a region in which the probability of occurrence of an artifact exceeds the threshold value is included in the region of interest, the selecting unit332excludes the image having the region regarding which the probability of occurrence of an artifact exceeds the threshold value from the object of input to the inference model. Accordingly, even if there is an artifact that has occurred in a certain image, this image can be input to the inference model without being excluded as long as the artifact has not occurred in the region of interest, and, accordingly, it is anticipated that inference results with even higher precision can be obtained.

Fourth Modification

A case of excluding images having a region in which the probability of occurrence of an artifact exceeds the threshold value from the object of input to the inference model has been described in the above embodiments. However, as one modification, an arrangement may be made where such images are not excluded from the object of input to the inference model, and a selection of the inference model and inferencing are performed using an image group including these images. The generating unit134may then generate information indicating that images with low quality are used in the inference, along with the inference results from the inference unit133. Alternatively, the generating unit134may perform correction regarding artifacts as to images having a region regarding which the probability of occurrence of an artifact exceeds the threshold value by machine learning, or the like, as correcting means, with the inference unit133performing inferencing as to the image group including the corrected images.

Embodiment(s) of the present invention can also be realized by a computer of a system or an apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., an application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., a central processing unit (CPU), or a micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and to execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray Disc (BD)™) a flash memory device, a memory card, and the like.

According to the technology of the present disclosure, inferencing regarding lesions in images, obtained by imaging a subject under a plurality of imaging conditions, can be performed more speedily.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.