MEDICAL IMAGE PROCESSING APPARATUS, X-RAY DIAGNOSTIC APPARATUS, AND METHOD OF MEDICAL IMAGE PROCESSING

A medical image processing apparatus of an embodiment includes processing circuitry acquiring an X-ray image about a subject, acquiring an ultrasonic image data about the subject, extracting an object contained in the X-ray image, and performing processing based on the position of the extracted object on the ultrasonic image data in accordance with the relative positional relation between a coordinate system in the X-ray image and a coordinate system in the ultrasonic image data to generate a composite image as a combination of a processed ultrasonic image data after being subjected to the processing and the X-ray image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2021-041654, filed on Mar. 15, 2021; and Japanese Patent Application No. 2022-031855, filed on Mar. 2, 2022, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a medical image processing apparatus, an X-ray diagnostic apparatus, and a method of medical image processing.

BACKGROUND

Various methods of treatment performed by inserting a medical device into the body of a subject are known. When such treatment is performed, by acquiring and displaying an X-ray image of the subject, operation of the medical device by a surgeon can be supported. That is to say, by referring to the X-ray image, the surgeon can smoothly proceed with a procedure while grasping the positional relation between a region to be treated and the medical device within the body of the subject.

However, in the X-ray image, some structures such as soft tissues, for example, are difficult to appear on the image. Given this, by displaying another type of medical image such as an ultrasonic image and the X-ray image in a combined manner, more information can be provided to the surgeon.

DETAILED DESCRIPTION

The following describes embodiments of a medical information processing apparatus, a medical information processing system, and a method of medical information processing in detail with reference to the accompanying drawings.

The following describes embodiments of a medical image processing apparatus, an X-ray diagnostic apparatus, and a method of medical image processing in detail with reference to the accompanying drawings.

A first embodiment describes a medical image processing system1including a medical image processing apparatus30.FIG. 1is a block diagram of an example of the configuration of the medical image processing system1according to the first embodiment. As illustrated inFIG. 1, the medical image processing system1according to the first embodiment includes an X-ray diagnostic apparatus10, an ultrasonic diagnostic apparatus20, and a medical image processing apparatus30.

As illustrated inFIG. 1, the X-ray diagnostic apparatus10, the ultrasonic diagnostic apparatus20, and the medical image processing apparatus30are connected to each other via a network NW. So long as they can be connected to each other via the network NW, the X-ray diagnostic apparatus10, the ultrasonic diagnostic apparatus20, and the medical image processing apparatus30can be installed at any locations. The medical image processing apparatus30may be installed in a different hospital or another facility from that of the X-ray diagnostic apparatus10and the ultrasonic diagnostic apparatus20, for example. That is to say, the network NW may include a local network closed within a hospital or be a network via the Internet.

The X-ray diagnostic apparatus10is an apparatus acquiring an X-ray image about a subject P. The X-ray diagnostic apparatus10acquires and displays the X-ray image while a procedure on the subject P is being performed, for example. To give an example, in cardiovascular treatment of structural cardiac diseases such as mitral valve repair, septal defect closure, and aortic valve repair, a surgeon inserts a medical device such as a catheter into the body of the subject P and operates it. The X-ray diagnostic apparatus10can acquire the X-ray image about the medical device inserted into the body of the subject P, a region to be treated within the body of the subject P, or the like and display it on a display.

The following describes an example of the X-ray diagnostic apparatus10with reference toFIG. 2.FIG. 2is a block diagram of an example of the configuration of the X-ray diagnostic apparatus10according to the first embodiment. As illustrated inFIG. 2, the X-ray diagnostic apparatus10includes an X-ray high voltage apparatus101, an X-ray tube102, a couchtop103, a detector104, an input interface105, a display106, a memory107, and processing circuitry108.

The X-ray high voltage apparatus101supplies high voltage to the X-ray tube102under the control of the processing circuitry108. The X-ray high voltage apparatus101has an electric circuit such as a transformer and a rectifier and has a high voltage generation apparatus generating high voltage to be applied to the X-ray tube102and an X-ray control apparatus controlling output voltage corresponding to X-rays to be applied by the X-ray tube102, for example. The high voltage generation apparatus may be of the transformer system or of the inverter system.

The X-ray tube102is a vacuum tube having a cathode (filament) generating thermoelectrons and an anode (target) generating X-rays upon collision with the thermoelectrons. The X-ray tube102applies the thermoelectrons from the cathode toward the anode using the high voltage supplied from the X-ray high voltage apparatus101to generate X-rays. Although omitted inFIG. 2, the X-ray diagnostic apparatus10may include an X-ray aperture near the X-ray application port of the X-ray tube102. The X-ray aperture includes a collimator narrowing the application range of the X-rays generated by the X-ray tube102and a filter adjusting the X-rays emitted from the X-ray tube102, for example.

The couchtop103is a bed on which the subject P is placed and is placed on a bed apparatus not illustrated. The subject P is not included in the X-ray diagnostic apparatus10. The bed apparatus has a drive mechanism such as a motor and an actuator and controls the couchtop103by operating the drive mechanism under control of the processing circuitry108described below, for example. The bed apparatus adds drive voltage to the drive mechanism in accordance with a control signal received from the processing circuitry108to translate or tilt the couchtop103, for example.

The detector104is an X-ray flat panel detector (FPD) having detector elements arranged in a matrix, for example. The detector104detects the X-rays emitted from the X-ray tube102and having passed through the subject P and outputs a detection signal corresponding to a detected X-ray dose to the processing circuitry108. The detector104may be an indirect conversion type detector having a grid, a scintillator array, and an optical sensor array or a direct conversion type detector having a semiconductor element converting incident X-rays into an electric signal.

The detector104may be placed at a certain position under the couchtop103or be configured to be movable. The X-ray tube102and the detector104may be held by separate supports or be held integrally by a support such as a C-arm. AlthoughFIG. 2illustrates an overtube type configuration, in which the X-ray tube102is positioned above the subject P, the X-ray diagnostic apparatus10may be configured as an undertube type, in which the X-ray tube102is positioned below the subject P.

The input interface105receives various kinds of input operations from a user such as the surgeon, converts the received input operations into electric signals, and outputs them to the processing circuitry108. The input interface105can be implemented by a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad performing input operations through touching on an operating surface, a touchscreen with a display screen and a touchpad integrated, a non-contact input circuit including an optical sensor, or a voice input circuit, for example. The input interface105may include a tablet terminal or the like that can wirelessly communicate with the X-ray diagnostic apparatus10main body. The input interface105may be a circuit receiving input operations from the user through motion capture. To give an example, by processing signals acquired via a tracker and images acquired about the user, the input interface105can receive user's body movements, gaze, and the like as input operations. The input interface105is not limited to those including physical operating components such as a mouse or a keyboard. Examples of the input interface105include an electric signal processing circuitry receiving electric signals corresponding to input operations from an external input device provided separately from the X-ray diagnostic apparatus10and outputting these electric signals to the processing circuitry108.

The display106displays various kinds of information. The display106displays a graphical user interface (GUI) for receiving user instructions and medical images such as X-ray images under the control of the processing circuitry108, for example. The display106is a liquid crystal display or a cathode ray tube (CRT) display, for example. The display106may be of a desktop type or include a tablet terminal or the like that can wirelessly communicate with the processing circuitry108.

The memory107is implemented by a semiconductor memory element such as a random access memory (RAM) or a flash memory, a hard disk, or an optical disc, for example. The memory107stores therein various kinds of medical images such as X-ray images and computer programs corresponding to various functions to be read and executed by the processing circuitry108, for example.

The processing circuitry108executes an acquisition function108aand an output function108bto control the operation of the entire X-ray diagnostic apparatus10.

The processing circuitry108reads a computer program corresponding to the acquisition function108afrom the memory107and executes it to acquire the X-ray image about the subject P, for example. The acquisition function108ais an example of a acquisition unit. The acquisition function108acontrols the X-ray high voltage apparatus101and adjusts the voltage to be supplied to the X-ray tube102to control an X-ray dose to be applied to the subject P and on and off, for example. The acquisition function108acontrols the operation of an imaging system including the X-ray tube102and the couchtop103to control an imaging range and an imaging angle. The acquisition function108agenerates an X-ray image based on the detection signal received from the detector104. The acquisition function108amay perform various kinds of image processing on the generated X-ray image. The acquisition function108aexecutes noise reduction processing with an image processing filter and scattered ray correction on the generated X-ray image, for example.

The processing circuitry108reads a computer program corresponding to the output function108bfrom the memory107and executes it to output the X-ray image acquired by the acquisition function108a, for example. The output function108bdisplays the X-ray image on the display106, for example. The output function108btransmits the X-ray image to an external apparatus via the network NW, for example. To give an example, the output function108btransmits the X-ray image to the medical image processing apparatus30. To give another example, the output function108btransmits the X-ray image to an image storage apparatus not illustrated. The image storage apparatus is a server of a picture archiving and communication system (PACS), for example.

In the X-ray diagnostic apparatus10illustrated inFIG. 2, each processing function is stored in the memory107in the form of a computer program that can be executed by a computer. The processing circuitry108is a processor reading the computer program from the memory107and executing it to implement the function corresponding to each computer program. In other words, the processing circuitry108having read the computer program has the function corresponding to the read computer program.

Although the above inFIG. 2describes a case in which the single processing circuitry108implements the acquisition function108aand the output function108b, a plurality of independent processors may be combined with each other to form the processing circuitry108, and each of the processors may execute the computer program to implement the function. Each processing function of the processing circuitry108may be implemented by being distributed or integrated into a single circuit or a plurality of processing circuits as appropriate.

Referring back toFIG. 1, the description is continued. The ultrasonic diagnostic apparatus20illustrated inFIG. 1is an apparatus acquiring ultrasonic image data about the subject P. The ultrasonic diagnostic apparatus20transmits and receives an ultrasonic wave using an ultrasonic probe while the procedure on the subject P is being performed to acquire the ultrasonic image data, for example. The ultrasonic image data acquired by the ultrasonic diagnostic apparatus20is combined with the X-ray image acquired by the X-ray diagnostic apparatus10. A composite image of the X-ray image and the ultrasonic image data will be described below.

The ultrasonic probe included in the ultrasonic diagnostic apparatus20is adjusted in the position and orientation with respect to the subject P so that the medical device inserted into the body of the subject P, the region to be treated within the body of the subject P, and the like are contained in the imaging range, for example. The type of the ultrasonic probe is not limited to a particular one. The ultrasonic probe may be an intracorporeal probe such as a transesophageal echocardiography (TEE) probe or a body surface probe to be attached to the body surface of the subject P, for example.

The ultrasonic probe included in the ultrasonic diagnostic apparatus20has a plurality of transducer elements (piezoelectric transducer elements, for example), for example. The ultrasonic diagnostic apparatus20vibrates these transducer elements to generate an ultrasonic wave. The transducer elements receive a reflected wave from the subject P and convert it into an electric signal. That is to say, when the ultrasonic wave is transmitted to the subject P, the transmitted ultrasonic wave is reflected one after another by an acoustic impedance discontinuous surface in the body tissue of the subject P and is received by the transducer elements of the ultrasonic probe as a reflected wave signal (an echo signal). The amplitude of the received reflected wave signal depends on the difference in acoustic impedance at the discontinuous surface in which the ultrasonic wave is reflected. The reflected wave signal when the transmitted ultrasonic pulse is reflected by a moving bloodstream or a surface such as the heart wall undergoes a frequency shift depending on the velocity component of a moving object with respect to an ultrasonic transmission direction due to the Doppler effect.

The ultrasonic probe included in the ultrasonic diagnostic apparatus20may be a one-dimensional ultrasonic probe in which a plurality of piezoelectric transducer elements are arranged in a row, a one-dimensional ultrasonic probe in which a plurality of piezoelectric transducer elements arranged in a row are mechanically oscillated, or a two-dimensional ultrasonic probe in which a plurality of piezoelectric transducer elements are arranged in two dimensions in a grid shape.

Further, the ultrasonic diagnostic apparatus20generates ultrasonic image data based on the reflected wave signal received by the ultrasonic probe. The ultrasonic diagnostic apparatus20has a preamplifier, an analog/digital (A/D) converter, a reception delay unit, an adder, and the like and performs various kinds of processing on the reflected wave signal received by the ultrasonic probe to generate reflected wave data, for example. The ultrasonic diagnostic apparatus20controls the transmission direction of an ultrasonic beam from the ultrasonic probe to scan a three-dimensional region of the subject P and generates three-dimensional reflected wave data from the reflected wave signal received by the ultrasonic probe, for example.

The ultrasonic diagnostic apparatus20generates ultrasonic image data based on the reflected wave data and transmits the ultrasonic image data to the medical image processing apparatus30. The type of the ultrasonic image data is not limited to a particular one and may be a B mode image or a Doppler image, for example. By performing logarithmic amplification, envelope detection processing, or the like on the reflected wave data, the ultrasonic diagnostic apparatus20can generate the B mode image, in which signal intensity for each sampling point is expressed in terms of the brightness of luminance, for example. By extracting motion information based on the Doppler effect of the moving object at each sampling point within a scanning region based on the reflected wave data, the ultrasonic diagnostic apparatus20can generate the Doppler image.

As illustrated inFIG. 1, for example, the medical image processing apparatus30has an input interface31, a display32, a memory33, and processing circuitry34.

The input interface31, the display32, and the memory33can be configured in the same manner as the input interface105, the display106, and the memory107, respectively, described above. The input interface31receives various kinds of input operations from the user, converts the received input operations into electric signals, and outputs them to the processing circuitry34, for example. The display32, under the control of the processing circuitry34, displays a GUI for receiving user instructions and various kinds of medical images such as X-ray images, ultrasonic image data, or composite images of these. The memory33stores therein various kinds of medical images such as X-ray images, ultrasonic image data, or composite images of these and stores therein computer programs corresponding to various kinds of functions read and executed by the processing circuitry34.

The processing circuitry34executes an X-ray image acquisition function34a, an ultrasonic image acquisition function34b, an object extraction function34c, an image generation function34d, and an output function34eto control the operation of the entire medical image processing apparatus30. The X-ray image acquisition function34ais an example of an X-ray image acquisition unit. The ultrasonic image acquisition function34bis an example of an ultrasonic image acquisition unit. The object extraction function34cis an example of an object extraction unit. The image generation function34dis an example of an image generation unit. The output function34eis an example of an output unit.

The processing circuitry34reads a computer program corresponding to the X-ray image acquisition function34afrom the memory33and executes it to acquire the X-ray image about the subject P, for example. The X-ray diagnostic apparatus10applies X-rays from the X-ray tube102to the subject P and detects the X-rays having passed through the subject P with the detector104to acquire the X-ray image, for example. The X-ray image acquisition function34aacquires the X-ray image acquired by the X-ray diagnostic apparatus10via the network NW. The X-ray image acquisition function34amay acquire the X-ray image directly from the X-ray diagnostic apparatus10or acquire it via another apparatus such as an image storage apparatus.

The processing circuitry34reads a computer program corresponding to the ultrasonic image acquisition function34bfrom the memory33and executes it to acquire the ultrasonic image data about the subject P, for example. The ultrasonic diagnostic apparatus20controls the transmission and reception of the ultrasonic wave using the ultrasonic probe to acquire the ultrasonic image data, for example. The ultrasonic image acquisition function34bacquires the ultrasonic image data acquired by the ultrasonic diagnostic apparatus20via the network NW. The ultrasonic image acquisition function34bmay acquire the ultrasonic image data directly from the ultrasonic diagnostic apparatus20or acquire it via another apparatus such as an image storage apparatus.

The processing circuitry34reads a computer program corresponding to the object extraction function34cfrom the memory33and executes it to extract an object contained in an X-ray image, for example. The processing circuitry34reads a computer program corresponding to the image generation function34dfrom the memory33and executes it to perform processing based on the position of the extracted object on the ultrasonic image data in accordance with the relative positional relation between a coordinate system in the X-ray image and a coordinate system in the ultrasonic image data and generates a composite image as a combination of processed ultrasonic image data after being subjected to the processing and the X-ray image, for example. The processing circuitry34reads a computer program corresponding to the output function34efrom the memory33and executes it to output the composite image generated by the image generation function34d, for example. Processing by the object extraction function34cthe image generation function34d, and the output function34ewill be described below.

In the medical image processing apparatus30illustrated inFIG. 1, each processing function is stored in the memory33in the form of a computer program that can be executed by a computer. The processing circuitry34is a processor reading the computer program from the memory33and executing it to implement the function corresponding to each computer program. In other words, the processing circuitry34having read each computer program has the function corresponding to the read computer program.

Although the above inFIG. 1describes a case in which the single processing circuitry34implements the X-ray image acquisition function34a, the ultrasonic image acquisition function34b, the object extraction function34c, the image generation function34d, and the output function34e, a plurality of independent processors may be combined with each other to form the processing circuitry34, and each of the processors may execute the computer program to implement the function. Each processing function of the processing circuitry34may be implemented by being distributed or integrated into a single circuit or a plurality of processing circuitries as appropriate.

The above has described a configuration example of the medical image processing system1. Under such a configuration, the medical image processing unit30in the medical image processing system1improves the visibility of the object contained in the X-ray image in the composite image of the X-ray image and the ultrasonic image data by processing by the processing circuitry34.

The following first describes a series of processing to generate and display the composite image of the X-ray image and the ultrasonic image data. After a procedure such as cardiovascular treatment is started, the X-ray diagnostic apparatus10acquires the X-ray image from the subject P, whereas the ultrasonic diagnostic apparatus20acquires the ultrasonic image data from the subject P, for example. The following describes a case in which the X-ray diagnostic apparatus10acquires an X-ray image I1as an example. The X-ray image I1is a two-dimensional image having two axes orthogonal to an X-ray application direction. The following describes a case in which the ultrasonic diagnostic apparatus20acquires ultrasonic image data I2as an example. As illustrated inFIG. 3Afor example, the ultrasonic image data12is a three-dimensional image data (volume data).FIG. 3Ais a diagram of an example of the ultrasonic image data I2according to the first embodiment.

The X-ray image acquisition function34aacquires the X-ray image I1acquired by the X-ray diagnostic apparatus10via the network NW. The ultrasonic image acquisition function34bacquires the ultrasonic image data I2acquired by the ultrasonic diagnostic apparatus20via the network NW. Next, the image generation function34dgenerates a composite image of the X-ray image I1and the ultrasonic image data I2.

Specifically, the image generation function34dfirst identifies the relative positional relation between a coordinate system in the X-ray image I1and a coordinate system in the ultrasonic image data I2. In other words, the image generation function34dconducts registration process between the X-ray image I1and the ultrasonic image data I2.

The image generation function34dextracts an ultrasonic probe Q used to acquire the ultrasonic image data I2from the X-ray image I1to identify the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2, for example. That is to say, depending on the placement of the ultrasonic probe Q, the ultrasonic probe Q may be contained in the imaging range of the X-ray image I1as illustrated inFIG. 3B. The image generation function34danalyzes the X-ray image I1and identifies the position and orientation of the ultrasonic probe Q to identify the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2.FIG. 3Bis a diagram for illustrating generation processing for the composite image according to the first embodiment.

To give an example, the image generation function34dextracts the ultrasonic probe Q from the X-ray image I1using a three-dimensional model showing the ultrasonic probe Q. The three-dimensional model showing the ultrasonic probe Q can be generated from a three-dimensional image data imaging the ultrasonic probe Q, for example. To give an example, the three-dimensional model showing the ultrasonic probe Q can be generated from an X-ray computed tomography (CT) image (volume data) imaging the ultrasonic probe Q by an X-ray CT apparatus. Alternatively, the three-dimensional model showing the ultrasonic probe Q may be computer-aided design (CAD) data.

The image generation function34dmatches the three-dimensional model showing the ultrasonic probe Q to the X-ray image I1to identify the position and orientation of the ultrasonic probe Q in the coordinate system of the X-ray image I1. By virtually projecting the three-dimensional model showing the ultrasonic probe Q onto a plane, the image generation function34dcan associate any position and orientation of the ultrasonic probe Q and the coordinate system of the X-ray image I1with each other for each projection direction, for example. The projection direction (the X-ray application direction) at the time of acquiring the X-ray image I1is known from the support angle of the X-ray diagnostic apparatus10or the like. By matching the three-dimensional model showing the ultrasonic probe Q to the X-ray image I1, the image generation function34dcan identify the position and orientation of the ultrasonic probe Q at the time of acquiring the X-ray image I1.

At the time of taking the ultrasonic image data I2, the ultrasonic diagnostic apparatus20transmits an ultrasonic beam from the transducer elements of the ultrasonic probe Q. In addition, the ultrasonic diagnostic apparatus20controls the transmission direction of the ultrasonic beam in accordance with an imaging condition to scan the three-dimensional region of the subject P. The ultrasonic image data I2is generated for this three-dimensional scanning region, and thus the position and orientation of the ultrasonic image data I2with respect to the ultrasonic probe Q are clear from the imaging condition. Consequently, by identifying the position and orientation of the ultrasonic probe Q in the coordinate system of the X-ray image I1, the image generation function34dcan identify the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2.

Although the above describes a case of identifying the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2by extracting the ultrasonic probe Q from the X-ray image I1, the embodiment is not limited to this example. That is to say, the method for identifying the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2is not limited to a particular method, and any method can be adopted.

When X-ray-opaque markers are attached to the ultrasonic probe Q, for example, the image generation function34dmay extract these markers from the X-ray image I1. Such markers are clearly depicted on the X-ray image I1, and thus they can be located easily and accurately. When three markers are attached, for example, the image generation function34dcan identify the position and orientation of the ultrasonic probe Q in the coordinate system of the X-ray image I1based on the positions of the markers with respect to the ultrasonic probe Q and the distance between the markers on the X-ray image I1. The X-ray-opaque markers may be attached to a fixture (such as a belt) for attaching the ultrasonic probe Q to the subject P.

To give another example, when a sensor is attached to the ultrasonic probe Q, by detecting the position and orientation of the ultrasonic probe Q in the coordinate system of the X-ray image I1with the sensor, the image generation function34dcan identify the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2. The sensor may be attached to the fixture for attaching the ultrasonic probe Q to the subject P.

After identifying the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2, the image generation function34dcan generate the composite image of the X-ray image I1and the ultrasonic image data I2in accordance with the identified positional relation. The image generation function34ddisplays the ultrasonic image data I2at a corresponding position on the X-ray image I1in a superimposed manner as illustrated inFIG. 3B, for example. When the ultrasonic image data I2is three-dimensional image data (volume data), for example, the image generation function34dperforms rendering processing in the X-ray application direction when the X-ray image I1has been acquired to generate a two-dimensional ultrasonic image and generates a composite image of the two-dimensional ultrasonic image and the X-ray image I1.

The output function34edisplays the generated composite image on the display32. Alternatively, the output function34emay transmit the generated composite image to another apparatus, and the composite image may be displayed on the other apparatus. The output function34etransmits the generated composite image to the X-ray diagnostic apparatus10, for example. In this case, the output function108bcan display the composite image on the display106.

By referring to the composite image of the X-ray image I1and the ultrasonic image data I2, the user can efficiently grasp the position and shape of the medical device inserted into the body of the subject P, blood vessels contrasted by a contrast medium, soft tissues, and the like. However, as illustrated inFIG. 3B, in the composite image, the ultrasonic image data I2overlaps with part of the X-ray image I1. Consequently, when the object such as the medical device operated by the user appears in the X-ray image I1, the object may be hidden by the ultrasonic image data I2. Given this, when generating the composite image of the X-ray image I1and the ultrasonic image data I2, the medical image processing apparatus30further performs the following processing to improve the visibility of the object contained in the X-ray image I1.

Specifically, after the X-ray image acquisition function34aacquires the X-ray image I1, the object extraction function34cextracts the object contained in the X-ray image I1. The object is an object that the user focuses on, for example. Specific examples of the object include the medical device operated by the user, a blood vessel in the travel direction of the medical device, and the region to be treated. The object may be preset or be selected by the user as appropriate.

The method for extracting the object from the X-ray image I1is not limited to a particular method. When the medical device such as a guidewire, a catheter, or a stent is used as the object, for example, by performing matching processing based on the shape of the medical device, the object extraction function34ccan extract the object contained in the X-ray image I1. The object extraction function34ccan also extract the object by methods such as thresholding and machine learning, for example. Alternatively, the object extraction function34cmay extract the object by receiving an operation to designate the object from the user having referred to the X-ray image I1.

Next, the image generation function34dperforms processing based on the position of the extracted object on the ultrasonic image data I2in accordance with the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2. The image generation function34dchanges the transmittance of a region corresponding to the position of the extracted object out of the ultrasonic image data I2, for example.

The image generation function34dfirst sets a region R for the ultrasonic image data I2in accordance with the position of the object extracted in the X-ray image I1, for example. Specifically, the X-ray image I1is a two-dimensional image having two axes orthogonal to the X-ray application direction, and the position of the object can be identified as two-dimensional coordinates. Thus, the image generation function34dsets an axis passing through the coordinates corresponding to the position of the object and parallel to the X-ray application direction and sets the region R so as to contain the set axis.

The object extraction function34cextracts an object D contained in the X-ray image as illustrated inFIG. 4, for example.FIG. 4illustrates a case in which the object D is a stent as an example. Next, the image generation function34dsets the region R so as to be circular when viewed from the X-ray application direction (an image observation direction) for the ultrasonic image data I2in accordance with the position of the object D. Specifically, the image generation function34dsets an axis passing through the coordinates corresponding to the position of object D and parallel to the X-ray application direction and defines a cylinder in which the set axis passes through the center of the bottom face thereof and the axis and the height direction thereof are parallel to each other. The image generation function34dthen sets a region in which the defined cylinder and the ultrasonic image data I2overlap with each other as the region R.FIG. 4is a diagram for illustrating the processing on the ultrasonic image data I2according to the first embodiment.

After setting the region R, the image generation function34dexecutes the processing on the ultrasonic image data I2. The image generation function34dhides the region R, for example. In other words, the image generation function34dchanges the transmittance of the region R to “100%”. The ultrasonic image data I2after being subjected to the processing based on the position of the object D is also referred to as processed ultrasonic image data I2′.

The image generation function34dthen generates a composite image I3as a combination of the processed ultrasonic image data I2′ and the X-ray image I1as illustrated inFIG. 5. The image generation function34dperforms rendering processing in the X-ray application direction on the processed ultrasonic image data I2′ to generate a two-dimensional ultrasonic image and then combines the two-dimensional ultrasonic image and the X-ray image I1with each other to generate the composite image I3, for example. The type of the rendering processing is not limited to particular processing; an example is volume rendering (VR) processing, which generates a two-dimensional image reflecting three-dimensional information from volume data.FIG. 5is a diagram of an example of the composite image according to the first embodiment.

By referring to the composite image I3, the user can efficiently grasp the position and shape of the medical device inserted into the body of the subject P, blood vessels contrasted by a contrast medium, soft tissues, and the like. In the case illustrated inFIG. 5in particular, although the position of the object D, which is a stent, and the imaging range of the ultrasonic image data I2overlap with each other, the user can also visually recognize the object D because part of the ultrasonic image data I2is hidden. That is to say, the medical image processing apparatus30can improve the visibility of the object D contained in the X-ray image I1in the composite image I3of the X-ray image I1and the ultrasonic image data I2.

Although the region R of the ultrasonic image data I2is hidden inFIG. 5, the displaying/hiding of the region R may be switchable. The output function34edisplays the composite image I3on the display32and switches the displaying/hiding of the region R in accordance with an input operation from the user, for example. To give an example, the user operates a pointing device such as a mouse. The output function34ecan then switch the displaying/hiding of the region R with the composite image I3being clicked with a mouse cursor or a certain button on the UI being pressed as a trigger.

Although the above inFIG. 5describes a case of displaying the composite image I3as a combination of the processed ultrasonic image data I2′ and the X-ray image I1, the X-ray image I1may further be displayed together with the composite image I3. The output function34edisplays the composite image I3and the X-ray image I1side by side as illustrated inFIG. 6, for example. With this configuration, the visibility of the object D can further be improved. That is to say, although in the composite image I3, the region R is hidden, so that the object D can be visually recognized, the surrounding area of the object D is hidden by the processed ultrasonic image data I2′. With the display example inFIG. 6, the object D can be observed including the surrounding area.

In addition, the output function34ecan display the composite image I3together with various images. The output function34emay display the composite image I3and the ultrasonic image data I2without the region R side by side, for example. The output function34emay display the composite image I3, the X-ray image I1, and the ultrasonic image data I2without the region R side by side, for example.

Although the above inFIG. 4andFIG. 5describes a case in which the region R is circular when viewed from the X-ray application direction, the shape of the region R can be changed as desired. The image generation function34dmay define a columnar region having a base shaped as desired and set a region in which the defined cylinder and the ultrasonic image data I2overlap with each other as the region R, for example. To give an example, the image generation function34dmay define a columnar region having a base corresponding to the shape of the object D and set a region in which the defined columnar region and the ultrasonic image data I2overlap with each other as the region R. That is to say, the image generation function34dmay perform processing based on the position and shape of the object D on the ultrasonic image data I2.

In addition, the shape of the region R can be changed in various ways. The image generation function34dmay define a columnar region having a height direction not parallel to the X-ray application direction and set a region in which the defined columnar region and the ultrasonic image data I2overlap with each other as the region R, for example. Alternatively, the image generation function34dmay make the region R a non-columnar shape. The image generation function34dmay set a spherical or spindle-shaped region as the region R, for example.

Although the above inFIG. 4andFIG. 5describes a case of hiding the region R, the image generation function34dmay make the region R semi-transparent. The image generation function34dmay change the transmittance of the region R in the ultrasonic image data I2to any value from “0°” to “100%”, for example. The user may be allowed to change the transmittance in this case as desired. The image generation function34dchanges the region R in the ultrasonic image data I2to a certain transmittance to generate the processed ultrasonic image data I2′ and generates the composite image13as a combination of the processed ultrasonic image data12′ and the X-ray image I1, for example. The output function34edisplays the composite image I3on the display32and changes the transmittance of the region R in accordance with an input operation from the user. To give an example, the user operates a pointing device such as a mouse. The output function34ethen changes the transmittance of the region R in accordance with the rotation of a mouse wheel or the operation of a certain bar on the UI.

The above inFIG. 4describes a case in which the ultrasonic image data I2is three-dimensional image data, and the region R is also set in three dimensions. However, embodiments are not limited to this example. The ultrasonic image data I2may be a two-dimensional image, and the region R may be set in two dimensions, for example.

In this case, the ultrasonic image acquisition function34bacquires the ultrasonic image data I2as a two-dimensional image. The ultrasonic image acquisition function34bacquires a three-dimensional ultrasonic image from the ultrasonic diagnostic apparatus20via the network NW and performs rendering processing in the X-ray application direction on the acquired three-dimensional ultrasonic image to acquire the two-dimensional ultrasonic image data I2, for example. Alternatively, the rendering processing can be performed in the ultrasonic diagnostic apparatus20, and the ultrasonic image acquisition function34bcan acquire the two-dimensional ultrasonic image data I2from the ultrasonic diagnostic apparatus20via the network NW. After the object D is extracted from the X-ray image I1by the object extraction function34c, the image generation function34dsets a two-dimensional region corresponding to the position of the object D out of the two-dimensional ultrasonic image data I2as the region R. The image generation function34dchanges the transmittance of the region R set in two dimensions to generate the processed ultrasonic image data12′. That is to say, the image generation function34dmay change the transmittance of each voxel with the region R as a three-dimensional region or change the transmittance of each pixel with the region R as a two-dimensional region.

The following describes an example of the procedure of processing by the medical image processing apparatus30with reference toFIG. 7.FIG. 7is a flowchart for illustrating the sequence of the processing by the medical image processing apparatus30according to the first embodiment. Step S101and Step S107correspond to the X-ray image acquisition function34aand the ultrasonic image acquisition function34b. Step S103corresponds to the object extraction function34c. Step S102, Step S104, and Step S105correspond to the image generation function34d. Step S106corresponds to the output function34e.

First, the processing circuitry34acquires the X-ray image I1and the ultrasonic image data I2(Step S101). Next, the processing circuitry34conducts registration process between the X-ray image I1and the ultrasonic image data I2(Step S102). That is to say, the processing circuitry34identifies the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2.

Next, the processing circuitry34extracts the object D from the X-ray image I1(Step S103). Next, the processing circuitry34performs the processing based on the position of the extracted object D on the ultrasonic image data I2in accordance with the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2to generate the processed ultrasonic image data I2′ (Step S104). Next, the processing circuitry34combines the X-ray image I1and the processed ultrasonic image data I2′ with each other to generate the composite image I3(Step S105) and displays the generated composite image I3on the display32(Step S106).

Next, the processing circuitry34determines the presence or absence of a new image (Step S107), and if there is a new image, it acquires the new image (affirmative at Step S107), and the process again moves to Step S102. That is to say, while the procedure on the subject P is being performed, the X-ray diagnostic apparatus10can repeatedly take the X-ray image I1at a certain frame rate. Similarly, the ultrasonic diagnostic apparatus20can repeatedly take the ultrasonic image data I2at a certain frame rate. In such a case, the processing circuitry34can successively acquire a new X-ray image I1and a new ultrasonic image data12, and if there is a new image, it can acquire the new image, again perform Step S102to Step S106based on the new image, and update the composite image I3to be displayed in real time. On the other hand, if it is determined that there is no new image at Step S107(negative at Step S107), the processing circuitry34ends the processing.

While the procedure on the subject P is being performed, the X-ray diagnostic apparatus10and the ultrasonic diagnostic apparatus20acquire the X-ray image I1and the ultrasonic image data I2, respectively, in real time, for example. In this case, the processing circuitry34, each time the X-ray image I1is newly acquired from the subject P by the X-ray diagnostic apparatus10, successively acquires the X-ray image I1and, each time the ultrasonic image data12is newly acquired from the subject P by the ultrasonic diagnostic apparatus20, successively acquires the ultrasonic image data I2. The processing circuitry34successively extracts the object D from the newly acquired X-ray image I1. The processing circuitry34successively performs the processing based on the position of the extracted object D on the newly acquired ultrasonic image data I2and successively generates the composite image I3as a combination of the processed ultrasonic image data I2′ after being subjected to the processing and the newly acquired X-ray image I1. The processing circuitry34then successively displays the generated composite image I3on the display32. In this case, the composite image I3displayed on the display32is a real-time image successively updated.

At Step S107, it may be determined that there is a new image when either the X-ray image I1or the ultrasonic image data I2is newly acquired. Assumed is a case in which the X-ray image I1is acquired in real time, whereas the ultrasonic image data I2is not acquired, for example. As the ultrasonic image data I2, not the real-time image but an image acquired before the start of the procedure, for example, can be used, for example. In such a case, by repeatedly executing the processing from Step S102to Step S107, the processing circuitry34can successively update the part based on the X-ray image I1out of the composite image I3and display it in real time, although it cannot update the part based on the ultrasonic image data I2.

Alternatively, assumed is a case in which the ultrasonic image data I2is acquired in real time, whereas the X-ray image I1is not acquired. To give an example, a technique called last image hold (LIH) is known, in which an X-ray image lastly acquired is displayed in place of the real-time image, since exposure occurs while acquisition of X-ray images is continued. As the X-ray image I1, not the real-time image but LIH can be used. In such a case, by repeatedly executing the processing from Step S102to Step S107, the processing circuitry34can successively update the part based on the ultrasonic image data I2out of the composite image I3and display it in real time, although it cannot update the part based on the X-ray image I1.

AlthoughFIG. 7illustrates a case in which the process moves to Step S102when it is determined that there is a new image at Step S107, the process may move to Step S103with Step S102skipped. That is to say, even when at least either the X-ray image I1or the ultrasonic image data I2is newly acquired at Step S107, if there is no particular change in the imaging angle, the body movement of the subject P, or the like, the processing at Step S104may be executed again in accordance with the positional relation identified in the past.

As described above, according to the first embodiment, the X-ray image acquisition function34aacquires the X-ray image I1about the subject P. The ultrasonic image acquisition function34bacquires the ultrasonic image data12about the subject P. The object extraction function34cextracts the object D contained in the X-ray image I1. The image generation function34dperforms the processing based on the position of the extracted object D on the ultrasonic image data I2in accordance with the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2and generates the composite image I3as a combination of the processed ultrasonic image data I2′ after being subjected to the processing and the X-ray image I1. Thus, the medical image processing apparatus30according to the first embodiment can improve the visibility of the object D contained in the X-ray image I1in the composite image I3of the X-ray image I1and the ultrasonic image data I2.

The first embodiment described above describes a case in which setting of the region R is performed based on one X-ray image I1. In contrast, a second embodiment describes a case in which setting of the region R is performed based on a plurality of X-ray images I1. The medical image processing system1according to the second embodiment has the same configuration as that of the medical image processing system1illustrated inFIG. 1, with part of the processing by the image generation function34dbeing different. Points having the same configuration as the configuration described in the first embodiment are denoted by the same signs as those inFIG. 1andFIG. 2, and descriptions thereof are omitted.

The medical device such as a stent used in the treatment of cardiac diseases may fluctuate in position within the body of the subject P due to the influence of a heartbeat and the like, for example. When the X-ray images I1of such a medical device are acquired over time, the position of the medical device on the X-ray images I1changes for each cardiac phase.

The following describes a case in which setting of the region R is performed using the X-ray images I1with reference toFIG. 8A,FIG. 8B,FIG. 9A, andFIG. 9B. The X-ray images I1is X-ray images of a plurality of time phase acquired in time series.FIG. 8A,FIG. 8B,FIG. 9A, andFIG. 9Bare diagrams of examples of region setting processing according to the second embodiment.

The X-ray diagnostic apparatus10acquires an X-ray image I11at a time T1, acquires an X-ray image I12at a time T2, and acquires an X-ray image I13at a time T3, for example. The X-ray images I11to I13are examples of the X-ray image I1. In this case, as illustrated inFIG. 8A, the position of the medical device at the time T1(hereinafter referred to as an object D1), the position of the medical device at the time T2(hereinafter referred to as an object D2), and the position of the medical device at the time T3(hereinafter referred to as an object D3) may be different from each other.

The object extraction function34cextracts the object D1contained in the X-ray image I11, and the image generation function34dsets a region R1according to the position of the object D1out of the ultrasonic image data I2. Similarly, the object extraction function34cextracts the object D2contained in the X-ray image I12, and the image generation function34dsets a region R2according to the position of the object D2out of the ultrasonic image data I2. Similarly, the object extraction function34cextracts the object D3contained in the X-ray image I13, and the image generation function34dsets a region R3according to the position of the object D3out of the ultrasonic image data I2. The regions R1to R3illustrated inFIG. 8Aare elliptic regions based on the position and shape of the extracted object D.

The image generation function34dcan generate processed ultrasonic image data I21′ with the transmittance of the region R1out of the ultrasonic image data I2changed and generate a composite image I31as a combination of the processed ultrasonic image data I21′ and the X-ray image I11. Similarly, the image generation function34dcan generate processed ultrasonic image data I22′ with the transmittance of the region R2out of the ultrasonic image data I2changed and generate a composite image I32as a combination of the processed ultrasonic image data I22′ and the X-ray image I12. Similarly, the image generation function34dcan generate processed ultrasonic image data I23′ with the transmittance of the region R3out of the ultrasonic image data I2changed and generate a composite image I33as a combination of the processed ultrasonic image data I23′ and the X-ray image I13.

However, when the composite image I31, the composite image I32, the composite image I33, and the like are successively displayed, the position of the region R with the transmittance changed successively moves, and the region R may be difficult to visually recognize. In addition, it is difficult to accurately extract the object D moving due to the influence of the heartbeat and the like from each of the X-ray images I1, and the tracking of the region R to be set may be insufficient.

Given these circumstances, the image generation function34dsets the region R in accordance with a plurality of positions of the object D corresponding to the respective X-ray images I1. That is to say, the image generation function34ddoes not set one region R from one X-ray image I1but sets one region R from the X-ray images I1.

The image generation function34dsets a single combined region as a combination of the regions R1to R3illustrated inFIG. 8Aand changes the transmittance of the combined region, for example. The transmittance of each position in this combined region can be profiled as illustrated inFIG. 8B, for example. Specifically, the positions of the respective objects D1to D3are contained on the line segment A-B illustrated inFIG. 8A. The image generation function34dprofiles the transmittance such that the transmittance at the positions of the respective objects D1to D3is higher than that at other positions.

The image generation function34dgenerates processed ultrasonic image data I24′ with the transmittance of each position of the ultrasonic image data I2changed in accordance with the transmittance profile illustrated inFIG. 8B. The image generation function34dalso generates a composite image I34as a combination of the processed ultrasonic image data I24′ and the X-ray image I11. The image generation function34dalso generates a composite image I35as a combination of the processed ultrasonic image data I24′ and the X-ray image I12. The image generation function34dalso generates a composite image I36as a combination of the processed ultrasonic image data I24′ and the X-ray image I13.

When the composite image I34, the composite image I35, the composite image I36, and the like are successively displayed, the position of the region R (the combined region of the regions R1to R3) with the transmittance changed does not move, and no reduction in visibility occurs. Even if the extraction of the object D is inaccurate in any of the X-ray images I1, in many cases the object D will be contained in the region R. That is to say, in the case illustrated inFIG. 8AandFIG. 8B, even if the tracking of the moving object D is insufficient, a reduction in visibility can be inhibited.

Alternatively, the image generation function34dmay set a single region R4containing the objects D1to D3as illustrated inFIG. 9A. The transmittance of each position in the region R4can be profiled as illustrated inFIG. 9B, for example. That is to say, the image generation function34dprofiles the transmittance for each position on the line segment A-B such that the transmittance in the region R4is higher than that at other positions.

The image generation function34dgenerates processed ultrasonic image data I25′ with the transmittance of each position of the ultrasonic image data I2changed in accordance with the transmittance profile illustrated inFIG. 9B. The image generation function34dalso generates a composite image I37as a combination of the processed ultrasonic image data I25′ and the X-ray image I11. The image generation function34dalso generates a composite image I38as a combination of the processed ultrasonic image data I25′ and the X-ray image I12. The image generation function34dalso generates a composite image I39as a combination of the processed ultrasonic image data I25′ and the X-ray image I13.

When the composite image I37, the composite image I38, the composite image I39, and the like are successively displayed, the position of the region R4with the transmittance changed does not move, and no reduction in visibility occurs. Even if the extraction of the object D is inaccurate in any of the X-ray images I1, in many cases the object D will be contained in the region R4. That is to say, in the case illustrated inFIG. 9AandFIG. 9B, even if the tracking of the moving object D is insufficient, a reduction in visibility can be inhibited. In the case illustrated inFIG. 9AandFIG. 9B, the transmittance profile is smoother than that in the case illustrated inFIG. 8AandFIG. 8B, and thus the visibility of the object D can further be improved.

Although the above describes a case in which the region R is set from three X-ray images I1, the number of the X-ray images I1used to set the region R can be changed as desired. The X-ray diagnostic apparatus10repeats the acquisition of the X-ray image I1for a period corresponding to one heartbeat of the subject P at a certain frame rate, for example. In this case, the image generation function34dcan set the region R based on the X-ray images I1corresponding to one heartbeat of the subject P. Alternatively, the image generation function34dmay set the region R based on the X-ray images I1corresponding to a plurality of heartbeats of the subject P. Although the above describes a case in which the object D is influenced by the heartbeat, the same can be applied to a case in which the object D moves due to the breathing of subject P, for example.

The first and second embodiments have been described; various different forms may be performed other than the embodiments described above.

The embodiments described above describe a case in which the object D is the medical device such as a stent, for example. However, embodiments are not limited to this example. The same can be applied to a case in which a blood vessel, an organ, or the like of the subject P is selected as the object D, for example. Any other structures can be selected as the object D so long as they appear on the X-ray image.

The embodiments described above describe a case in which the processing on the ultrasonic image data I2is performed based on the extraction result of the object D by the object extraction function34cto generate the processed ultrasonic image data I2′. However, embodiments are not limited to this example. The image generation function34dmay perform processing based on an input operation from the user on the ultrasonic image data I2to generate the processed ultrasonic image data I2′, for example.

The image generation function34dperforms processing based on both the extraction result of the object D by the object extraction function34cand the input operation from the user on the ultrasonic image data I2to generate the processed ultrasonic image data I2′, for example. To give an example, the image generation function34dfirst sets the region R in the ultrasonic image data I2in accordance with the position of the object extracted by the object extraction function34c. The output function34edisplays the set region R on the display32, and the image generation function34dreceives an operation to process the position and shape of the region R from the user. The image generation function34dchanges the transmittance of the region R after being processed based on the input operation from the user out of the ultrasonic image data I2to generate the processed ultrasonic image data I2′.

The image generation function34dreceives an input operation from the user when the object D has not been extracted from the X-ray image I1, for example. The object extraction function34ccannot necessarily recognize the object D due to image noise in the X-ray image I1and the like, for example. In such a case, the output function34edisplays the X-ray image I1on the display32, and the image generation function34dreceives an operation to designate the position of the object D from the user. The image generation function34dchanges the transmittance of the region R corresponding to the position designated by the user out of the ultrasonic image data I2to generate the processed ultrasonic image data I2′. Alternatively, the output function34edisplays the ultrasonic image data I2on the display32, and the image generation function34dreceives an operation to set the position and shape of the region R from the user. The image generation function34dchanges the transmittance of the region R set by the user out of the ultrasonic image data I2to generate the processed ultrasonic image data I2′.

The embodiments described above describe a case in which the processing circuitry34of the medical image processing apparatus30executes the various kinds of functions such as the X-ray image acquisition function34a, the ultrasonic image acquisition function34b, the object extraction function34c, the image generation function34d, and the output function34e. However, embodiments are not limited to this example. The processing circuitry108of the X-ray diagnostic apparatus10may execute functions corresponding to the functions of the processing circuitry34, for example.

The following describes this point with reference toFIG. 10.FIG. 10is a block diagram of an example of the configuration of the X-ray diagnostic apparatus10according to a third embodiment. The X-ray diagnostic apparatus10illustrated inFIG. 10differs from the X-ray diagnostic apparatus10illustrated inFIG. 2in that the processing circuitry108further has an ultrasonic image acquisition function108c, an object extraction function108d, and an image generation function108e.

The ultrasonic image acquisition function108cis a function corresponding to the ultrasonic image acquisition function34b. The object extraction function34cis a function corresponding to the object extraction function108d. The image generation function34dis a function corresponding to the image generation function108e. The ultrasonic image acquisition function34bis an example of the ultrasonic image acquisition unit. The object extraction function34cis an example of the object extraction function unit. The image generation function34dis an example of the image generation function unit.

The acquisition function108aacquires the X-ray image I1about the subject P, for example. The ultrasonic image acquisition function34bacquires the ultrasonic image data12about the subject P via the network NW. The object extraction function108dextracts the object D contained in the X-ray image I1. The image generation function108eperforms the processing based on the position of the extracted object D on the ultrasonic image data I2in accordance with the relative positional relation between the coordinate system in the X-ray image I1and the coordinate system in the ultrasonic image data I2to generate the composite image I3as a combination of the processed ultrasonic image data I2′ after being subjected to the processing and the X-ray image I1. The output function108bcan display the generated composite image I3on the display106.

The term “processor” used in the above description means a circuit such as a CPU, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), or a field programmable gate array (FPGA), for example), for example. When the processor is a CPU, for example, the processor reads a computer program stored in a memory and executes it to implement a function. On the other hand, when the processor is an ASIC, for example, in place of storing the computer program in the memory, the function is directly embedded in the circuitry of the processor as a logic circuit. Each processor of the embodiments is not limited to being configured as a single circuit for each processor but may also be configured as one processor by combining a plurality of independent circuits to implement its functions. Further, a plurality of components in each drawing may be integrated into one processor to implement their functions.

The above inFIG. 1describes a case in which the single memory33stores therein the computer program corresponding to each processing function of the processing circuitry34. The above inFIG. 2andFIG. 10describes a case in which the single memory107stores therein the computer program corresponding to each processing function of the processing circuitry108. However, embodiments are not limited to this example. A plurality of memories33may be placed in a distributed manner, and the processing circuitry34may read from the individual memory33the corresponding computer program, for example. Similarly, a plurality of memories107may be placed in a distributed manner, and the processing circuitry108may read from the individual memory107the corresponding computer program. In place of storing the computer program in the memory33or the memory107, the computer program may directly be embedded in the circuitry of the processor. In this case, the processor reads the computer program embedded in the circuitry and executes it to implement its function.

The components of each apparatus according to the embodiments described above are functionally conceptual ones and do not necessarily need to be physically configured as illustrated in the drawing. That is to say, the specific form of the dispersion and integration of each apparatus is not limited to the one illustrated in the drawing, but the whole or part thereof can be configured in a functionally or physically distributed and integrated manner in any unit in accordance with various kinds of loads, use conditions, and the like. Further, the whole or any part of the processing functions performed by each apparatus can be implemented by a CPU and a computer program that is analyzed and executed by the CPU or be implemented as hardware by wired logic.

The method of medical image processing described in the embodiments described above can be implemented by executing a computer program prepared in advance on a computer such as a personal computer or a workstation. This computer program can be distributed via a network such as the Internet. This computer program can also be executed by being recorded on a computer-readable, non-transitory recording medium such as a hard disk, flexible disk (FD), a compact disc read only memory (CD-ROM), magneto-optical (MO), or a digital versatile disc (DVD) and being read from the recording medium by a computer.

At least one of the embodiments described above can improve the visibility of the object contained in the X-ray image in the composite image of the X-ray image and the ultrasonic image data.

Note 1. A medical image processing apparatus comprising:an X-ray image acquisition unit configured to acquire an X-ray image about a subject;an ultrasonic image acquisition unit configured to acquire an ultrasonic image data about the subject;an object extraction unit configured to extract an object contained in the X-ray image; andan image generation unit configured to perform processing based on a position of the extracted object on the ultrasonic image data in accordance with a relative positional relation between a coordinate system in the X-ray image and a coordinate system in the ultrasonic image data to generate a composite image as a combination of a processed ultrasonic image data after being subjected to the processing and the X-ray image.
Note 2. The image generation unit may change transmittance of a region corresponding to the position of the object out of the ultrasonic image data as the processing.
Note 3. The processing based on a position of the extracted object may include increasing transmittance of a region corresponding to the position of the object out of the ultrasonic image data.
Note 4. The object may be an object that a user focuses on. Note 5. The object may be a medical device operated by a user, a blood vessel in the travel direction of the medical device, or a region to be treated.
Note 6. The X-ray image acquisition unit may acquire X-ray images of a plurality of time phase,

the object extraction unit may extract the object from each of the X-ray images, and

the image generation unit may set the region in accordance with a plurality of positions of the object corresponding to the respective X-ray images to change the transmittance of the region.

Note 7. The ultrasonic image data may be a three-dimensional image data, and

the image generation unit may change transmittance of a three-dimensional region corresponding to the position of the object out of the ultrasonic image data to generate the processed ultrasonic image data.

Note 8. The ultrasonic image data may be a two-dimensional image data, and

the image generation unit may change transmittance of a two-dimensional region corresponding to the position of the object out of the ultrasonic image data to generate the processed ultrasonic image data.

Note 9. The X-ray image acquisition unit may, each time the X-ray image is newly acquired from the subject by an X-ray diagnostic apparatus, successively acquire the X-ray image,

the ultrasonic image data acquisition unit may, each time the ultrasonic image data is newly acquired from the subject by an ultrasonic diagnostic apparatus, successively acquire the ultrasonic image data,

the object extraction unit may successively extract the object from the newly acquired X-ray image, and

the image generation unit may successively perform the processing on the newly acquired ultrasonic image data in accordance with the positional relation to successively generate the composite image as a combination of the processed ultrasonic image data after being subjected to the processing and the newly acquired X-ray image.

Note 10. The image generation unit may perform processing based on an input operation from a user on the ultrasonic image data to generate the processed ultrasonic image data.
Note 11. The image generation unit may changes the transmittance of the region corresponding to the position of the object in accordance with rotation of a mouse wheel or operation of a certain bar on an UI.
Note 12. The image generation unit may receive the input operation when the object has not been extracted from the X-ray image.
Note 13. The image generation unit may extract an ultrasonic probe used to acquire the ultrasonic image data from the X-ray image to identify the positional relation.
Note 14. The image generation unit may perform the processing based on a position and shape of the extracted object on the ultrasonic image data.
Note 15. An output unit configured to display the composite image and the X-ray image side by side may further be included.
Note 16. An X-ray diagnostic apparatus comprising:

an acquisition unit configured to acquire an X-ray image about a subject;

an ultrasonic image data acquisition unit configured to acquire an ultrasonic image data about the subject;

an object extraction unit configured to extract an object contained in the X-ray image; and

an image generation unit configured to perform processing based on a position of the extracted object on the ultrasonic image data in accordance with a relative positional relation between a coordinate system in the X-ray image and a coordinate system in the ultrasonic image data to generate a composite image as a combination of a processed ultrasonic image data after being subjected to the processing and the X-ray image.

Note 17. A computer program causing a computer to execute each component of the medical image processing apparatus.
Note 18. A method of medical image processing comprising:

acquiring an X-ray image about a subject;

acquiring an ultrasonic image data about the subject;

extracting an object contained in the X-ray image; and

performing processing based on a position of the extracted object on the ultrasonic image data in accordance with a relative positional relation between a coordinate system in the X-ray image and a coordinate system in the ultrasonic image data to generate a composite image as a combination of a processed ultrasonic image data after being subjected to the processing and the X-ray image.

Note 19. In the method of medical image processing, transmittance of a region corresponding to the position of the object out of the ultrasonic image data may be changed as the processing.
Note 20. The processing based on a position of the extracted object may include increasing transmittance of a region corresponding to the position of the object out of the ultrasonic image data.
Note 21. The object may be an object that a user focuses on.
Note 22. The object may be a medical device operated by a user, a blood vessel in the travel direction of the medical device, or a region to be treated.
Note 23. In the method of medical image processing,

X-ray images of a plurality of time phase may be acquired,

the object may be extracted from each of the X-ray images, and

the region may be set in accordance with a plurality of positions of the object corresponding to the respective X-ray images to change the transmittance of the region.

Note 24. In the method of medical image processing,

the ultrasonic image data may be a three-dimensional image data, and

transmittance of a three-dimensional region corresponding to the position of the object out of the ultrasonic image data may be changed to generate the processed ultrasonic image data.

Note 25. In the method of medical image processing,

The ultrasonic image data may be a two-dimensional image data, and

transmittance of a two-dimensional region corresponding to the position of the object out of the ultrasonic image data may be changed to generate the processed ultrasonic image data.

Note 26. In the method of medical image processing,

each time the X-ray image is newly acquired from the subject by an X-ray diagnostic apparatus, the X-ray image may successively be acquired,

each time the ultrasonic image data is newly acquired from the subject by an ultrasonic diagnostic apparatus, the ultrasonic image data may successively be acquired,

the object may successively be extracted from the newly acquired X-ray image, and

the processing may successively be performed on the newly acquired ultrasonic image data in accordance with the positional relation to successively generate the composite image as a combination of the processed ultrasonic image data after being subjected to the processing and the newly acquired X-ray image.

Note 27. In the method of medical image processing,

processing based on an input operation from a user may be performed on the ultrasonic image data to generate the processed ultrasonic image data.

Note 28. In the method of medical image processing,

the transmittance of the region corresponding to the position of the object may be changed in accordance with rotation of a mouse wheel or operation of a certain bar on an UI.

Note 29. In the method of medical image processing,

the input operation may be received when the object has not been extracted from the X-ray image.

Note 30. The method of medical image processing according to claim13, in the method of medical image processing, an ultrasonic probe used to acquire the ultrasonic image data is extracted from the X-ray image to identify the positional relation.
Note 31. In the method of medical image processing, the processing based on a position and shape of the extracted object may be performed on the ultrasonic image data.
Note 32. In the method of medical image processing, the composite image and the X-ray image may be displayed side by side.