Patent Description:
Ultrasonic imaging apparatuses irradiate ultrasonic waves on a subject and form an image of an internal structure of the subject from reflection signals of the waves. Therefore, they enable non-invasive and real-time observation of a patient.

On the other hand, other medical imaging apparatuses such as X-ray CT (Computed Tomography) apparatuses and MRI (Magnetic Resonance Imaging) apparatuses enable imaging of a wide area with high resolution, and therefore fine lesions or relationship of internal organs can be easily grasped with them. For example, tumors such as liver cancer can be found in an MRI image or an X-ray CT image at an early stage.

Patent document <NUM> discloses a diagnostic imaging system that obtains an ultrasonograph of an arbitrary section with an ultrasound probe having a position sensor, constructs a two-dimensional image of the corresponding section from volume data of the same subject obtained beforehand with another medical imaging apparatus, and displays both the images side by side. In this technique, a two-dimensional image of a current position of the ultrasound probe is constructed in real time from volume data obtained beforehand with another medical imaging apparatus. Therefore, it is necessary to perform a processing for matching the position of the ultrasonograph and the corresponding position in the volume data beforehand. Patent document <NUM> discloses a procedure for matching the positions. First, a user such as medical practitioner manually moves an ultrasound probe on a subject to search for a position where an ultrasonograph including a disease part suitable as an alignment image can be obtained. Then, the user selects a position of the disease part and a position of a characteristic structure other than the disease part on the ultrasonograph obtained by the search. The user further specifies a position corresponding to the position of the disease part and a position of the characteristic structure of the ultrasonograph on an image of volume data obtained with another medical imaging apparatus by manually using a mouse or the like. The diagnostic imaging system aligns the volume data so that the two positions on the ultrasonograph selected by the user and the two positions specified on the image obtained with the other medical imaging apparatus_ match with each other. Patent document <NUM> relates to an information processing apparatus to register an ultrasonic image and a three-dimensional medical image, which includes a coordinate transformation unit, which transforms coordinates of the medical image or the ultrasonic image with reference to a contact position on the ultrasonic image, so that image information of the medical image matches that of the ultrasonic image. Patent document <NUM> relates to a system and method for image registration, which includes tracking a scanner probe in a position along a skin surface of a patient.

However, according to the technique of Patent document <NUM>, in order to carry out alignment of the ultrasonograph and the volume data obtained with another medical imaging apparatus, a user such as medical practitioner must carry out a plurality of the complicated manual operations of manually moving an ultrasound probe to search for a position of the ultrasonograph suitable as an alignment image, manually selecting a plurality of positions on the ultrasonograph, and further specifying a plurality of corresponding positions on the image of volume data. These complicated manual operations for the alignment not only impose heavy burdens on the user who is a medical practitioner, but also impose heavy burdens on the subject being waited on a bed during the manual operations for the alignment with being applied with the ultrasound probe. Further, since the position used as the basis of the alignment consists of two positions on one ultrasonograph, highly precise three-dimensional alignment is difficult.

Further, although the technique described in Patent document <NUM> supposes use thereof with a treatment or surgical operation not accompanied by abdominal incision of the subject such as radio-frequency ablation (RFA), it is desired in recent years to directly put an ultrasound probe on an internal organ of a subject in an abdominally incised state and thereby confirm a region to be operated such as tumors with an ultrasonograph and a corresponding MRI image or CT image of high resolution during the surgical operation. Therefore, it is desired to avoid as much as possible that the user such as medical practitioner touches a switch, mouse, or the like of an input device with the hand for the alignment of the volume data during the operation. It is also desired to perform the alignment of the volume data in a short time as much as possible, in order to reduce the burden imposed on the subject in an abdominally incised state. Furthermore, in order to correctly confirm positions of tumor or the like on an ultrasonograph, exact alignment of the volume data of high resolution and the ultrasonograph is desired.

An object of the present disclosure is to provide an ultrasonic imaging apparatus that enables correct alignment of an ultrasonograph and volume data obtained beforehand without requiring complicated operations of users.

The ultrasonic imaging apparatus of the present disclosure comprises an ultrasound probe that transmits a ultrasonic wave to a subject and receives a ultrasonic wave from the subject, a position sensor attached to the ultrasound probe, an image generator that generates an ultrasonograph from a signal received by the ultrasound probe and generates first volume data from the ultrasonograph and positional information of the ultrasound probe obtained from the position sensor, and an image processing device that receives second volume data obtained for the subject by another external imaging apparatus and processes them. The image processing device is provided with an aligner that carries out alignment of the first volume data and the second volume data. The aligner comprises a receptor that receives a predetermined imaging part selected from a plurality of imaging parts of a subject from a user, and a rotation processor. The rotation processor initially rotates the second volume data by a rotation angle corresponding to the imaging part that is received by the receptor, and further carries out alignment of the initially rotated second volume data and the first volume data.

According to the present invention, alignment of ultrasonic volume data and volume data obtained with another imaging apparatus can be automatically and correctly carried out.

Hereafter, embodiments of the present invention will be explained in detail with reference to the drawings. In all of the drawings for explanation of the embodiments, the same parts are indicated with the same numerals in principle, and repetitive explanations thereof are omitted.

The inventors of the present invention considered that the cause of the complicated operations for the alignment of ultrasonograph and volume data is that imaging direction and view of ultrasonic imaging apparatus are greatly different from those of other medical imaging apparatuses such as MRI and X-ray CT apparatuses. It is difficult to apply a general automatic alignment procedure to alignment of images of greatly different imaging directions and views. Therefore, according to the present invention, an initial rotation processing is carries out according to type of imaging region so that imaging directions of ultrasonograph and image obtained with another medical imaging apparatus coincide to each other, and the automatic alignment procedure can be applied to them.

When an ultrasonic imaging apparatus is used, while an ultrasound probe is moved along a body surface of a subject or surface of internal organ of an abdominally incised subject, ultrasonic waves are transmitted from the ultrasound probe to the subject to scan the subject, reflection waves are received, and an image is obtained from them. Therefore, when the body surface or surface of internal organ of the subject is curving, the ultrasound probe inclines along the curved surface, and there is obtained an ultrasonograph of a surface for which signals are transmitted and received with the ultrasound probe applied to the surface at the inclination angle. Further, when a user who is a medical practitioner desires to see an ultrasonograph of a direction having a desired angle corresponding to a certain structure in an internal organ, the user may incline the ultrasound probe by the desired angle. Furthermore, when, for example, the longitudinal direction of internal organ is slanting to the body axis, a user may successively obtain images with moving the ultrasound probe slantly to the body axis. Therefore, the plane of the ultrasonograph (scanning plane) is not perpendicular to the body axis of the subject, but inclines with respect to the body axis depending on degree of curve of body surface or organ surface, and direction in which the user puts the ultrasound probe on the surface. On the other hand, with an X-ray CT apparatus or MRI apparatus, an image of a section perpendicular to the direction of the body axis of the subject is obtained, and such imaging is repeated a plurality of times to obtain volume data.

Therefore, in order to carry out alignment of both the images by an automatic alignment method, it is necessary to rotate volume data obtained with an X-ray CT apparatus or MRI apparatus according to the inclination from the body axis in the ultrasonograph. General automatic alignment methods are methods of automatically extracting a characteristic shape included in the two images, and calculating moving magnitude through pattern matching or the like so that the characteristic shapes of the images coincide to each other. The inclination angle with respect to the body axis in the ultrasonograph is not a small angle, and differs depending on type of organ, direction along which the user desired to see it, or the like. Since resolutions of ultrasonograph and volume data obtained with an X-ray CT apparatus or MRI apparatus greatly differ, extracted shapes differ even if they are those for the same characteristic shape. In addition, the extracted shapes are three-dimensional shapes. For these reasons, the automatic alignment is difficult, and in order to carry out automatic alignment, huge amount of calculation is required. Therefore, such calculation requires much time to keep the abdominally incised subject and user waiting, and thus it has conventionally been difficult to actually carry out it.

According to the present invention, such a configuration as described below is employed in consideration that shape of surface of a specific organ is substantially the same although there is some individual difference, and the direction along which users desire to see an imaging part (part of internal organ) is also substantially the same for each imaging part. That is, volume data obtained with another medical imaging apparatus are first initially rotated by a rotation amount corresponding to a specific part of internal organ. Then, alignment of ultrasonograph and volume data obtained with another medical imaging apparatus is carried out by the automatic alignment method. This makes it possible to perform automatic alignment with sufficient accuracy in a short time. As the rotation amount of the initial rotation, a value determined beforehand for each imaging part (part of internal organ) may be used as in the embodiment <NUM>, or it may be calculated for each imaging part as in the embodiment <NUM>.

As for the configuration of the ultrasonic imaging apparatus of the present invention, the apparatus comprises, for example, an ultrasound probe <NUM>, a position sensor <NUM>, an image generator <NUM>, and an image processing device <NUM>, as shown in <FIG>. The ultrasound probe <NUM> transmits ultrasonic waves to a subject <NUM>, and receives ultrasonic waves reflected by the subject <NUM>. The position sensor <NUM> is attached to the ultrasound probe <NUM>. The image generator <NUM> generates an ultrasonograph from the signals received by the ultrasound probe <NUM>, and generates first volume data from the ultrasonograph and positional information of the ultrasound probe <NUM> obtained from the position sensor <NUM>. The image processing device <NUM> receives second volume data obtained by another external imaging apparatus for the subject <NUM>, and processes them. For this processing, the image processing device <NUM> carries out alignment of the first volume data and the second volume data. The image processing device <NUM> receives a predetermined imaging part selected by a user from a plurality of imaging parts of the subject, obtains a rotation angle corresponding to the received imaging part on the basis of relation of a plurality of imaging parts and rotation angles defined beforehand, and initially rotates the second volume data by the obtained rotation angle. The image processing device <NUM> further carries out alignment of the initially rotated second volume data and the first volume data to enable automatic alignment with good accuracy in a short time.

Hereafter, specific configuration of the ultrasonic imaging apparatus of the embodiment <NUM> will be further explained. As shown in <FIG> and mentioned above, the ultrasonic imaging apparatus of this embodiment comprises the ultrasound probe <NUM>, position sensor <NUM>, image generator <NUM>, and image processing device <NUM>, and further comprises a transmitter <NUM>, a transmission and reception switch <NUM>, a receptor <NUM>, a position detection unit <NUM>, a user interface <NUM>, and a controller <NUM>. Under control by the controller <NUM>, the transmitter <NUM> generates transmission signals and sends them to each of a plurality of ultrasonic wave devices constituting the ultrasound probe <NUM>. As a result, each of the plurality of the ultrasonic devices of the ultrasound probe <NUM> transmits ultrasonic waves toward the subject <NUM>. The ultrasonic waves, for example, reflected by the subject <NUM> reach the plurality of the ultrasonic devices of the ultrasound probe <NUM> again, and are received thereby and converted into electric signals. The signals received by the ultrasonic devices are delayed by the receptor <NUM> for predetermined delaying amounts corresponding to the position of reception focus, and then added (phasing addition). This processing is repeated for a plurality of the reception focuses. The signals subjected to the phasing addition are sent to the image generator <NUM>. The transmission and reception switch <NUM> selectively connects the transmitter <NUM> or receptor <NUM> to the ultrasound probe <NUM>.

The position detection unit <NUM> detects the position of the ultrasound probe <NUM> from output of the position sensor <NUM>. For example, a magnetic sensor unit can be used as the position detection unit <NUM>. The position detection unit <NUM> forms a magnetic field space, the position sensor <NUM> detects the magnetic field, and coordinates from a position serving as a base point can be thereby detected.

The image generator <NUM> carries out processings such as arranging the phase-added signals received from the receptor <NUM> at corresponding positions to generate an ultrasonograph. The image generator <NUM> further receives positional information of the ultrasound probe <NUM> at that position from the position detection unit <NUM>, and imparts positional information to the ultrasonograph. As a result, when a user moves the ultrasound probe <NUM> in a predetermined area, the image generator <NUM> generates an ultrasonograph imparted with the positional information of the ultrasound probe <NUM> at that position, and volume data of three-dimensional ultrasonograph (henceforth also referred to as ultrasonic volume data or first volume data) can be thereby generated.

The image processing device <NUM> receives volume data obtained for the subject <NUM> by another imaging apparatus (second volume data) via the user interface <NUM>, and carries out alignment of the first volume data and the second volume data, and so forth. In the following explanation, the other imaging apparatus such as ultrasonic MRI apparatus, X-ray CT apparatus, and other ultrasonic diagnostic apparatuses is referred to as medical modality. In this embodiment, for example, an X-ray CT apparatus is used as the medical modality, and volume data of X-ray CT apparatus are referred to as CT volume data (second volume data).

Hereafter, configurations and operations of the image processing device <NUM> and the user interface <NUM> will be explained in detail.

<FIG> is a block diagram showing the hardware configurations of the image processing device <NUM> and the user interface <NUM>. The hardware configurations shown in <FIG> are commonly used also in the other embodiments mentioned later.

The image processing device comprises and is constituted by CPU (processor) <NUM>, ROM (non-volatile memory, read-only storage medium) <NUM>, RAM (volatile memory, data-writable storage medium) <NUM>, a memory <NUM>, and a display controller <NUM>. The user interface <NUM> comprises and is constituted by an image inputter <NUM>, a medium inputter <NUM>, an input controller <NUM>, and an input device <NUM>. These parts, and the ultrasonograph generator <NUM> and position detection unit <NUM> are connected with each other via a data bus <NUM>. A display <NUM> is connected to the display controller <NUM>.

A program and data required for realizing the operation of the image processing device <NUM> in the operation processing performed by CPU <NUM> are stored beforehand in at least one of ROM <NUM> and RAM <NUM>. When CPU <NUM> executes the program stored beforehand in at least one of ROM <NUM> and RAM <NUM>, various processings of the image processing device <NUM> are realized. The program executed by CPU <NUM> may also be stored in a storage medium (for example, optical disc) <NUM>, and a medium inputter <NUM> (for example, optical disc drive) may read the program, and load it in RAM <NUM>. The program may also be stored in a storage device <NUM>, and loaded in RAM <NUM> from the storage device <NUM>. The program may also be stored in ROM <NUM> beforehand.

The image inputter <NUM> is an interface for inputting CT volume data (second volume data) obtained with an X-ray CT apparatus (medical modality) <NUM>. The storage device <NUM> is a magnetic storage device that stores the second volume data and so forth inputted via the image inputter <NUM>. The storage device <NUM> may be provided with a non-volatile semiconductor storage medium (for example, flash memory). An external storage device connected via a network or the like may also be used.

The input device <NUM> is a device for receiving operations of a user, and may comprise, for example, a keyboard, trackball, navigational panel, foot switch, and so forth. The input controller <NUM> is an interface for receiving the input for the operations inputted by the user. The input for the operations received by the input controller <NUM> is processed by CPU <NUM>.

The display controller <NUM> performs control so that, for example, image data obtained by processings in CPU <NUM> are displayed on the display <NUM>. The display <NUM> displays an image under control by the display controller <NUM>.

<FIG> is a functional block diagram showing the functions of the image processing device <NUM>. As shown in <FIG>, the image processing device <NUM> comprises an ultrasonic volume data (first volume data) acquisitor <NUM>, a characteristic data extractor <NUM> for ultrasonic volume data, a CT volume data receptor <NUM>, and a characteristic data extractor <NUM> for CT volume data. The image processing device <NUM> also comprises a CT characteristic data initially rotator <NUM> and a characteristic data aligner <NUM> as an aligner. It further comprises an image display <NUM>, an alignment result confirmation and initial rotation redo part <NUM>, and a CT image calculator <NUM>.

The image processing device <NUM> further comprises an imaging part receptor <NUM> and a table <NUM> showing relation between a plurality of imaging parts and rotation angles. As shown in <FIG>, the receptor <NUM> is a functional block for displaying a screen on the display <NUM> for receiving zones S1 to S8 of an internal organ (liver) chosen from the input device <NUM> by a user, and receiving a zone selected from the zones S1 to S8 by the user via the input device <NUM>. The zones S1 to S8 are set according to an anatomically known sectionalization method, and surface shapes and internal structures of the zones are also anatomically known. Therefore, the angles at which a user puts the ultrasound probe <NUM> on the organ are substantially fixed for the zones depending on the surface shapes and internal structures of the zones, thus the rotation angles for the ultrasonograph to be obtained and with respect to the body axis of the subject are calculated beforehand, and the table <NUM> showing the angles corresponding to the zones is prepared as shown in <FIG>. The table <NUM> is stored beforehand in ROM <NUM>, RAM <NUM>, or the storage device <NUM>. Distances for parallel translation of the zones are also shown in the table shown in <FIG>. These are distances of parallel translation for the cases where parallel translation is also required besides rotation, when the CT image is coincided by rotation to the ultrasonograph obtained with the inclination. As for alignment by parallel translation, the alignment can be performed by a known alignment method with processing in the characteristic data aligner <NUM>, and therefore the table <NUM> may not necessarily contain parallel translation distances. Since it is considered that the rotation angle should differ depending on sex and age of the subject, and whether the subject is abdominally incised or not, a plurality of kinds of different tables <NUM> may be prepared, and one of them may be chosen and used.

Hereafter, the processings performed by the image processing device <NUM> will be explained with reference to the flowchart shown in <FIG>.

First, in the step S201, the CT volume data receptor <NUM> receives CT volume data from the imaging apparatus (X-ray CT apparatus) <NUM> via the image inputter <NUM>.

In the step S202, the imaging part receptor <NUM> displays such an image for receiving specification of a zone of an internal organ as shown in <FIG> on the display <NUM>, and receives specification of a zone of an internal organ (S1 to S8) made by a user through a touch on a touch panel of the screen, or an operation using the input device <NUM> such as foot switch.

In the step S203, the ultrasonic volume data acquisitor <NUM> displays an indication urging the user to put the ultrasound probe <NUM> on the zone of the internal organ and move it (perform scanning) on the display <NUM>. When the user moves the ultrasound probe <NUM> in the zone of the internal organ, three-dimensional ultrasonic volume data are generated by the transmitter <NUM>, the receptor <NUM>, and the image generator <NUM>. The ultrasonic volume data acquisitor <NUM> receives the ultrasonic volume data generated by the image generator <NUM>. For example, if the user performs scanning with putting the ultrasound probe on the liver zone S5, data for the potal vein, which is a characteristic part of the liver, are included in the ultrasonic volume data.

In the step S204, the characteristic data extractors <NUM> and <NUM> extract point group data of the characteristic parts such as blood vessel contained in the ultrasonic volume data and CT volume data, respectively. The extracted blood vessel data are three-dimensional coordinate data of voxels in segmented vascular regions. Blood vessel data extracted from the ultrasonic volume data are shown in <FIG>, and blood vessel data extracted from the CT volume data are shown in <FIG>. Although <FIG> show blood vessel data of the corresponding internal organ zone, resolutions of the ultrasonic volume data and CT device volume data greatly differ from each other, and in addition, the imaging directions and views thereof are also greatly differ from each other. Therefore, the shapes of the blood vessel data of the both also greatly differ from each other.

In the step S205, the CT characteristic data initially rotator <NUM> rotates the CT blood vessel data extracted in the step S204 according to the internal organ zone specified in the step S202. That is, the CT characteristic data initially rotator <NUM> refers to the table <NUM> of the internal organ zones and rotation angles shown in <FIG>, reads the rotation angle corresponding to the internal organ zone specified in the step S202, and rotates the CT blood vessel data by the read rotation angle (initial rotation). When parallel translation distances are included in the table <NUM>, it reads the parallel translation distance corresponding to the internal organ zone, and carries out parallel translation of the CT blood vessel data by that distance. The CT blood vessel data are thereby geometrically converted so that they are rotated by the angle corresponding to the inclination of the ultrasonograph specific to the internal organ zone, and therefore the initial positions of the ultrasonic blood vessel data and the CT blood vessel data are approximately coincided to each other. <FIG> shows the rotated CT blood vessel data. <FIG> shows that the direction of the rotated CT blood vessel data approximately corresponds to the direction of ultrasonic blood vessel data, and they show shapes that can be superimposed.

Then, in the step S206, the characteristic data aligner <NUM> performs alignment of the point groups of the ultrasonic blood vessel data and the rotated CT blood vessel data. Since the initial positions of the blood vessel data have already been approximately coincided in the step S205, alignment of the both can be performed by a known automatic alignment method. As such a known automatic alignment method, the known ICP (Iterative Closest Point) method can be used. By the ICP method, a point group of CT blood vessel data is geometrically converted (parallel translation and rotation), the distance from the corresponding point in the point group of the ultrasonic blood vessel data is obtained, and the calculation is repetitively performed so that the distance is minimized. The alignment of the both can be thereby carried out.

In the step S207, the image display <NUM> changes color of one of the aligned CT blood vessel data and ultrasonic blood vessel data after to generate a transparently superimposed image, and displays it on the display <NUM>. As shown in <FIG>, for example, there is displayed an image with which it can be confirmed that the aligned CT blood vessel data and ultrasonic blood vessel data can overlap with each other. The image display <NUM> may also apply the result of the alignment to the CT volume data, and display them so that they overlap with the ultrasonic volume data. The Image display <NUM> can also generate and display an image in which the aligned CT blood vessel data are transparently superimposed on the aligned CT volume data in a different color, and the ultrasonic blood vessel data are transparently superimposed on the ultrasonic volume data in a different color.

In a state that the image shown in <FIG> is displayed, the alignment result confirmation and initial rotation redo part <NUM> displays an indication inquiring whether the user judges that the alignment has been successfully performed or not on the display <NUM>, and receives judgment of the user via the input device <NUM>. When the user inputs a judgment that the alignment has been successfully performed via the input device <NUM>, the alignment processing is ended, and in the step S210, the alignment result confirmation and initial rotation redo part <NUM> performs the rotation and parallel translation performed for the CT blood vessel data in the steps S205 and S206 for the whole CT volume data to generate aligned CT volume data.

On the other hand, when the user judges that the alignment has been unsuccessful, the process advances to the steps S208 and S209, and the alignment result confirmation and initial rotation redo part <NUM> redoes the initial rotation by another method. That is, in the steps S208 and S209, it obtains the rotation angle for the initial rotation of the CT volume data by calculation without using the rotation angles of the table <NUM>.

First, in the step S208, sections including a characteristic part defined beforehand for each internal organ zone are extracted from the ultrasonic volume data and the CT volume data not initially rotated in the step S205, respectively. For example, such images of a characteristic section including the inferior vena cava of the liver as shown in <FIG> are extracted and generated from ultrasonic volume data and CT volume data, respectively. <FIG> shows an example of sectional image of the inferior vena cava extracted from ultrasonic volume data. <FIG> shows an example of sectional image of the inferior vena cava extracted from CT volume data. As the method of searching for and extracting characteristic section, for example, the AdaBoost method, which is a known method of machine learning, can be used.

Then, in the step S209, the rotation angle by which the CT blood vessel data should be rotated is calculated by using the positional information of the two extracted images of the characteristic section so that the images should coincide to each other. The CT blood vessel data are rotated by the calculated rotation angle (initial rotation).

The process returns to the step S206, and alignment of the point groups of the ultrasonic blood vessel data and the rotated CT blood vessel data is performed. Until it is judged in the step S207 that the alignment is successfully performed, the steps S206, S207 S208, and S209 are repeatedly performed. If it is judged by the user that the alignment is successfully performed, the process advances to the step S210, the rotation and parallel translation performed for the CT blood vessel data in the steps S209 and S206 are performed for the whole CT volume data to generate aligned CT volume data.

By the procedures explained above, CT volume data aligned so that they coincide to the ultrasonic volume data are generated.

Then, the CT image calculator <NUM> performs the process of the flowchart shown in <FIG> to generate a CT image of a section corresponding to the ultrasonograph generated by the ultrasound probe <NUM> at the current position, and display it with the ultrasonograph side by side.

In the step S501 shown in <FIG>, the CT image calculator <NUM> receives the aligned CT volume data generated in the step S210 shown in <FIG>.

In the step S502, the positional information of the ultrasound probe <NUM> at the time of the acquisition of the ultrasonic volume data is received from the image generator <NUM>. Then, when the user put the ultrasound probe <NUM> on the objective internal organ at a desired position, the image generator <NUM> is allowed to generate an ultrasonograph (step S503). At the same time, positional information of the ultrasound probe <NUM> is obtained from the position detection unit <NUM> (step S504).

The CT image calculator calculates positional relationship of the positional information of the ultrasound probe <NUM> at the time of the acquisition of the ultrasonic volume data obtained in the step S502, and the current positional information of the ultrasound probe <NUM> obtained in the step S504, and cuts out (calculates) a CT image of a section corresponding to the ultrasonograph generated in the step S503 from the aligned CT volume data received in the step S501 on the basis of the calculated positional relationship (step S505). By displaying the ultrasonograph obtained in the step S503, and the CT image obtained in the step S505 side by side, an ultrasonograph obtained with the ultrasound probe <NUM> and a CT sectional image of the same position can be generated and displayed in real time.

As explained above, according to this embodiment, CT volume data aligned so that they coincide to ultrasonic volume data can be generated. Since this alignment can be automatically carried out in a short time without calling on a user to perform complicated alignment processing, its burdens imposed on user and subject are small. Further, since the alignment is performed for three-dimensional data, i.e., ultrasonic volume data and CT volume data, accuracy of the alignment is high. Therefore, it becomes possible to display a CT image of high resolution or the like on an ultrasonograph of a narrow field including many noises so that the CT image highly precisely coincides to the ultrasonograph in real time, and therefore it becomes possible for a user to recognize even a small tumor or the like in the ultrasonograph.

According to this embodiment, the alignment processing of ultrasonic volume data and CT volume data is performed by extracting data of characteristic part (blood vessel), therefore the alignment can be carried out with blood vessel data of a smaller data amount compared with volume data, and thus calculation amount can be reduced. Accordingly, the alignment can be performed at high speed.

Although the embodiment <NUM> has been explained for the configuration that the image processing device <NUM> is provided in the inside of the ultrasonic imaging apparatus <NUM>, it is also possible to provide the image processing device <NUM> shown in <FIG> and <FIG> as an apparatus separate from the ultrasonic imaging apparatus <NUM>. In such a case, the image processing device <NUM> and the ultrasonic imaging apparatus <NUM> are connected via a signal wire or network. For example, the following configuration is employed. The image processing device <NUM> is implemented in an image processing device such as common computer or workstation, and connected with the ultrasonic imaging apparatus <NUM> via a network. The image processing device <NUM> receives ultrasonic volume data and CT volume data to be aligned from a client terminal via a network, and performs the alignment processing. The aligned CT volume data are transmitted to the ultrasonic imaging apparatus as the client terminal. It is thereby made unnecessary that the image processing device <NUM> that requires comparatively large operation amount is carried on the ultrasonic imaging apparatus <NUM>. The ultrasonic imaging apparatus <NUM> can perform the alignment processing by using operation ability of the image processing device <NUM> connected via a network. Therefore, there can be provided the ultrasonic imaging apparatus <NUM> that is small and simple, but can display an ultrasonograph and a CT image of the same section on real time.

In the embodiment <NUM>, CT volume data are initially rotated by a rotation angle obtained beforehand for each zone of internal organ. However, the present invention is not limited to such a configuration, and the angle for the initial rotation can also be obtained by calculation. This configuration will be explained as the embodiment <NUM>. In the explanation of the embodiment <NUM>, the same configurations and processings as those of the embodiment <NUM> are indicated with the same numerals, and explanations thereof are omitted.

The ultrasonic imaging apparatus of the embodiment <NUM> does not comprises the table <NUM> shown in <FIG> in which a rotation angle is matched with each imaging part. As shown in the flowchart of <FIG>, in the steps S201 to S204, the image processing device <NUM> extracts ultrasonic blood vessel data and CT blood vessel data in the same manner as that of the embodiment <NUM>. Then, the angle for the initial rotation of the CT blood vessel data is obtained by calculation through the steps S208 and S209 of the embodiment <NUM>, and the CT blood vessel data are initially rotates by the obtained angle.

Specifically, the process advances to the step S208 after the step S204, and sections including a predetermined characteristic section included in the zone of internal organ specified in the step S202 are extracted from the ultrasonic volume data and the CT volume data not initially rotated, respectively. For example, images of such a characteristic section including the inferior vena cava of the liver as shown in <FIG> are extracted from ultrasonic volume data and CT volume data, respectively. As the method of searching for and extracting characteristic section, for example, the AdaBoost method, which is a known method of machine learning, can be used.

Alignment is performed so that the initially rotated CT blood vessel data should coincide to the ultrasonic blood vessel data (step S206). Then, in the step S210, the rotation and parallel translation performed for the CT blood vessel data in the steps S209 and S206 are performed for the whole CT volume data to generate aligned CT volume data.

With the configuration of the embodiment <NUM>, the initial rotation angle can be obtained by calculation, and therefore the embodiment <NUM> has an advantage that the initial rotation can be performed with an initial rotation angle matching with actual ultrasonic volume data and CT volume data. Further, since automatic alignment is performed after the initial rotation, the same effect as that of the embodiment <NUM> can be obtained.

The embodiment <NUM> will be explained below.

In the embodiment <NUM>, alignment accuracy is improved by further performing rigid body alignment for the ultrasonic volume data subjected to the alignment according to the embodiment <NUM> or <NUM>, and aligned CT volume data.

The ultrasonic imaging apparatus of the embodiment <NUM> further comprises, in addition to the functional blocks shown in <FIG>, an image-based rigid body aligner <NUM> shown in <FIG> in the image processing device <NUM>. The image-based rigid body aligner <NUM> is a device for performing alignment of the aligned CT volume data obtained in the step S210 of the embodiment <NUM> shown in <FIG> or the step S210 of the embodiment <NUM> shown in <FIG> as a floating image <NUM>, and ultrasonic volume data as a reference image <NUM>.

The image-based rigid body aligner <NUM> comprises a characteristic region sampler <NUM> and an aligner <NUM>.

<FIG> shows a flowchart for explaining the while operation of the rigid body aligner <NUM>. The steps shown in <FIG> will be explained below.

The ultrasonic volume data as the reference image <NUM>, and the aligned CT volume data obtained in the step S210 of the embodiment <NUM> or <NUM> as the floating image are inputted into the rigid body aligner <NUM> (S301). The characteristic region sampler <NUM> receives characteristic data of the ultrasonic volume data (ultrasonic blood vessel data) <NUM> extracted in the step S204 of the embodiments <NUM> and <NUM> (S302). The characteristic region sampler <NUM> extracts image sampling points at the coordinates of the reference image <NUM> and the characteristic data of the reference image <NUM>, and outputs them to the aligner <NUM> (S303). These image sampling points are used for calculating image similarity of the reference image <NUM> and the floating image <NUM> in the aligner <NUM>.

As for the extraction of the image sampling points, although all the pixels of the reference image <NUM> as the object of the alignment processing and the imaging region of the characteristic data of the reference image <NUM> may be extracted as the sampling points, only the pixels at nodes of grid placed on the images may be used as the sampling points in order to improve the speed of the alignment processing. A predetermined number of coordinates may be randomly chosen from coordinates of a region as the object of the sampling, for example, the characteristic data of the reference image <NUM>, and luminosity values at the obtained coordinates may be used as luminosity values of the sampling points.

The aligner <NUM> comprises geometrical conversion information <NUM>, a coordinate geometrical converter <NUM>, an image similarity calculator <NUM>, an image similarity maximization part <NUM>, and a floating image geometrical converter <NUM>.

The geometrical conversion information <NUM> is information representing the result of the alignment of the ultrasonic blood vessel data and the CT blood vessel data. That is, as the initial value for the image-based rigid body alignment performed in the aligner <NUM>, the result of the alignment of the ultrasonic blood vessel data and the CT blood vessel data is used.

The coordinate geometrical converter <NUM> geometrically converts the coordinates of the sampling points extracted from the reference image <NUM> to coordinates of corresponding points in the floating image <NUM> (S304). The image similarity calculator <NUM> obtains luminosity data at the sampling points of the reference image <NUM>, and luminosity data at the corresponding sampling points of the floating image <NUM>. The image similarity calculator <NUM> applies a predetermined evaluation function to the luminosity data at these sampling points to calculate image similarity between the reference image <NUM> and the floating image <NUM> (S305). As the image similarity, a known mutual information amount can be used.

The image similarity maximization part <NUM> obtains the image similarity between the reference image <NUM> and the floating image <NUM> calculated by the image similarity calculator <NUM>. In this part, convergence calculation is carried out in order to obtain geometrical conversion information that provides the maximum (or local maximum) of the image similarity between the reference image <NUM> and the floating image <NUM> (S306). When the image similarity has not converged in the step S306, the image similarity maximization part <NUM> updates the geometrical conversion information <NUM> in order to obtain a higher image similarity (S307). Then, the steps S304 to S306 are performed again by using the updated geometrical conversion information <NUM>.

On the other hand, when the image similarity has converged in the step S306, the aligner <NUM> geometrically converts the floating image <NUM> by using the obtained geometrical conversion information <NUM> to generate an aligned floating image <NUM> (S308). By performing the above processings, the processings of the aligner <NUM> are completed.

As explained above, in this embodiment <NUM>, the rigid body aligner <NUM> carries out the image-based rigid body alignment of the reference image <NUM> (ultrasonic volume data) and the floating image <NUM> (CT volume data). The rigid body aligner <NUM> extracts sampling points from the reference image <NUM> by using the characteristic data <NUM> of the reference image. The rigid body aligner <NUM> calculates coordinates corresponding to the sampling points of the extracted reference image <NUM> in the floating image <NUM> by using the result of the alignment of the ultrasonic blood vessel data and the CT blood vessel data as initial value for the geometrical conversion. The rigid body aligner <NUM> calculates image similarity by using the sampling points of the reference image <NUM> and the corresponding sampling points of the floating image <NUM>. Updating calculation of the geometrical conversion information of the floating image <NUM> is carried out so that the calculated image similarity should be maximized. As a result, an appropriate initial value for the geometrical conversion can be used, the image similarity can be calculated with good accuracy for an objective internal organ, and therefore stable and highly precise alignment processing can be realized.

As described above, alignment accuracy can be improved by further performing rigid body alignment for the ultrasonic volume data and CT volume data aligned according to the embodiment <NUM> or <NUM>. Therefore, if a CT image is obtained by performing the process of the flowchart shown in <FIG> using the ultrasonic volume data and the CT volume data aligned according to this embodiment <NUM>, real time ultrasonograph and CT image can be further highly precisely coincided to each other. Therefore, highly precise matching is possible between the both images, and small tumors and so forth can be confirmed with higher precision.

In the embodiment <NUM>, alignment accuracy is improved by further performing non-rigid body alignment for the ultrasonic volume data aligned according to the embodiment <NUM> or <NUM>, and the aligned CT volume data. That is, the image processing device <NUM> comprises, in addition to the functional blocks shown in <FIG>, an image-based non-rigid body aligner <NUM> shown in <FIG> in the inside. The image-based non-rigid body aligner <NUM> is a device for performing alignment of the aligned CT volume data obtained in the step S210 of the embodiment <NUM> shown in <FIG> or the step S210 of the embodiment <NUM> shown in <FIG> as a floating image <NUM>, and ultrasonic volume data as a reference image <NUM>.

The image-based non-rigid body aligner <NUM> transforms the floating image <NUM> by using the aligned CT volume data as the floating image <NUM>, and the ultrasonic volume data as the reference image <NUM>. In order to transform the floating image <NUM>, a control grid is placed on the floating image <NUM>, and by moving control points in this control grid, the floating image is transformed. Image similarity is obtained between the transformed floating image and the reference image, and optimization calculation is performed on the basis of the obtained image similarity to obtain moving magnitudes of the control points in the control grid (transformation magnitude). In this calculation, the moving magnitude of a pixel between the control points in the control grid is calculated by interpolation of moving magnitudes of control points provided around that pixel. By using the obtained moving magnitude of each pixel, coordinate conversion of the floating image is performed, and such alignment that the image is locally changed is carried out. Deformation of internal organ and so forth can be thereby corrected, and accuracy and robustness of the alignment can be further improved.

Before the transformation of the floating image <NUM>, the control points in the control grid are geometrically converted as an initial value for the non-rigid body alignment in order to arrange the control points at more exact positions. For this geometrical conversion of the control points, the result of the alignment of the point group of the ultrasonic blood vessel data and the point group of CT blood vessel data of the embodiment <NUM> or <NUM> may be used. Alternatively, the result of the rigid body alignment of the ultrasonic volume data and CT volume data of the embodiment <NUM> may also be used.

The configurations of the alignment of the point group of ultrasonic blood vessel data and the point group of CT blood vessel data, and the rigid body alignment of ultrasonic volume data and CT volume data are the same as those of Examples <NUM> or <NUM>, and therefore differences are mainly explained below.

<FIG> shows a functional block diagram of the image-based non-rigid body aligner <NUM> of the ultrasonic imaging apparatus according to this embodiment. The image-based non-rigid body aligner <NUM> is a device for transforming the floating image <NUM> according to the reference image <NUM> to generate an aligned floating image <NUM>, and comprises a characteristic region sampler <NUM>, a control point geometrical converter <NUM>, an aligner <NUM>, and a floating image transformer <NUM>.

The reference image <NUM>, the floating image <NUM>, and characteristic data <NUM> of the reference image are the same as the reference image <NUM>, the floating image <NUM>, and the characteristic data <NUM> of the reference image of the embodiment <NUM>, respectively. The characteristic region sampler <NUM> receives the reference image <NUM> and the characteristic data <NUM> of the reference image, performs the same processings as those performed by the characteristic region sampler <NUM> of the embodiment <NUM>, and outputs the obtained sampling points of the reference image <NUM> to the aligner <NUM>.

The geometrical conversion information <NUM> is information outputted to the aligner <NUM> as initial value for the non-rigid body alignment. As the geometrical conversion information <NUM>, the result of the alignment of the ultrasonic blood vessel data and the CT blood vessel data may be used, or the result of the rigid body alignment of the ultrasonic volume data and the CT volume data may be used.

The aligner <NUM> comprises a control point geometrical converter <NUM>, a coordinate geometrical converter <NUM>, an image similarity calculator <NUM>, and an image similarity maximization part <NUM>.

<FIG> is a flowchart for explaining the whole operation of the aligner <NUM>. Each of the steps mentioned in <FIG> will be explained below.

The control point geometrical converter <NUM> receives the geometrical conversion information <NUM> (S401), carries out geometrical conversion of positions of control points, and outputs control point moving magnitude information <NUM> to the coordinate geometrical converter <NUM> (S402).

The coordinate geometrical converter <NUM> obtains the sampling data of the reference image <NUM> and the floating image <NUM> (S403 and S404). The coordinate geometrical converter <NUM> further arranges a control grid on the obtained floating image <NUM>, obtains control point moving magnitude information <NUM> from the control point geometrical converter <NUM>, and sets positions of the control points in the aforementioned control grid on the basis of the control point moving magnitude information <NUM>. The coordinate geometrical converter <NUM> also carries out coordinate conversion of the coordinates of the sampling points of the reference image <NUM> by using the control point moving magnitude information <NUM> (S405). This step is for calculating coordinates of the image data of the floating image <NUM> corresponding to the coordinates of the sampling points of the reference image <NUM>. In this example, by performing interpolation of coordinates for coordinate of a certain sampling point on the basis of positions of control points around the sampling point using, for example, the known B-spline function, coordinates of corresponding sampling points in the floating image <NUM> are calculated.

Then, the coordinate geometrical converter <NUM> calculates luminosity value of a sampling point corresponding to each corresponding sampling point of the floating image <NUM> (sampling point corresponding to each sampling point of the reference image <NUM>) by, for example, linear interpolation calculation (S406). Coordinate (sampling point) of the floating image changed by the movement of the control point and luminosity value at the coordinate (sampling point) are thereby obtained. That is, transformation of the floating image accompanying the movement of the control point is performed in this converter <NUM>.

The image similarity calculator <NUM> obtains luminosity data of the sampling points of the reference image <NUM>, and luminosity data of corresponding sampling points of the geometrically converted floating image <NUM> (data generated in S405). The image similarity calculator <NUM> applies a predetermined evaluation function to the data at these sampling points to calculate the image similarity between the reference image <NUM> and the floating image <NUM> (S407). As the image similarity, known mutual information can be used as in the case of the rigid body alignment.

The image similarity maximization part <NUM> obtains the image similarity between the reference image <NUM> and the floating image <NUM> calculated by the image similarity calculator <NUM>. In this example, convergence calculation is carried out in order to calculate moving magnitude of each control point that provides the maximum (or local maximum) of the image similarity between the reference image <NUM> and the floating image <NUM> (S408). When the image similarity has not converged in the step S408, in order to obtain higher image similarity, the image similarity maximization part <NUM> updates the control point moving magnitude information <NUM> (S409). Then, the steps S405 to S409 are carried out again by using the updated control point moving magnitude information <NUM>.

On the other hand, when the image similarity has converged in the step S408, the aligner <NUM> outputs the obtained control point moving magnitude information <NUM> to the floating image transformer <NUM> (S410). By performing the above processings, the processings of the aligner <NUM> are completed.

The floating image transformer <NUM> obtains the floating image <NUM> and the control point moving magnitude information <NUM>. The floating image transformer <NUM> calculates coordinates of all the pixels of the floating image <NUM> by the same interpolation calculation as that of the step S204 on the basis of the control point moving magnitude information <NUM>. Then, the floating image transformer <NUM> calculates luminosity at the obtained coordinates by the same interpolation calculation as that of the step S406 to generate an aligned floating image <NUM>.

According to the embodiment <NUM>, by using the result of the alignment of the ultrasonic blood vessel data and the CT blood vessel data or the result of the rigid body alignment of the ultrasonic volume data and the CT volume data, initial value (position) of a control point used at the time of the alignment between the reference image and the floating image is set. This enables setting of more appropriate initial value of the control grid, and it makes possible to improve the accuracy of the alignment. It is also possible to shorten the time required for the alignment.

Claim 1:
An ultrasonic imaging apparatus comprising:
an ultrasound probe (<NUM>) that transmits a ultrasonic wave to a subject (<NUM>) and receives a ultrasonic wave from the subject (<NUM>),
a position sensor (<NUM>) attached to the ultrasound probe (<NUM>),
an image generator (<NUM>) that generates an ultrasonograph from the signal received by the ultrasound probe (<NUM>), and generates first volume data from the ultrasonograph and positional information of the ultrasound probe (<NUM>) obtained from the position sensor (<NUM>), and
the image processor (<NUM>) according to claim <NUM>, that receives and processes second volume data obtained for the subject (<NUM>) by another external imaging apparatus (<NUM>).