Patent Description:
Conventionally, CT (Computed Tomography) and MRI (Magnetic Resonanse Imaging) inspections have been largely performed for the diagnosis of diseases. Since there are many medically useful information in the images obtained by CT and MRI inspections, sharing such information among many medical institutions and medical providers is highly useful for medical research and education.

However, CT images and MRI images are managed according to a standard called DICOM (Digital Imaging and COmmunication in Medicine), and there is a problem that data cannot be easily shared from the viewpoint of privacy protection because personal information of the patient is recorded there. Moreover, in the case of sharing data in general, tagging is performed in order to improve the usability but tagging the information recording the personal information needs to be performed more carefully from the viewpoint of privacy protection because tagging makes access to data easier. Therefore, until now, sharing information of patient's CT images and MRI images has not been conducted across hospital boundaries.

On the other hand, in recent years, advances in technologies related to VR (Virtual Reality) and the like have been remarkable, and in addition to VR, there are concepts such as AR (Augmented Reality) and MR (Mixed Reality).

In that trend, a virtual image restoration system for a Buddha image is known, which is capable of restoring a wooden art object such as a Buddha image using the virtual reality. (refer to Patent literature <NUM>) In this method, a tomographic image is acquired and corrected using a CT scanner, and a plurality of <NUM>-dimensional tomographic images is stacked to construct a <NUM>-dimensional stereoscopic image. Then, the amount of data is reduced by smoothing or a similar technology in an occasion to be converted into a virtual reality image.

However, the virtual image restoration system of Buddha image disclosed in the above-mentioned Patent Document <NUM> constructs a <NUM>-dimensional image by laminating <NUM>-dimensional tomographic images and performs smoothing etc., but it does not employ a contrivance such as, for example, to perform sorting for easy tagging, which is a problem in terms of the availability and convenience of the data.

In addition, a system that visually supports surgery by synthetically displaying an image of a portion that is intrinsically invisible by using a <NUM>-dimensional digital magnifying glass at the time of surgery is known. (refer to Patent literature <NUM>).

This is for producing a surface polygon model based on <NUM>-dimensional medical image data. However, the <NUM>-dimensional digital magnifying glass system disclosed in Patent Document <NUM> has no problem as long as the obtained <NUM>-dimensional data is used for the patient, but in order to share data with a large number of users, a contrivance to improve the usability of the 3D image is necessary but this kind of contrivance has not been seen, which is a problem.

Under the present circumstances, when <NUM>-dimensional medical image data is used as a <NUM>-dimensional image data and a virtual reality space is created, there is no contrivance such as to obtain segmented data in order to enhance the usability of the data.

In view of such a situation, it is an object of the present invention to provide a medical information virtual reality server system which facilitates communication between users using three-dimensional data employed in a virtual reality space, facilitates communication relating to data between joint owners thereof, is also suitable for facilitating protection of privacy and handling of data relating to persons whose data has been acquired, and also improves the usability of the data.

The following references to embodiments should be understood as such only to the extent that they imply all the steps and/or features of the independent claims.

According to the medical information virtual reality server system of the present invention, non-verbal knowledge possessed by a skilled healthcare provider can be easily transmitted to other healthcare providers or subjects, etc. during learning, training or simulation, with an effect of facilitating communication. Further, since the amount of the <NUM>-dimensional polygon data is light, the calculation processing cost can be suppressed, and the service can be provided at a lower cost. Moreover, since the subject's personal information is not stored in the database, privacy can also be protected.

Embodiments of the present invention will be described in detail below with reference to the drawings. The present invention is not limited to the following embodiment and examples of shown in the figure, and the present invention can be variously changed in design.

<FIG> and <FIG> show functional block diagrams of the medical information virtual reality server system of the Embodiment <NUM>.

As shown in <FIG>, the medical information virtual reality server system <NUM> of the present embodiment is provided with an input means <NUM>, a sorting means <NUM>, a medical image data conversion means <NUM>, a tagging means <NUM>, a <NUM>-dimensional polygon data management means <NUM> and an output means <NUM>.

The medical information virtual reality server system <NUM> does not have to be configured by one computer, and may be configured with PCs and servers connected via a wire or wirelessly. For example, as shown in <FIG>, the system may be configured with the PC 3d and the server <NUM>. Here, the PC 3d and the server <NUM> are provided with an input unit <NUM> and an output unit <NUM>, respectively, and are configured to be communicable with other devices. In addition, a plurality of PCs and servers may be provided.

Further, although not shown here, the medical information virtual reality server system <NUM> is provided with a coordinate associating means, and the user can correlate the <NUM>-dimensional polygon data with an arbitrary coordinate position in an occasion to experience the virtual reality space using the <NUM>-dimensional polygon data.

Note that the input means and the output means in the PC and the server are the same as in the following embodiments, and the description thereof is omitted in the second and the subsequent embodiments.

<FIG> shows a system configuration diagram of the medical information virtual reality server system of the Embodiment <NUM>. As shown in <FIG>, the medical information virtual reality server system <NUM> includes a PC 3d and a server <NUM>, and the PC 3d and the server <NUM> are connected to the network <NUM>. Medical image data is input from the medical imaging apparatus <NUM> to the PC 3d, and as shown in <FIG>, the <NUM>-dimensional polygon data is divided by the dividing means <NUM> into segments for each living body region, medical instrument or medical device. Medical image data is converted into <NUM>-dimensional polygon data by the medical image data conversion means <NUM> provided in the PC 3d. The tagging unit <NUM> tags <NUM>-dimensional polygon data or segments. The tagged <NUM>-dimensional polygon data is sent to the server <NUM> via the network <NUM> as shown in <FIG>. In the server <NUM>, the <NUM>-dimensional polygon data management means <NUM> shown in <FIG> stores the <NUM>-dimensional polygon data in the database and outputs the data based on the tag.

As shown in <FIG>, the <NUM>-dimensional polygon data is output according to the user's request. Here, a medical institution 8a, an educational institution 8b, and a health institution 8c are illustrated as users.

Note that, a medical institution, an educational institution, or a health institution is shown as an example here, but the user to be shared is not limited to this, and other users such as for-profit companies and individuals are also targeted. Further, since the network <NUM> shares the system, a plurality of medical institutions, research and educational institutions, etc. can be added.

<FIG> shows a functional block diagram of the medical information virtual reality system of the Embodiment <NUM>.

As shown in <FIG>, the medical information virtual reality system 1a according to this embodiment includes a medical imaging apparatus <NUM>, a PC 3d and a server <NUM>.

The medical image capturing apparatus <NUM> is furnished with a medical image data acquisition means <NUM>. The PC 3d is furnished with a personal information deletion means <NUM>, a classification means <NUM>, a medical image data conversion means <NUM> and a tagging means <NUM>. Further, the server <NUM> is furnished with a <NUM>-dimensional polygon data management means <NUM>.

<FIG> shows a system configuration diagram of a medical information virtual reality system of the Embodiment <NUM>.

As shown in <FIG>, the medical information virtual reality system 1a comprises a PC (3a to 3d), a server <NUM>, a network <NUM>, a head mounted display <NUM>, and a sensor <NUM>.

In the medical image capturing apparatus <NUM>, as shown in <FIG>, medical image data of a subject to be a target of medical image acquisition is acquired using the medical image data acquisition means <NUM>. Personal information is deleted from the acquired medical image data by a personal information deleting means <NUM> furnished in the PC 3d. Further, the dividing means <NUM> divides the <NUM>-dimensional polygon data into segments for each living body part, a medical device or a medical apparatus. Medical image data is converted into <NUM>-dimensional polygon data by medical image data conversion means <NUM> furnished in the PC 3d. The tagging unit <NUM> tags <NUM>-dimensional polygon data or <NUM>-dimensional polygon segments. The tagged <NUM>-dimensional polygon data is sent to the server <NUM> via the network <NUM> as shown in <FIG>. In the server <NUM>, the <NUM>-dimensional polygon data management means <NUM> shown in <FIG> stores the <NUM>-dimensional polygon data in the database and the data is output based on the tag.

The medical institution 8a is furnished with a PC 3a, the educational institution 8b is furnished with a PC 3b, and the health institution 8c is furnished with a PC 3c. Although the subject did not receive imaging of the medical image here at the medical institution 8a, the educational institution 8b or the health institution 8c, the data is stored in the server <NUM> in a state where the subject's personal information is deleted, making possible to search and use said data by the tag in any one of the medical institution 8a, the educational institution 8b, and the health institution 8c. Furthermore, it may be configured in such a way wherein medical image data, for example, may be acquired by a medical imaging apparatus furnished in a medical institution 8a and personal information may be deleted using a personal information deletion means provided in advance inside of the PC 3a.

When the data stored in the server <NUM> is output by the tag in the medical institution 8a, the user can experience the virtual reality space <NUM> by using the head mounted display <NUM> and the sensor <NUM> connected with PC 3a via a wire or wirelessly. The data experiencing the virtual reality space <NUM> is tagged automatically by PC 3a or manually by PC3a or by attached controller (not shown) connected with PC3a via a wire or wirelessly, and is sent to the server <NUM> and stored via a network <NUM>. By a repetition of the system usage in such a way, the data utilization by users is accumulated and accordingly a database with higher usability can be constructed.

<FIG> shows the functional block diagram of the medical information virtual reality system of the Embodiment <NUM>. As shown in <FIG>, the medical information virtual reality system <NUM> b of the present embodiment is composed of a PC (3d, 3e) and a server <NUM>.

The medical image pickup device <NUM> is furnished with a medical image data acquisition means <NUM>. The personal information deletion means <NUM> is provided in the PC 3d. Further, the PC 3e is furnished with a classification unit <NUM>, a medical image data conversion unit <NUM>, and a tagging unit <NUM>. The server <NUM> is furnished with a <NUM>-dimensional polygon data management means <NUM>.

<FIG> shows a system configuration diagram of the medical information virtual reality system of the Embodiment <NUM>.

As shown in <FIG>, the medical information virtual reality system 1b is comprised of a medical imaging device <NUM>, PC's (3a to 3e), a server <NUM>, a network <NUM>, a head mounted display <NUM> and a sensor <NUM>.

In the medical imaging apparatus <NUM>, as shown in <FIG>, the medical image data of a subject to be a target of the medical image acquisition is acquired using the medical image data acquisition means <NUM>. The acquired medical image data has its personal information deleted by the personal information deleting means <NUM> provided in the PC 3d, and is sent to the PC 3e via the network <NUM> as shown in <FIG>. In the PC 3e, the <NUM>-dimensional polygon data is divided into segments for each living body region by a dividing means <NUM> shown in <FIG>. A medical image data conversion means <NUM> converts medical image data into <NUM>-dimensional polygon data. The tagging unit <NUM> tags the <NUM>-dimensional polygon data or segments. The <NUM>-dimensional polygon data is stored in a database provided in the server <NUM> by the <NUM>-dimensional polygon data managing means <NUM> and is made to be able to be output by a tag.

A means for deleting personal information from medical image data is provided in PC 3d connected to the medical image pickup apparatus <NUM> via a wire or wirelessly, and personal information is cut off in the PC 3d. In this state, it is connected to the Internet <NUM>, and privacy protection is achieved. The means for converting medical image data into <NUM>-dimensional polygon data is not necessarily provided in the PC 3d connected to the medical image pickup device <NUM> via a wire or wirelessly, but unlike this, it may be provided in the PC 3e or the server <NUM> via the Internet <NUM>.

The medical institutions 8a, the educational institutions 8b, and the health institutions 8c are each provided with a PC (3a to 3c), and can experience the virtual reality space as in the Embodiment <NUM>.

<FIG> is an image view of the <NUM>-dimensional polygon data divided into segments for the upper limbs of a human body, (<NUM>) showing a front view, and (<NUM>) showing a perspective enlarged view. As shown in <FIG> (<NUM>), in the image, the liver <NUM>, portal vein <NUM>, pancreas <NUM>, kidney <NUM>, aorta <NUM>, heart <NUM>, ribs <NUM>, spleen <NUM>, spine <NUM> and pelvis <NUM> are segmented and displayed. Also, each part is divided into segments, and the names of the parts are tagged. Therefore, as shown in <FIG> (<NUM>), the name of each part is displayed by a text, and it is displayed by the arrow so that it can be clearly recognized which segment is which part. As described above, the biological part is color-coded and displayed, and the name of each site is displayed as a text, thereby improving the visibility and the usability of data.

<FIG> (<NUM>) shows a state in which the image shown in <FIG> (<NUM>) is enlarged and viewed from slightly below. As shown in <FIG> (<NUM>), the pancreas <NUM>, the kidney <NUM>, the spleen <NUM> and the gallbladder <NUM> are segmented and displayed. Since the segment of each part is stored and managed as <NUM>-dimensional polygon data, the name of each part is displayed to indicate the respective part even if the viewing angle is changed, as shown in <FIG> (<NUM>). Further, in <FIG> (<NUM>), the gallbladder <NUM> which is not displayed in <FIG> (<NUM>) is displayed. In this way, depending on the display angle or the like, it is possible to display an image that is easy for the user to view and understand, such as displaying only the name of the part that is considered to be necessary, or displaying all of them.

A method of such segmentation is described by referring to <FIG> shows a flow diagram of medical image data segmentation.

As shown in <FIG>, firstly, the <NUM>-dimensional medical image data is automatically divided (S21). The <NUM>-dimensional medical image data is converted to <NUM>-dimensional polygon data (S22). The <NUM>-dimensional polygon data is automatically divided (S23). Both the automatic segmentation based on the <NUM>-dimensional medical image data and the automatic segmentation based on the <NUM>-dimensional polygon data may be performed as shown in this embodiment, or only one of them may be performed.

The <NUM>-dimensional medical image data and the <NUM>-dimensional polygon data are compared and evaluated (S24). If there is no error in the automatic classification (S25), the classification result is stored in the database (S26). If there is an error in the automatic classification, the classification is manually corrected (S27). For example, in a case where a specific organ or blood vessel is the problem based on the information of the medical provider under the circumstance that <NUM>-dimensional polygon data is produced, the portion where applicable in the <NUM>-dimensional polygon data is marked. If it is not clear which organ, or tumor, etc. is the problem, from the knowledge of the medical provider, the relevant part in the <NUM>-dimensional polygon data is marked by comparing with the <NUM>-dimensional medical image data of the part corresponding to the relevant part.

As a specific decision factor of the classification, the gradation difference of the image or the <NUM>-dimensional position coordinate information becomes an important decision factor.

In addition, evaluation/modification by a health care provider is not essential, but it is also possible to make a system that does not require manual correction by accumulation of data.

<FIG> shows a flow diagram of the use and accumulation of medical image data. As shown in <FIG>, in the medical information virtual reality system of the present embodiment, classification is automatically performed using a database (S31). Next, evaluation and correction are performed by the health care provider (S32). The result of the classification is stored in the database (S33). The steps mentioned above are repeatedly performed.

Here, the results of classification to be stored include not only the final result that has been evaluated and corrected by the health care provider but also the difference between the results of automatic classification and the results of manual classification, etc., and the accumulation of such data enables automatic segmentation with higher accuracy.

By performing the segmentation as described above, tagging becomes easy. As a method of tagging, it can be performed for the whole living body from which data is to be acquired, or for each segment such as an organ or a blood vessel. For example, when tagging is performed by age, tagging is performed for the whole living body. In addition, in the case of tagging a site with a tumor, tagging is performed with respect to an organ having a tumor or a tumor site segmented into an organ having a tumor.

In addition, it is possible to tag specific <NUM>-dimensional coordinates regardless of each segment, or to tag by setting a spatial range.

As types of tagging, marking, numerical input, comments in text, etc. are possible. For example, although the age of the living body and so on is numerically input, when a qualitative comment is recorded, a comment can be made by text. In addition, markers can be used to mark the <NUM>-dimensional space.

As shown in <FIG>, the medical information virtual reality system <NUM> includes PC's (3a, 3d), smart phones (30a to 30c), a server <NUM>, a network <NUM>, a head mounted display <NUM>, and a sensor <NUM>.

The medical institution 8a is furnished with a PC 3a, smart phones (30a to 30c), a head mounted display <NUM>, and a sensor <NUM>. The smartphones (30a to 30c) are connected to the PC 3a via the network <NUM>. The PC 3a is connected to the head mounted display <NUM> and the sensor <NUM> via a wire or wirelessly. Although the head mounted display <NUM> and the sensor <NUM> are not connected to the smartphones (30a to 30c), a smartphone VR application (not shown) for using the medical information virtual real i-ty system <NUM> is installed, realizing a structure that allows simultaneous viewing of the virtual reality space <NUM> expressed by using PC 3a, the head mounted display <NUM> and a sensor <NUM>.

As a specific structure, a virtual reality application (not shown) installed on PC 3a and a virtual reality application (not shown) installed on smartphones (30a to 30c) are connected by a network <NUM>. In the present embodiment, the network <NUM> is shown to be communicating wirelessly, but this may be a wired communication. The position information of the head mounted display <NUM> acquired using the sensor <NUM> in the virtual reality application of the PC 3a and the information of the controller (not shown) are transmitted to the smartphones (30a to 30c), and in the smartphones (30a to 30c), the virtual reality space <NUM> can be viewed at the same position in the virtual reality space as the position wherein user views the virtual reality space <NUM> in the head mounted display <NUM>. Also, the virtual reality space <NUM> can be viewed by changing the angle based on the operation of the screen of the smartphones (30a to 30c). The position information of the controller is sensed not only by the position of the controller but also by the orientation of the controller by the sensor <NUM>, and accordingly, the motion of the hand of the user using the virtual reality application in the PC 3a can be viewed on the virtual reality space which a smartphone (30a to 30c) displays. Namely, the user using the head mounted display <NUM> or the controller and the user using the smartphones (30a to 30c) can recognize the virtual reality space that matches the respective position information upon sharing the same virtual reality space.

As described here, by using the smartphone, the virtual reality system can be used easily and inexpensively, even when there are many participants.

<FIG> shows a data flow diagram of the medical information virtual reality system of the Embodiment <NUM>. As shown in <FIG>, first, search information is input in the client terminal (S01). Data on search information is transmitted to the server (S02). The server receives the data (S03). The server performs a search and extracts a search result (S04). The search result is transmitted from the server to the client terminal (S05). The client terminal receives the search result (S06). The client terminal creates a virtual reality space based on the search result (S07). Data constituting the created virtual reality space is transmitted from the client terminal to the mobile terminal (S08). The mobile terminal receives the data constituting the virtual reality space (S09). The virtual reality space is displayed on the client terminal (S10). In the mobile terminal, a virtual reality space is displayed (S11).

<FIG> show a production flow diagram of medical information virtual reality data of the Embodiment <NUM>. First, as shown in <FIG>, when producing medical information virtual reality data, medical image data is input to the medical information virtual reality server system <NUM> (S41). Next, each feature site is classified and divided into segments by using at least any one of <NUM>-dimensional medical image data of CT image data, MRI image data or ultrasonic image data or <NUM>-dimensional position measurement data of a living body part, implant, medical device, index marker or a feature part of insertion object (S42). Medical image data is converted into <NUM>-dimensional polygon data having segments (S43). And the image data is tagged to the <NUM>-dimensional polygon data or segment (S44). The <NUM>-dimensional polygon data is stored in the database and output based on the tagged tag (S45).

As a configuration different from the above, a configuration may be considered in which personal information present in medical image data is deleted. For example, as shown in <FIG>, after medical image data has been input (S51), personal information is deleted (S52), and then divided into segments for each characteristic portion using any of the medical image data described above (S53). Here, deletion of personal information does not have to be after input of medical image data, and a method of inputting medical image data wherefrom personal information has already been deleted may be employed.

Note that the flow after division into segments (S53) is the same as that in <FIG>.

Furthermore, as a configuration different from the above, a procedure is considered, wherein the data when the user experiences virtual reality space is recorded and transmitted to the medical information virtual reality server system <NUM> by utilizing the <NUM>-dimensional polygon data output from the medical information virtual reality server system <NUM>, which is made to be a database of higher According to the invention, as shown in <FIG>, after the data output from the medical information virtual reality server system <NUM> (S65), it is possible to create medical information virtual reality data having higher usability by storing data in which the user's actions and the like in the virtual reality space are recorded in association with time series data of <NUM>-dimensional polygon data (S66).

Note that the flow before data output (S65) from the medical information virtual reality server system <NUM> is the same as that in <FIG>.

Claim 1:
A medical information virtual reality server, being connected a client terminal for displaying a virtual reality space interactively to a user via a network to transmit and receive data, comprising:
data input means for inputting medical image data; and
segmentation means for segmenting medical image data into segments for each characteristic site including a living body site, an implant, a medical device, an index marker or a mounting object; and
<NUM>-dimensional data conversion means for converting medical image data into <NUM>-dimensional polygon data having segments; and
tagging means for tagging <NUM>-dimensional polygon data or segments; and
coordinate relating means for relating <NUM>-dimensional polygon data to an arbitrary world coordinate position; and
data output means for storing <NUM>-dimensional polygon data in a database and outputting <NUM>-dimensional polygon data based on the tagged tag; and
wherein the server receives the tag information input from client terminal, and
extracts <NUM>-dimensional polygon data based on the received tag information and transmits them to the client terminal, and
wherein the server is configured to receive recorded data of the user's motion in the virtual reality space based on the <NUM>-dimensional polygon data and to change the virtual reality space based on the motion from the client terminal, and is configured to store the recorded data in association with time series data of <NUM>-dimensional polygon data in the database.