Surgical operation support system, surgical operation support apparatus, surgical operation support method, surgical operation support program, and information processing apparatus

A system of this invention is directed to a surgical operation support system that supports determination of an appropriate disposition of a bone in a living body during surgery. The surgical operation support system includes a storage that stores 3D data of a first target bone that is one of two divided surgery target bones and 3D data of a reference bone partially overlapping the first target bone in association with position data of a first marker fixed to the first target bone, and stores 3D data of a second target bone that is the other of the two divided surgery target bones in association with position data of a second marker fixed to the second target bone, an image capturer that captures the first marker fixed to the first target bone and the second marker fixed to the second target bone, and a display that changes display in accordance with a change in relative positions of the first marker and the second marker using the data stored in the storage such that the target position of the second marker with respect to the first marker when the second target bone overlaps the reference bone can be grasped.

RELATED APPLICATION

This application is an application under 35 U.S.C. 371 of International Application No. PCT/JP2014/065447 filed on Jun. 11, 2014, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a technique of supporting a surgical operation by image processing.

BACKGROUND ART

In the above-described technical field, patent literature 1 discloses a technique of attaching a marker indicating the position of the skull of a patient and capturing the marker, thereby automatically controlling the attachment position of a robot for surgery support. Non-patent literature 1 shows software that generates STL (Stereo Lithography) data of a 3D bone surface model from DICOM (Digital Imaging and Communication in Medicine) data that is a standard format of a medical image of CT/MRI or the like. Non-patent literature 2 shows software that simulates bone and joint surgery in advance using 3D bone surface model (STL) data.

CITATION LIST

Patent Literature

Patent literature 1: Japanese PCT National Publication No. 2008-526422

SUMMARY OF THE INVENTION

Technical Problem

However, the technique described in patent literature 1 controls the attachment position of a robot to a skull but cannot support determination of an appropriate disposition of a bone in a living body. In addition, the techniques of non-patent literatures 1 and 2 are used to simulate bone and joint surgery in advance, and are not intended to support determination of an appropriate disposition of a bone in a living body during actual surgery.

The present invention enables to provide a technique of solving the above-described problems.

Solution to Problem

One aspect of the present invention provides a surgical operation support system comprising:

a storage that stores 3D data of a target bone that undergoes a surgical operation and position data of a marker in association with each other;

an image capturer that captures the marker of the target bone; and

a display that changes display of the target bone in accordance with a change in a position of the captured marker using the data stored in the storage.

Another aspect of the present invention provides a surgical operation support apparatus comprising:

a storage that stores 3D data of a target bone that undergoes a surgical operation and position data of a marker in association with each other;

an image capturer that captures the marker of the target bone; and

a display that changes display of the target bone in accordance with a change in a position of the captured marker using the data stored in the storage.

Still other aspect of the present invention provides a surgical operation support method comprising:

storing 3D data of a target bone that undergoes a surgical operation and position data of a marker in a storage in association with each other;

capturing the marker of the target bone; and

changing display of the target bone in accordance with a change in a position of the captured marker using the data stored in the storage.

Yet another aspect of the present invention provides a surgical operation support program that causes a computer to execute a method comprising:

storing 3D data of a target bone that undergoes a surgical operation and position data of a marker in a storage in association with each other;

capturing the marker of the target bone; and

changing display of the target bone in accordance with a change in a position of the captured marker using the data stored in the storage.

Still yet another aspect of the present invention provides an information processing apparatus comprising:

a storage that stores 3D data of a target bone that undergoes a surgical operation and position data of a marker in association with each other;

a receiver that receives, from an image capturer, an image of the marker of the target bone captured by the image capturer; and

a display image generator that generates a display image of the target bone, which changes in accordance with a change in a position of the captured marker, using the data stored in the storage.

Advantageous Effects of Invention

According to the present invention, it is possible to support determination of an appropriate disposition of a bone in a living body during surgery.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

A surgical operation support system100according to the first embodiment of the present invention will be described with reference toFIG. 1. The surgical operation support system100is a system that supports a surgical operation by image processing.

As shown inFIG. 1, the surgical operation support system100includes a storage110, an image capturer120, and a display150.

The storage110stores 3D data of a first target bone111that is one of two divided parts of a surgery target bone101and 3D data of a reference bone113partially overlapping the first target bone111in association with the position data of a first marker102fixed to the first target bone111. The storage110also stores 3D data of a second target bone112that is the other of the two divided parts of the surgery target bone101in association with the position data of a second marker103fixed to the second target bone112.

The image capturer120captures the first marker102fixed to the first target bone111and the second marker103fixed to the second target bone112. The display150changes display in accordance with a change in the relative positions of the first marker102and the second marker103using the data stored in the storage110such that the target position of the second marker103with respect to the first marker102when the second target bone112overlaps the reference bone113can be grasped.

According to this embodiment, it is possible to support determination of an appropriate disposition of a bone in a living body during surgery.

Second Embodiment

A surgical operation support system according to the second embodiment of the present invention will be described next. The surgical operation support system according to this embodiment generates 3D data of a first target bone that is a part of a surgery target bone serving as the reference of the disposition of the surgery target bone and 3D data of a reference bone in advance, and stores the data in association with a first marker (for example, a 2D code) fixed to the first target bone. The surgical operation support system also generates 3D data of a second target bone that is the other part of the surgery target bone, and stores the data in association with a second marker (for example, a 2D code) fixed to the second target bone. In surgery, the 3D positions of the first target bone and the second target bone are determined from the captured first and second markers using the AR (Augmented reality) technology and displayed based on the stored 3D data. It is determined whether the second target bone and the reference bone adequately overlap, thereby determining an appropriate disposition of the surgery target bone. This processing supports determining an appropriate disposition of the surgery target bone by a doctor.

Surgical Operation Support System

The outline of the arrangement and processing of the surgical operation support system according to this embodiment will be described below with reference toFIGS. 2 to 7. The surgical operation support system is roughly divided into a preoperative preparation data generation system and an intraoperative image processing system. The preoperative preparation data generation system is a system that generates and displays 3D data of a first target bone, a second target bone, and a reference bone before surgery, and generates and stores data to be used during surgery. The intraoperative image processing system is a system that generates and displays a target bone image and a reference bone image based on marker image capturing, and supports determination of the disposition of the surgery target bone. However, the preoperative preparation data generation system and the intraoperative image processing system may be formed as one integrated system.

Outline of Surgical Operation

FIG. 2is a view for explaining the outline of a whole surgical operation according to this embodiment.FIG. 2shows an example of corrective osteotomy of an affected bone (surgery target bone) with malunion. The corrective osteotomy includes a preparation stage201, a surgery target bone alignment stage202, and a surgery target bone position fixing stage203. In this embodiment, malunion surgery of a distal radius will be described as an example. However, the present invention is not limited to this, and is also applicable to malunion of another part or another bone or fracture treatment surgery.

In the preparation stage201, pairs of pins211and212each having a predetermined interval (for example, an interval of 1 cm or 2 cm) are fixed as support members for two markers at two points sandwiching the bone cutting plane of the surgery target bone of a forearm213. Portions having a sufficient strength and sectional area and capable of fixing two pins in the longitudinal direction of the surgery target bone are preferable as positions to insert and fix the pins. A length of about 5 cm to 10 cm suffices as a pin length that enables to set markers outside the forearm and easily capture them, although the length changed depending on the affected portion or bone. CT (Computed Tomography) imaging is performed in a state in which the pins211and212are fixed, thereby generating and storing 3D data of the surgery target bone. In addition, the positions and directions of markers to be fixed to the pins211and212later are set in advance to generate the position data of the markers. The position data of the markers, the 3D data of the surgery target bone, and the 3D data of the reference bone are associated with each other.

For example, the 3D data of pins included in the 3D data of the surgery target bone may be displayed, and the user may be caused to designate the proximal and distal end positions of the two pins using a pointing device or the like to define the position and direction of the marker to be attached to the pins. The relationship between a plane formed by the two pins and the position and direction of the marker may be set in advance or selected from a plurality of relationships (for example, the marker is parallel or perpendicular to the plane formed by the two pins, or makes an angle of 45° with respect to the plane). Alternatively, for example, 3D model data of the pins and 3D model data of one or a plurality of jigs to be used to fix the marker to the pins may be prepared. Then, the 3D model data may be overlaid in a 3D space on the 3D data of the pins acquired by CT imaging, and the jigs may be attached to define the position of the marker. The relationship between the position and direction of the marker and the positions and directions of the surgery target bone and the reference bone is thus stored in a database.

During surgery, the affected part is cut open, and bone cutting is carried out. After that, in the surgery target bone alignment stage202, markers221and222are shot using a digital camera. The positions, sizes, and directions of the markers221and222are recognized from the captured image, and a database is referred to, thereby deriving the positions, sizes, and directions of surgery target bones. Surgery target bones223and224of the derived positions, sizes, and directions are displayed.

When the doctor holds the forearm213of the patient by a hand226and bends or twists the arm, the state of the marker221in the captured image changes. The surgery target bone223in the displayed image is displayed such that its display position, size, and tilt change in accordance with the change in the position, size, and tilt of the marker221. On the other hand, 3D shape data of a reference bone225is stored in advance together with the relative relationship to the position, size, and tilt of the marker222. When the marker222is captured, the reference bone225is displayed at a predetermined position. When the doctor finds a position at which the surgery target bone223overlaps the reference bone225, the process advances to the surgery target bone position fixing stage203.

In the surgery target bone position fixing stage203, to maintain the determined appropriate relative disposition of the surgery target bones223and224in the forearm213of the patient, the pins211and212at the position at which the surgery target bone223overlaps the reference bone225are fixed by a fixing tool231.

With the support by the surgical operation support system, it is possible to make the incision part small and speed up the surgery. Note that inFIG. 2, the pins211and212project outside the wound. However, the present invention is not limited to this. For example, pins that are short (1 to 2 cm) enough to put their distal ends within the wound may be used. During surgery (alignment stage202), long pins may newly be connected to the short pins, and the markers221and222may then be attached.

Alternatively, only a bone may be captured by CT imaging without inserting pins, and virtual pins may be inserted into the thus generated CG data of the bone. After that, a wound may be opened during surgery, and actual pins may be inserted to the position as in the CG data. At this time, the position of a marker may be determined using the CG data of the bone with the virtual pins. A pattern (pattern with pins) that exactly fits on the bone of the affected part may be formed by a 3D printer, and pins may be inserted based on the pattern, thereby inserting actual pins to the same position as in the CG data. The marker may be attached to the pattern itself in a state in which the pattern is exactly fitted on the bone. Feature points of the bone captured by the digital camera may be discriminated and overlaid on the CG data with pins, thereby inserting the pins to the same position in the same direction as in the CG data. This can suppress the burden on the patient and establishment of an infectious disease after CT imaging with the pins being inserted.

Pin Fixing Processing

FIG. 3Ais a view for explaining the outline of insertion processing of marker pins (to be referred to as pins hereinafter) into a bone.FIG. 3Ashows an example in which the pairs of pins211and212are fixed on two points of a surgery target bone311which sandwich an area estimated to include a bone cutting plane.FIG. 3Ashows an alignment stage301of placing a pin fixing tube body312on the skin, a pin fixing stage302of inserting the pins into the pin fixing tube body312and fixing them to the bone, and a tube body removing stage303.

First, in the alignment stage301, the pin fixing tube body312is placed on the two points of the forearm213of the affected part which sandwich the area estimated to include a bone cutting plane. The pin fixing tube body312includes two tubular portions used to insert and fix the pins accurately at a predetermined interval. The pins212are inserted into the pin fixing tube body312.

Next, in the pin fixing stage302, the pins212inserted in the tubular portions are inserted into the forearm213of the affected part and fixed to the surgery target bone311. Threads are cut on the distal ends of the pins212. The pins are rotatably inserted into the bone.

In the tube body removing stage303, only the pin fixing tube body312is removed while leaving the pins212. The alignment stage301, the pin fixing stage302, and the tube body removing stage303as described above are repeated to fix the other pair of pins211. The pins211and212are thus fixed to the surgery target bone311.

Arrangement of Preoperative Preparation Data Generation System

FIG. 3Bis a block diagram showing the arrangement of a preoperative preparation data generation system320.

The preoperative preparation data generation system320includes an information processing apparatus324configured to generate a reference image, and a CT scanner321that acquires a tomographic image of a patient322, which are connected via a network323. The preoperative preparation data generation system320may also include, as an option, an STL data generation server325that generates 3D bone surface data (STL data) from tomographic image data. Note that the network can be either a WAN or a LAN.

In this embodiment, tomographic images of the affected part of the patient322and a part serving as the reference of the affected part are acquired by the CT scanner321. In this example, for example, tomographic images of the right forearm in which four pins are inserted and fixed in the surgery target bone and tomographic images of the left forearm on the unaffected side are acquired. The tomographic image data are sent to the information processing apparatus324via the network323and converted into 3D data by the information processing apparatus324. Note that the conversion from tomographic image data to 3D data may be done by the STL data generation server325.

Note that living body data used in this embodiment is not limited to data acquired by CT/MRI, and 3D data is not limited to STL data.

Preoperative Preparation Data Generation Processing

FIG. 4is a view for explaining the outline of preoperative preparation image generation processing using the information processing apparatus324. Images401to406are CG (Computer Graphics) images displayed on the display screen of the information processing apparatus324, which correspond to the stages of the preoperative preparation data generation processing, respectively.

In the first stage, as indicated by the image401, an unaffected bone at a position (on the unaffected side) bilaterally symmetrical to the surgery target bone of the forearm213is internally captured by CT scan or the like. Thus generated 3D data411of the unaffected bone is inverted to generate mirror image data. Accordingly, 3D data (to be referred to as a reference bone hereinafter)412of a reference bone having the same shape as (at least partially overlapping) the surgery target bone is generated.

In the second stage, as indicated by the image402, the surgery target bone of the forearm213is internally captured by CT scan or the like, and thus generated 3D data (to be referred to as a surgery target bone hereinafter)421of the surgery target bone (affected bone) is displayed. The surgery target bone421is generated from STL data captured in a state in which the pins211and212are fixed, and therefore includes the pins211and212even in the 3D data. The reference bone412and the surgery target bone421are compared on the display screen, and the state of the surgery target bone421is confirmed.

In the third stage, the surgery target bone421is manipulated on the image403while referring to the enlarged display image405in which the observation point in the 3D space is moved close to the surgery target bone or the divided display image406in which a plurality of images from different observation points (in this example, images from three directions) are simultaneously displayed. That is, the surgery target bone421is moved and rotated with respect to the reference bone412to overlay the end portions of the reference bone412on the end portions of the surgery target bone421.

If the bone cutting plane can be estimated to exist on the upper end side, first, the lower ends of the surgery target bone421and the reference bone412are overlaid to determine the bone cutting plane of the surgery target bone421, as shown on the left side. In particular, the shapes of the joint portions are overlaid to recognize the distortion, bending, or deformation of the surgery target bone421. Then, the image is observed gradually from the lower end, and a branch position where deviation from the reference bone starts is determined as a bone cutting plane431. Note that the doctor may determine the bone cutting plane431while observing the overlay state between the reference bone412and the surgery target bone421. However, for example, a position where the non-overlay volume per unit length from the lower end between the surgery target bone421and the reference bone412exceeds a predetermined value may automatically be discriminated as the bone cutting plane431. Alternatively, the surface of the reference bone412may finely be divided into unit areas, and positions at which the vertical distance to the surface of the surgery target bone421in each unit area exceeds a predetermined value may be connected to automatically derive the bone cutting plane431.

The upper ends of the surgery target bone421and the reference bone412are overlaid, and the position of the section in the upper separated bone is confirmed, as shown on the right side of the image403. When the bone cutting plane431is finally determined, the surgery target bone421is divided on the bone cutting plane431, and 3D data of two target bones441and442are generated.

In the fourth stage, the set of the target bone442and the reference bone412which are overlaid is stored in association with the marker222attached to the pins212. As indicated by the image404, the target position of the target bone441with respect to the target bone442or the reference bone412is stored in association with the position data of the marker221attached to the pins211. Accordingly, if the position or tilt of the marker221can be recognized in the real space, the target position or tilt of the target bone441can be estimated. Furthermore, the data of the position, shape, and tilt of the bone cutting plane431are stored in association with the position data of the marker221or222. The position and direction of the marker221with respect to the pins211and the position and direction of the marker with respect to the pins212may be determined to one pattern in advance. In this embodiment, the position and direction can be selected from a plurality of (for example, four) patterns. In a first marker attachment type, a marker is attached to be parallel to the pin plane formed by the two pins. In a second marker attachment type, a marker is attached to a plane that is parallel to the axial direction of the pins and perpendicular to the pin plane. In a third marker attachment type, a marker is attached to a plane that is parallel to the axial direction of the pins and makes an angle of 45° with respect to the pin plane. In a fourth marker attachment type, a marker is attached to a plane that is parallel to the axial direction of the pins and makes an angle of 135° with respect to the pin plane. Alternatively, a marker may be attached to a plane perpendicular to the axial direction of the pins. The relative positional relationship between a marker and a surgery target bone or reference bone to be displayed is changed in accordance with how the marker is attached to the actual pins.

By using the thus prepared data, image display of the target bone441and the reference bone412, image display of the target bone442, and image display of the bone cutting plane431can be performed based on the positions, sizes, and directions of the markers captured in surgery. Note that a gap443between the target bone441and the target bone442represents the shape of a connecting bone necessary in surgery. Hence, the 3D shape of the connecting bone necessary in surgery can also be acquired at this time.

Note that in surgery, the combination of the target bones441and442determined as the target disposition on the image404may integrally be used and displayed without using the reference bone412generated from the unaffected side. In this case, the positions of the pins211and212serving as the support members of the first and second markers221and223in a state in which both of the target bones441and442are overlaid on the reference bone412are stored in the storage as target relative position data. The target positions of the pins212of the second marker222are displayed based on the stored target relative position data. In this embodiment, since corrective osteotomy of an affected bone (surgery target bone) with malunion is carried out, the bone cutting position is determined in this stage. In simple fracture treatment surgery, the bone cutting position need not be determined because a bone is separated into two from the beginning. That is, 3D data generated by CT scan or the like is directly used and overlaid on the reference bone. In the fourth stage, the rotation direction, the rotation angle, and the moving distance in millimeter with respect to the target bone442necessary for the target bone441to overlap the reference bone412may be stored as numerical values. This makes it possible to visually (by an image of an arrow or the like) indicate the rotation direction, the amount of rotation, the moving direction, and the amount of movement necessary for the marker221(that is, the arm) fixed to the actual pins211in surgery.

Arrangement of Intraoperative Image Processing System

FIG. 5Ais a view showing the schematic arrangement of an intraoperative image processing system500according to this embodiment.

The intraoperative image processing system500includes a tablet computer501as an information processing apparatus, and a display device502. The tablet computer501includes a display511and a camera512.

The tablet computer501is fixed at a position at which the display511faces a doctor503, and the camera512faces the markers221and222. The tablet computer501stores the 3D data of the surgery target bone in advance, and recognizes the position and direction of the surgery target bone from the images of the markers221and222. The tablet computer501displays the image of the surgery target bone at the recognized position on the display511. Accordingly, the doctor503can grasp the positional relationship between the affected part and the bone in it at a glance.

When the doctor503holds the forearm213of the patient322and twists or stretches it, the positions of the markers221and222change accordingly. Hence, the surgery target bone421in the display511also moves or rotates. The forearm213is moved in this way to overlay the target bone442in the display511on the reference bone412, thereby determining the target position of the surgery target bone. The pins211and212are fixed at the determined position using the fixing tool231.

Intraoperative Target Bone Alignment Processing

FIG. 5Bis a screen transition diagram for explaining the outline of a bone cutting operation and alignment operation of the surgery target bone during surgery. Before surgery, the markers221and222are fixed to the pins211and212.

In a bone cutting stage, the bone cutting plane431is three-dimensionally displayed on the display511, like an image521, and the surgery target bone is cut at an appropriate position. In the image521, a thick line indicates an image captured by the camera512, and a thin line indicates a CG image generated from 3D data.

The doctor inserts a bone cutting blade into the affected part according to the bone cutting plane431and separate the affected bone with malunion. The doctor then manipulates the target bone441with respect to the target bone442by moving the forearm of the patient while referring to an image522of a coordinate space or divisionally displayed images523to526. In the images521to526, the target bones441and442of positions, sizes, and directions according to the positions, sizes, and directions of the markers221and222obtained by image capturing are displayed.

The image522displays the angles between the observation point and the X-axis, Y-axis, and Z-axis of the 3D space. The relative positions of the reference bone412and the target bones441and442in the 3D space are extracted and displayed. The image of the target bones441and442can be rotated on the screen by moving the observation point. The images523to526are divisionally displayed images displayed on one screen. The image523is the overlay image of the captured image and the CG image, like the image521. The image524corresponds to only the CG image extracted from the image523, and displays the reference bone and the target bone with the pins. The image525is the image of the reference bone412and the target bones441and442viewed from the axial direction of the bones, which makes an angle of 90° with respect to the camera512. The image526is the image of the reference bone412and the target bones441and442viewed from the pin insertion direction which makes an angle of 90° with respect to the camera512. That is, the images524to526are three display images with observation points in the three axial directions of the 3D space. The doctor determines an appropriate disposition of the target bones441and442while observing these display screens.

An image527shows a state in which the target bone441is overlaid on the reference bone412. In this state, the pins211and212attached to the target bones441and442are fixed by the fixing tool.

Processing Procedure of Surgical Operation Support System

FIG. 6is a flowchart showing the processing procedure of the entire surgical operation support system including the preoperative preparation data generation system320and the intraoperative image processing system500.

First, in step S601, the preoperative preparation data generation system320acquires a tomographic image (for example, a CT image) of a surgery target bone to which pins are fixed and a tomographic image of an unaffected bone, and generates 3D data of the bones.

Next, in step S603, while displaying the generated 3D shape data, the bone cutting plane431and an appropriate disposition of the bone after bone cutting are determined, and the position data thereof are stored.

Then, in step S605, the intraoperative image processing system500captures markers fixed to the surgery target bone.

In step S607, the intraoperative image processing system500generates the bone image of the reference bone and the first target bone that changes in accordance with the movement of the marker and the bone image of the second target bone, and displays the bone images overlaid on the captured affected part image. The doctor moves the forearm while viewing the display screen.

In step S609, the intraoperative image processing system500confirms that the two target bones of the forearm are disposed such that the bone image of the second target bone matches the bone image of the reference bone. If the bone images do not match, the intraoperative image processing system500returns to step S605to continue the processing until the target bones are disposed at the matching position.

Surgical Operation Support Instruments

FIG. 7is a view showing surgical operation support instruments used in the surgical operation support system according to this embodiment. Note that instruments normally used in an inspection or surgery are not illustrated inFIG. 7.

The surgical operation support instruments include support instruments710used to fix the pairs of pins at two points of the surgery target bone before CT imaging of the surgery target bone or before the start of surgery after CT imaging of the surgery target bone. The surgical operation support instruments also include support instruments720used to attach a marker to two pins in surgery.

The support instruments710include the pairs of pins211and212, the pin fixing tube body312used to insert the pins211and212exactly at an interval of 1 cm, and a pin fixing tube body712used to insert the pins211and212exactly at an interval of 2 cm.

The support instruments710are used as described with reference toFIG. 3A.

The support instruments720include 2D codes721each printed on paper or a plastic film, marker supports722to725that supports the 2D codes729, pin connectors727and728used to attach the marker supports722to725to the pins, and an alignment jig726formed into an inverted L shape and used to align the connectors727and728.

The marker support722is a support that attaches a marker in parallel to a pin plane formed by two pins. The marker support723is a support that attaches a marker to a plane that is parallel to the axial direction of the pins and perpendicular to the pin plane. The marker support724is a support that attaches a marker to a plane that is parallel to the axial direction of the pins and makes an angle of 45° (45° to the right) with respect to the pin plane. The marker support725is a support that attaches a marker to a plane that is parallel to the axial direction of the pins and makes an angle of 135° (45° to the left) with respect to the pin plane. The marker supports according to the present invention are not limited to these, and a support that attaches a marker to a plane perpendicular to the axial direction of the pins or a support with a hinge capable of attaching a marker in an arbitrary direction with respect to the pins may be prepared. The relative positional relationship between the marker and the surgery target bone or reference bone to be displayed is changed in accordance with the manner the marker is attached to the actual pins.

The alignment jig726includes a convex portion726a, a concave portion726b, and groove portions726c. The convex portion726aand the concave portion726bare formed into sizes that exactly fit in or on concave portions727band728band convex portions727aand728aprovided on the pin connectors727and728.

First, in the first step, the alignment jig726and the pin connector727are combined using the concave portions and the convex portions. In the second step, the pins are inserted along the grooves726cand727c. The pin connector727and the pins are fixed by a screw727dat a position where the pins abut against a ceiling surface726d. In the third step, the alignment jig726is detached from the pin connector727. The first to third steps are repeated in a similar manner, thereby connecting the pin connector728and the pins at an accurate position using the alignment jig726.

The pins211and212are thus accurately connected to the pin connectors727and728, respectively.

Each of the marker supports722to725also includes a convex portion and a concave portion. The convex portions727aand728aand the concave portions727band728bare fitted, thereby fixing one of the marker supports722to725to one of the pin connectors727and728.

Functional Arrangement of Information Processing Apparatus in Preoperative Preparation Data Generation System

FIG. 8is a block diagram showing a functional arrangement example324A of the information processing apparatus324. Note thatFIG. 8shows CT data as tomographic image data, and STL data as 3D bone surface model data. However, the data are not limited to these. Each functional unit of the information processing apparatus324A is implemented when a CPU processes image data by executing a program using a memory.

A CT data acquirer811shown inFIG. 8acquires CT data (DICOM) from the CT scanner321as an image of the patient322. A CT database812searchably accumulates the CT data acquired by the CT data acquirer811.

A bone shape data generator813generates STL data from the CT data as 3D bone surface model data. An STL data DB814searchably accumulates the STL data generated by the bone shape data generator813.

A display/operation unit815is formed from a display, a touch panel, or the like. The display/operation unit815performs 3D display of a bone image based on the STL data generated by the bone shape data generator813, and performs 3D movement (rotation and movement) of the bone image in accordance with an instruction of the doctor. In this example, the image of the surgery target bone and the image of the unaffected bone of the patient322are displayed simultaneously such that they can be overlaid. The display/operation unit815can also input bone cutting position information of the surgery target bone. The display/operation unit815can independently display 3D movement (rotation and movement) of a plurality of partial bones (first target bone/second target bone) obtained by cutting and separating the surgery target bone at the bone cutting position. A reference bone data generator816laterally inverts the 3D data of the unaffected bone, thereby generating reference bone data.

A 3D data generator817overlays the 3D shape data of the first target bone separated based on the bone cutting position information and that of the reference bone in a virtual 3D space to generate 3D standard bone data. The generated 3D standard bone data is stored in a preoperative preparation data DB819. A 3D data generator818generates 3D shape data of the second target bone. The generated 3D shape data is stored in the preoperative preparation data DB819. Overlay of the target bone and the reference bone may be done based on an operation of the doctor or automatically performed by the 3D data generators817and818based on the bone shape (in particular, the shape of a joint portion). The preoperative preparation data DB819accumulates the 3D data generated by the 3D data generators817and818such that the 3D data can be searched by STL data. The STL data accumulated in the preoperative preparation data DB819is used by the intraoperative image processing system500.

FIG. 9is a block diagram showing another functional arrangement example324B of the information processing apparatus324. Note that the same reference numerals as inFIG. 8denote the same functional components inFIG. 9, and a description thereof will be omitted. Each functional unit shown inFIG. 9is implemented when a CPU processes image data by executing a program using a memory.

In the arrangement shown inFIG. 9, the information processing apparatus324does not have the function of generating STL data from CT data (the program is not installed), unlike the arrangement shown inFIG. 8. Hence, STL data is requested from the external STL data generation server325. An STL data requester921transmits CT data to the STL data generation server325and requests it to generate STL data. An STL data acquirer922receives generated STL data from the STL data generation server325. Note that the CT data to the STL data generation server325or the STL data from the STL data generation server325may be transmitted/received using a storage medium.

STL Data DB

FIG. 10is a view showing the arrangement of the STL data DB814according to this embodiment. The STL data DB814searchably accumulates STL data representing a 3D bone surface model according to this embodiment. Note that the arrangement of the STL data DB814is not limited to that shown inFIG. 10.

The STL data DB814stores a CT data acquisition date/time1002, a patient name1003, an affected part1004, a symptom1005, and CT data1006in association with an image ID1001. The STL data DB814also stores STL data1007generated from the CT data1006, and an STL data generation source1008if the STL data is externally generated.

3D Preoperative Preparation Image DB

FIG. 11is a view showing the arrangement of the preoperative preparation data DB819according to this embodiment. The preoperative preparation data DB819searchably accumulates STL data representing a 3D bone image according to this embodiment. Note that the arrangement of the preoperative preparation data DB819is not limited to that shown inFIG. 11.

The preoperative preparation data DB819stores an affected part1102, a symptom1103, 3D data1104associated with a first marker, and 3D data1105associated with a second marker in association with a patient name1101. The 3D data1104includes the 3D data of a first target bone, the 3D position data of a first marker support instrument, and the 3D data of a reference bone. The 3D data1105includes the 3D data of a second target bone and the 3D position data of a second marker support instrument. Note that the 3D data1104and1105are stored in a format that allows a displayed bone image to move and rotate in the 3D space.

Reference Bone Image Generation Table

FIG. 12is a view showing the arrangement of a reference bone data generation table1200according to this embodiment. The reference bone data generation table1200is a table used by the reference bone data generator816shown inFIG. 8 or 9to generate reference bone data.

The reference bone data generation table1200stores a patient name1202, an affected part1203, a symptom1204, unaffected bone STL data1205, and reference bone STL data1206as a reference bone in association with a 3D reference bone image ID1201.

Processing Procedure of Information Processing Apparatus

FIG. 13is a flowchart showing a processing procedure in the information processing apparatus324. This flowchart is executed by the CPU of the information processing apparatus324using a RAM as a preoperative preparation data generation program to implement the functional components shown inFIG. 8 or 9.

In step S1301, the information processing apparatus324acquires the CT images of an unaffected bone and a target bone with pins being fixed. In step S1303, the information processing apparatus324generates STL data from the CT image data. The information processing apparatus324B shown inFIG. 9transmits the CT image data and acquires STL data.

In step S1305, the information processing apparatus324performs position conversion (STL data coordinate conversion) of setting a bone cutting plane and overlaying a first target bone and a second target bone separated at the bone cutting plane on the reference bone. In step S1307, the information processing apparatus324determines whether the surgery target bone and the reference bone are appropriately overlaid. If the overlay is automatic processing of the information processing apparatus324, the determination is done by shape determination.

If the target bone and the reference bone are not appropriately overlaid, the information processing apparatus324returns to step S1305to repeat rotation and movement of the second target bone with respect to the first target bone. If the target bone and the reference bone are appropriately overlaid, in step S1309, the information processing apparatus324stores the 3D data of the target bone and the reference bone and the position data of the markers and the bone cutting plane with the positional relationship in the preoperative preparation data DB819.

Functional Arrangement of Information Processing Apparatus in Intraoperative Image Processing System

FIG. 14is a block diagram showing the functional arrangement of the tablet computer501in the intraoperative image processing system500according to this embodiment. Each functional unit of the tablet computer501is implemented when a CPU (not shown) executes a program using a memory. Note that in this embodiment, the tablet computer501is used. However, the present invention is not limited to this, and any information processing terminal including a display and a camera is usable. The camera or display/operation unit may be separated from the information processing apparatus, and data communication may be performed between them.

The camera512captures an affected part of the patient322in an operating room. The image capturing range of the camera512includes the markers221and222fixed at two points of the surgery target bone of the forearm213of the patient322. A marker analyzer1411refers to a marker DB1412, and analyzes the type of an image to be displayed and the position and direction to display the image from a marker image captured by the camera512.

Preoperative preparation data1419is the same as the data stored in the preoperative preparation data DB819shown inFIG. 8 or 9. For example, the preoperative preparation data1419may be duplicated from the information processing apparatus324shown inFIG. 8 or 9to the tablet computer501by communication or copied via a storage medium. Alternatively, the preoperative preparation data1419may be acquired by accessing from the tablet computer501to the preoperative preparation data DB819in the information processing apparatus324directly by communication.

A CG image generator1414generates a CG image to be displayed, based on the 3D position and direction of each marker acquired from the marker analyzer1411, the 3D data of the target bone and the reference bone included in the preoperative preparation data1419, and the like. The CG image generator1414functions as a first bone image generator that generates the bone image of the first target bone and the bone image of the reference bone from the 3D data of the first target bone and the 3D data of the reference bone based on the position, size, and direction of the captured first marker. The CG image generator1414also functions as a second bone image generator that generates the bone image of the second target bone from the 3D data of the second target bone based on the position, size, and direction of the captured second marker.

A display image generator1415overlays the surgery target bone image and the reference bone image generated by the CG image generator1414on the affected part image of the forearm213of the patient322captured by the camera512to generate display image data for the display. The display511thus simultaneously displays the target bone image and the reference bone image overlaid on the affected part image. It is also possible to display an image from a moved observation point or simultaneously display images from a plurality of observation points. That is, to search for the positions of the first marker and the second marker at which the second target bone overlaps the reference bone, the display image generator1415displays the bone image of the first target bone, the bone image of the reference bone, and the bone image of the second target bone. In this display, the display image generator1415displays the bone image of the first target bone and the bone image of the second target bone such that their relative positions change in accordance with a change in the relative positions of the first marker and the second marker.

Marker DB

FIG. 15is a view showing the arrangement of the marker DB1412according to this embodiment. The marker DB1412is used by the marker analyzer1411to analyze the 3D position and direction of each marker (that is, the position and direction of a pair of pins) from image data captured by the camera512.

The marker DB1412stores matrix data1502in a case in which a 2D code is captured from the front side in association with a marker ID1501. Here, the matrix data1502arranges, for example, binary or multilevel bit data representing white/black or colors on 2D coordinates. A 3D position and direction can be recognized based on a change in coordinate values. Note that the 2D code is not limited to this. The marker DB1412also stores a marker shape1503in a case in which the marker is captured from the front side, and a marker size1504at a predetermined distance.

Marker Analysis Table

FIG. 16Ais a view showing the arrangement of a marker analysis table1601used by the marker analyzer1411. The marker analysis table1601is a table used to obtain 2D data on the marker, the position, size, and direction of the marker, or 3D data of a marker support instrument from the image of a marker captured by the camera512and generate 3D display data of the target bone image or reference bone image.

The marker analysis table1601stores a 2D code frame1611of a marker extracted from a capture image, matrix data1612of the 2D code of the marker, and a marker ID1613discriminated from the matrix data1612. The marker analysis table1601also stores a position, size, and direction1614of the marker, and a 3D position and direction1615of the marker calculated from the position, size, and direction1614of the marker. The position, size, and direction to display 3D data of the target bone to be displayed on the display can be determined in accordance with the 3D position and direction1615of the marker.

3D Data Generation Table

FIG. 16Bis a view showing the arrangement of intraoperative target bone alignment tables1602and1603used by the CG image generator1414. The intraoperative target bone alignment table1602stores analyzed 3D position data1622of the first marker, and 3D position data1623of the first marker stored in the preoperative preparation data DB819in association with a first target bone and reference bone ID1621. Using a conversion vector that converts the 3D position data1623of the first marker into the 3D position data1622, the coordinates of 3D data of the first target bone stored in the preoperative preparation data DB819are converted. The intraoperative target bone alignment table1602stores 3D data1624of the first target bone for display, which is generated by the coordinate conversion. In addition, the coordinates of the 3D data of the reference bone stored in the preoperative preparation data DB819are converted using the same conversion vector, thereby generating and storing 3D data1625of the reference bone for display.

The intraoperative target bone alignment table1603stores analyzed 3D position data1632of the second marker, and 3D position data1633of the second marker stored in the preoperative preparation data DB819in association with a second target bone ID1631. Using a conversion vector that converts the 3D position data1633of the second marker into the 3D position data1632, the coordinates of 3D data of the second target bone stored in the preoperative preparation data DB819are converted. The intraoperative target bone alignment table1603stores 3D data1634of the second target bone for display, which is generated by the coordinate conversion.

Processing Procedure of Information Processing Apparatus in Intraoperative Image Processing System

FIG. 17is a flowchart showing the processing procedure of the tablet computer501according to this embodiment. This flowchart is executed by the CPU of the tablet computer501using a RAM as an intraoperative image generation program to implement the functional components shown inFIG. 14.

In step S1701, the tablet computer501captures an affected area (in this example, the forearm portion) and acquires image data of two markers and the affected part image. In step S1703, the tablet computer501extracts a frame including a 2D code from the image data of the affected area. Note that in this example, the frame including the 2D code has a rectangular shape. However, a circular shape or any other shape is also usable. In step S1705, the tablet computer501acquires the matrix of the 2D code in the frame.

In step S1707, the tablet computer501compares the acquired matrix of the 2D code and the 2D code viewed from the front side, which is stored in the marker DB1412, thereby specifying the marker. The tablet computer501also analyzes the marker coordinate system (the position and direction in the 3D space) in consideration of the position, size, and direction of the marker. In step S1709, the tablet computer501calculates the 3D data of the first marker fixed to the first target bone and the 3D data of the second marker fixed to the second target bone based on the analyzed 3D positions and directions of the markers. In step S1711, the tablet computer501calculates the 3D data of the first target bone and the reference bone for display from the calculated 3D data of the first marker support instrument based on 3D data stored as the preoperative preparation data1419. In step S1713, the tablet computer501calculates the 3D data of the second target bone for display from the calculated 3D data of the second marker support instrument based on 3D data stored in the preoperative preparation data DB819. In step S1715, the tablet computer501overlays and displays the captured affected part image, the generated images of the first target bone and the reference bone, and the generated bone image of the second target bone.

In step S1717, the tablet computer501determines whether the generated second target bone image is appropriately overlaid on the reference bone image. If the reference bone image and the generated second target bone image are not appropriately overlaid, the tablet computer501returns to step S1701to detect the positions and directions of the two markers according to the movement of the surgery target bone again. If the reference bone image and the generated second target bone image are appropriately overlaid, the processing ends. Note that in actuality, the pins at two points are fixed when the reference bone image and the generated second target bone image are appropriately overlaid, thereby fixing the disposition of the surgery target bone at an appropriate position.

According to this embodiment, since an appropriate disposition of the surgery target bone can be determined without making a large incision in the affected part, surgery can be performed with a minimum incision in the affected part. In addition, since the appropriate disposition of the surgery target bone is determined by preoperative processing, surgery can be carried out quickly and properly. That is, it is possible to support an accurate disposition of the surgery target bone, accurate setting of the bone cutting position, creation of a necessary connecting bone, and proper bonesetting processing at the time of surgery.

Note that in this embodiment, the bone cutting plane431is determined in advance by the method described with reference toFIG. 4, and the determined bone cutting plane431is also presented to the doctor during surgery to attain accuracy in bone cutting. However, the present invention is not limited to this. For example, concerning bone cutting, the plane may be determined during surgery. In this case, the AR technology may be used to only accurately set the relative positional relationship between the two bones sandwiching the bone cutting plane to the reference bone. In this case, the pins need not be inserted before surgery, as shown inFIG. 3B. That is, after the bone cutting plane is set at an appropriate position, only the relative moving distances and rotation angles of the separated target bones441and442to attain a disposition at an ideal position, as indicated by the image404shown inFIG. 4, are stored. This obviates the necessity of performing CT scan of the bone with the pins inserted in advance. The pins are inserted at appropriate positions sandwiching the bone cutting plane, and the markers are attached during surgery. Since how the relative positional relationship of the pins before bone cutting needs to be changed after bone cutting is known, the target displacement (rotation amount, rotation direction, moving amount, and moving direction) of the bone is displayed using an arrow or the like and presented to the doctor. Especially in normal fracture treatment, since a bone is separated into two from the beginning, the bone cutting position need not be determined. That is, 3D data generated by CT scan or the like is directly used and overlaid on the reference bone.

In the fourth stage, the rotation direction, the rotation angle, and the moving distance in millimeter with respect to the target bone442necessary for the target bone441to overlap the reference bone412may be stored as numerical values. This makes it possible to visually (by an image of an arrow or the like) indicate the rotation direction, the amount of rotation, the moving direction, and the amount of movement necessary for the marker221(that is, the arm) fixed to the actual pins211in surgery.

Third Embodiment

A surgical operation support system according to the third embodiment of the present invention will be described next. The surgical operation support system according to this embodiment is different from the second embodiment in that an affected part and a display screen can be overlaid and observed using a head mounted display (Eye-Trek). The rest of the components and operations is the same as in the second embodiment. Hence, the same reference numerals denote the same components and operations, and a detailed description thereof will be omitted. Note that in this embodiment, a binocular head mounted display1801includes a stereo camera and can therefore perform 3D image capturing. The binocular head mounted display1801is a binocular optical see-through type, and can perform 3D display. However, one camera may be provided to do 2D display. The head mounted display may be a monocular type or video see-through type.

Arrangement of Intraoperative Image Processing System

FIG. 18is a view showing the schematic arrangement of an intraoperative image processing system1800according to this embodiment. Note that the same reference numerals as inFIG. 5Adenote the same constituent elements inFIG. 18, and a description thereof will be omitted.

The binocular head mounted display1801serves as glasses also functioning as a display with a camera. With the binocular head mounted display1801, markers221and222in an affected area (in this example, the forearm portion) of a patient322can be captured by the camera, and a target bone image and a reference bone image can be displayed. In addition, the affected part of the patient322can be seen through via the display.

When the binocular head mounted display1801is used, it is possible to overlay and observe the affected part of the patient and the display screen for alignment.

Processing of Information Processing Apparatus

FIG. 19is a view for explaining the processing of an information processing apparatus1802of the intraoperative image processing system1800according to this embodiment. Note that the same constituent elements as inFIG. 5Aare not illustrated inFIG. 19, or a description thereof will be omitted. Alternatively, the constituent elements are denoted by the same reference numerals, and a description thereof will be omitted.

The binocular head mounted display1801includes cameras1911and1912that capture the markers221and222, a display1913that displays a reference bone image and a target bone image, and a communicator1914that communicates with the information processing apparatus1802. The display1913displays the CG images of target bones441and442and a reference bone412. Since the display1913is a translucent display, a forearm213that is the affected part of the patient322can be observed through the display, as indicated by the broken line. Note that althoughFIG. 19shows the two cameras1911and1912, one camera may be used. A camera may separately be prepared.

Note that the information processing apparatus1802shown inFIG. 19has an arrangement obtained by removing the camera512and the display511from the functional arrangement of the tablet computer501described with reference toFIG. 14and providing a communicator1921. Hence, the information processing apparatus1802receives the image data of the 2D codes of the markers221and222from the cameras1911and1912, and transmits display data of a prepared reference bone image and a generated target bone image to the display1913, unlike the tablet computer501.

Functional Arrangement of Information Processing Apparatus in Intraoperative Image Processing System

FIG. 20is a block diagram showing the arrangement of an information processing apparatus2110in the intraoperative image processing system according to this embodiment. Note that the same reference numerals as inFIG. 14denote the same functional components inFIG. 20, and a description thereof will be omitted.

The communicator1921controls reception of image data from the cameras1911and1912and transmission of display image data to the display1913(a right-eye unit1913aand a left-eye unit1913b) of the binocular head mounted display1801. An image receiver2011receives the image data of images captured by the cameras1911and1912.

An eye coordinate system estimator2012estimates an eye coordinate system based on the line of sight or visual field of the doctor wearing the binocular head mounted display1801from the received captured image data of the cameras1911and1912.

A right-eye HMD display data generator2016refers to eye coordinate system information from the eye coordinate system estimator2012, and converts display image data on a 3D camera coordinate system into right-eye display data for a 2D HMD screen coordinate system. A left-eye HMD display data generator2017refers to the eye coordinate system information from the eye coordinate system estimator2012, and converts display image data on the 3D camera coordinate system into left-eye display data for the 2D HMD screen coordinate system. The display position of the converted display data for the 2D HMD screen coordinate system is adjusted such that the 3D target bone image and the reference bone image overlap the forearm213of the affected part seen through the display1913of the binocular head mounted display1801. It is also possible to display an image from a moved observation point or simultaneously display images from a plurality of observation points. Note that image display conversion by moving the observation point can be performed by converting the coordinate system, and a detailed description thereof will be omitted. An image transmitter2018transmits the display image data for the 2D HMD screen coordinate system to the display1913of the binocular head mounted display1801via the communicator1921.

According to this embodiment, since the affected part of the patient and the display screen for alignment can be overlaid and observed, the burden on the doctor in alignment can be reduced. Note that considering the influence of wireless communication in the operating room, wired communication may be used between the glasses and the information processing apparatus.

Fourth Embodiment

A surgical operation support system according to the fourth embodiment of the present invention will be described next. The surgical operation support system according to this embodiment is different from the second embodiment and the third embodiment in that the degree of overlay (matching ratio) between a surgery target bone and a reference bone and a direction and distance to bend or stretch an affected part, and the like are displayed on the display screen of a bone image. The rest of the components and operations is the same as in the second embodiment. Hence, the same reference numerals denote the same components and operations, and a detailed description thereof will be omitted. Note that in this embodiment, the matching ratio is displayed as a percentage. However, any other display method such as display using a difference in the color or the length of a bar chart is usable.

Display of Matching Ratio

FIG. 21is a view showing a display example of the matching ratio between a reference bone and a surgery target bone according to this embodiment.FIG. 21corresponds to the images522to526shown inFIG. 5Bof the second embodiment.

A display screen2101displays the matching ratio in addition to the images of the surgery target bone and the reference bone. A display screen2102displays the matching ratio in addition to the images523to526. Note that as the matching ratio, for example, a value obtained by multiplying the average value or maximum value of the distances of 3D bone surface data by a predetermined value is usable. However, the value is not limited to this, and various existing matching ratios are usable.

A display screen2127displays an arrow2181representing how to manipulate the affected part to appropriately overlay the surgery target bone on the reference bone. According to the example of this drawing, when the hand is bent based on the arrow2181, correction of the affected bone can be done more easily, quickly, and properly.

According to this embodiment, it is possible to not only visually and sensibly grasp the overlay state of bone images and the overlay state of markers or pins on the display screen but also determine them by objective numerical values. Hence, the target bone can be disposed at a more appropriate position. Note that a marker position that should be may be determined in advance and displayed during surgery, and the matching ratio between the actual marker position and the displayed marker position may be displayed. A sound may be produced according to the matching ratio.

Fifth Embodiment

A surgical operation support system according to the fifth embodiment of the present invention will be described next. The surgical operation support system according to this embodiment is different from the second to fourth embodiments in that when generating preoperative preparation data, virtual 3D markers are generated on a screen and created by a 3D printer without disposing actual markers on a target bone. Note that the same reference numerals denote the same components and operations as in the above-described embodiments, and a detailed description thereof will be omitted.

Note that in this embodiment, artificial joint replacement surgery of an elbow will mainly be described. However, this embodiment is also applicable to other techniques, for example, corrective osteotomy for malunion, surgery of osteoarthritis, and the like.

Outline of Surgical Operation Support Processing

FIGS. 22A and 22Bare views for explaining the outline of the processing of a surgical operation support system2200according to this embodiment.FIG. 22Ais a view showing the entire sequence of artificial joint replacement surgery.FIG. 22Bis a view showing details of each processing according to this embodiment. Note that surgery of replacing the joint between the humerus and the ulna in an elbow will be described with reference toFIGS. 22A and 22B.FIGS. 22A and 22Binclude both a case in which the screen is the display screen of a display and a case in which the screen shows the overlay state of the display screen and a visually observed image in the optical through HMD.

Preoperative Preparation Processing

In the artificial joint replacement surgery according to this embodiment, first, CT imaging of the affected part is performed, and STL data is generated based on the CT data as preoperative preparation. While displaying the images of the humerus and the ulna by the STL data, virtual 3D markers are generated and disposed on the humerus and the ulna. Note that as for the disposing positions, each 3D marker is preferably disposed at a position near the artificial joint implantation position between the humerus and the ulna, where the 3D marker attains a characteristic shape, in a direction in which the marker surface of the 3D marker can easily be captured by a camera (the marker surface does not disappear from the visual field of the camera). Note that to facilitate image capturing by the camera, the direction of the marker surface and the base block (base) of the 3D marker placed on a bone can be changed, as shown on the lower side ofFIG. 22A. At this time, the positions and directions of a bone cutting plane if bone cutting is necessary, the implant of an artificial joint to be used in this surgery, a bone surface (to be resected) to place the implant or a bone hole to fix the implant, and the like and the positions and directions of the 3D markers are set and stored in association with each other by 3D data.

Next, implants to be used are prepared, and the 3D markers are produced by a 3D printer. In addition, instruments to be used to cut a bone or resect a bone surface or instruments used to form a bone hole during surgery are prepared.

Intraoperative Processing

During surgery, the base block of the 3D marker for the humerus is placed so as to match the bone shape, as indicated by screens2213and2214. The 3D marker is captured by the camera, thereby determining the position and direction of the humerus from the position and direction of the 3D marker. Bone cutting is executed while displaying a bone cutting plane2211of a humerus2210overlaid on the surgery target humerus. Next, the bone is resected while displaying the image of STL data of a humerus with a shape matching the implant overlaid on the surgery target humerus. In addition, a hole is formed in the bone while displaying a bone hole to fix the implant overlaid on the surgery target humerus. Then, a humerus component2212of the implant is placed.

Similarly, the base block of the 3D marker for the ulna is placed so as to match the bone shape, as indicated by screens2223and2224or screens2225and2226. The 3D marker is captured by the camera, thereby determining the position and direction of the ulna from the position and direction of the 3D marker. Bone cutting is executed while displaying a bone cutting plane2221of an ulna2220overlaid on the surgery target ulna. Next, the bone is resected while displaying the image of STL data of an ulna with a shape matching the implant overlaid on the surgery target ulna. In addition, a hole is formed in the bone while displaying a bone hole to fix the implant overlaid on the surgery target ulna. Then, an ulna component2222of the implant is placed.

In the above-described way, each 3D marker produced by the 3D printer is placed so as to match the bone shape during surgery. This makes it possible to do surgery without forming holes in the surgery target bone of the patient and placing markers before and during surgery as in the above-described embodiment.

FIG. 22Bis a view for explaining the outline of the processing of the surgical operation support system2200according to this embodiment.FIG. 22Bshows screens according to this embodiment in each processing during artificial joint replacement surgery. Note that the 3D markers inFIG. 22Bare identical. Note thatFIG. 22Bshows an extracted part for understanding of the embodiment and is not limited to this.

A screen2230is a screen when performing bone cutting of the humerus. A 3D marker2231placed on the humerus, a bone cutting plane2232, and an image2233of STL data of the ulna are shown on the screen2230. A screen2240is a screen when the bone surface is being resected for matching between the surface of the humerus and the implant to be placed. A 3D marker2241, an instrument2243used to resect the bone, and a disposing plane2242of the instrument2243are shown on the screen2240. A screen2250is a screen when a bone hole to fix the implant to be disposed on the humerus is being formed. A 3D marker2251, an instrument2253used to form a hole, and a target position image2252of the bone hole are shown.

A screen2260shows a screen before the implant is placed on the humerus. A 3D marker2261and an image2262of STL data of the ulna are shown on the screen2260. A screen2270is a screen when the implant is being placed on the humerus. A 3D marker2271, an implant disposition image2272, and an actual implant2273are shown on the screen2270. A screen2280is a screen before the implant is placed on the ulna. A 3D marker2281, the implant2273placed on the humerus, and an implant image2282to be disposed on the ulna are shown on the screen2280.

Processing Procedure of Surgical Operation Support Processing

FIG. 23is a flowchart showing the processing procedure of the surgical operation support system2200according to this embodiment.

In step S2301, the surgical operation support system2200performs CT imaging of the affected part of a patient. In step S2303, the surgical operation support system2200forms a 3D model based on, for example, STL data. In step S2305, the surgical operation support system2200makes preoperative planning while displaying the 3D data. For example, a 3D marker is generated on the screen, and data to produce the 3D marker is generated. In addition, the 3D marker is associated with a surgery target bone, a bone cutting plane, a bone hole, an implant, and the like on 3D coordinates. In step S2307, the surgical operation support system2200produces a 3D marker having a base block matching the target bone by a 3D printer based on the data of the 3D marker.

In step S2309, the surgical operation support system2200inputs the processing program of an intraoperative application and each data associated with the 3D marker. In step S2311, the surgical operation support system2200executes surgery support based on the processing program of the intraoperative application and each data associated with the 3D marker.

Functional Arrangement of Preoperative Preparation Data Generation System

FIG. 24is a block diagram showing the functional arrangement of an information processing apparatus2410in a preoperative preparation data generation system2400according to this embodiment. Note that the same reference numerals as inFIG. 8denote the same functional components inFIG. 24, and a description thereof will be omitted.

As shown inFIG. 24, when capturing an affected part by a CT321, no marker is placed on a patient322. A bone image data generator2411is a functional component including the reference bone data generator816and the 3D data generators817and818shown inFIG. 8. A 3D marker data generator2412generates 3D data of a 3D marker generated based on 3D marker information input to a display/operation unit2415. An artificial joint data generator2413generates 3D data of an artificial joint based on artificial joint information input to the display/operation unit2415. Note that if an artificial joint prepared in advance is to be used, the data may be stored in an STL data DB814in advance. A preoperative preparation data DB2419stores 3D data of a surgery target bone, a bone cutting plane, a bone hole, an implant of an artificial joint, and the like in association with the 3D data of the 3D marker.

A 3D printer2420produces a 3D marker based on 3D printer data generated from the 3D data of the 3D marker.

3D Preoperative Preparation Image DB

FIG. 25is a view showing the arrangement of the preoperative preparation data DB2419according to this embodiment.FIG. 25shows the arrangement of preparation data planned in a technique unique to this embodiment. Note thatFIG. 25also includes the arrangement illustrated inFIG. 11.

The preoperative preparation data DB2419stores an affected part2502and a technique2503in association with a patient name2501. The preoperative preparation data DB2419also stores a planning item2504necessary for the affected part2502and the technique2503, and 3D data necessary for the planning item in association with a 3D marker.

Processing Procedure of Preoperative Preparation Data Generation System

FIG. 26is a flowchart showing the processing procedure of the information processing apparatus2410in the preoperative preparation data generation system2400according to this embodiment. This flowchart is executed by the CPU of the information processing apparatus2410using a RAM to implement the functional components shown inFIG. 24. Note that artificial joint replacement surgery will be described with reference toFIG. 26. However, this flowchart is also applicable to other techniques.

In step S2601, the information processing apparatus2410acquires a CT image of an affected part of the patient and, if necessary, a CT image of an unaffected bone. In step S2603, the information processing apparatus2410generates STL data from the CT image data. When requesting an external apparatus to generate STL data, the information processing apparatus2410acquires the STL data. In step S2605, the information processing apparatus2410determines the technique.

If the technique is artificial joint replacement surgery, in step S2607, the information processing apparatus2410acquires implant shape and set position information, bone cutting plane and bone hole position information, and 3D marker shape and set position information. In step S2609, the information processing apparatus2410generates implant 3D data, 3D data of the bone cutting plane and bone hole, 3D marker data, and the like in association with the 3D data of the STL bone. In step S2611, the information processing apparatus2410associates the generated 3D data and stores them in the preoperative preparation data DB2419. In step S2613, when newly generating an implant, the information processing apparatus2410outputs the implant 3D data and also outputs the 3D marker data for the 3D printer.

If the technique is another technique in step S2605, in step S2615, the information processing apparatus2410generates 3D preparation data of the other technique in association with a 3D marker (for the data of other techniques, seeFIG. 25).

Functional Arrangement of Intraoperative Image Processing System

FIG. 27is a block diagram showing the functional arrangement of a tablet computer2710in an intraoperative image processing system2700according to this embodiment. The same reference numerals as inFIG. 5 or 14denote the same functional components inFIG. 27, and a description thereof will be omitted.

The preoperative preparation data DB2419stores the same preparation data generated by the preoperative preparation data generation system2400shown inFIG. 24. A CG image generator2714performs 3D coordinate conversion of 3D data of the surgery target bone, the bone cutting plane, the bone hole, and the like from the preoperative preparation data DB2419in correspondence with the position and direction of the 3D marker from a marker analyzer1411, thereby generating a CG image to be overlaid on a visible surgery part. A display image generator2715converts the image generated by the CG image generator2714into a display image to be displayed on a display511, an external monitor2720, or an HMD2730. Note that in this embodiment, an optical see-through HMD is preferably used.

Processing Procedure of Intraoperative Image Processing System

FIG. 28is a flowchart showing the processing procedure of the tablet computer2710in the intraoperative image processing system2700according to this embodiment. This flowchart is executed by the CPU of the tablet computer2710shown inFIG. 27using a RAM to implement the functional components shown inFIG. 27. Note that artificial joint replacement surgery will be described with reference toFIG. 28. However, this flowchart is also applicable to other techniques.

In step S2801, the tablet computer2710determines the technique. If the technique is artificial joint replacement surgery, in step S2803, the tablet computer2710captures the elbow joint portion of the humerus and a 3D marker produced by the 3D printer2420and placed on the humerus. In step S2805, the tablet computer2710analyzes the 2D code on the 3D marker and calculates the position and direction of the humerus. In step S2807, the tablet computer2710displays the positions and directions of a bone cutting plane, a bone hole, an artificial joint implant, and each instrument overlaid on the elbow joint portion in correspondence with the position and direction of the 3D marker as the surgery progresses (seeFIGS. 22A and 22B). In step S2809, the tablet computer2710determines whether the processing of the humerus has ended. If the processing has not ended, the tablet computer2710returns to step S2803to process the humerus.

In step S2811, the tablet computer2710captures the elbow joint portion of the ulna and a 3D marker produced by the 3D printer2420and placed on the ulna. In step S2813, the tablet computer2710analyzes the 2D code on the 3D marker and calculates the position and direction of the ulna. In step S2815, the tablet computer2710displays the positions and directions of a bone cutting plane, a bone hole, an artificial joint implant, and each instrument overlaid on the elbow joint portion in correspondence with the position and direction of the 3D marker as the surgery progresses (seeFIGS. 22A and 22B). In step S2817, the tablet computer2710determines whether the processing of the ulna has ended. If the processing has not ended, the tablet computer2710returns to step S2811to process the ulna.

Note thatFIG. 28illustrates performing the ulna after the processing of the humerus. However, the processes may be performed in a reverse order or progress simultaneously.

According to this embodiment, during surgery, a 3D marker produced by the 3D printer is placed so as to match the shape of a bone during surgery. This makes it possible to support surgery without forming holes in the surgery target bone of the patient and placing markers before and during surgery.

Sixth Embodiment

A surgical operation support system according to the sixth embodiment of the present invention will be described next. The surgical operation support system according to this embodiment is different from the second to fifth embodiments in that 3D data of a target bone is acquired by a depth sensor in intraoperative image processing using, as a marker, 3D data of a part in which the surgical operation of a target bone is performed. Note that the same reference numerals denote the same components and operations as in the above-described embodiments, and a detailed description thereof will be omitted.

Note that preoperative preparation data according to this embodiment is similar to that of the above-described embodiments except that separate marker information is not included because the 3D surface image of the surgery target bone is used as a marker, and a description thereof will be omitted. In the following embodiment, a case in which an HMD and a depth sensor are integrated will be described. If the HMD and the depth sensor are separated, position determination needs to be done by adding a marker to a position sensor (for example, GPS) or depth sensor.

Functional Arrangement of Intraoperative Image Processing System

FIG. 29is a block diagram showing the functional arrangement of an information processing apparatus2910in an intraoperative image processing system2900according to this embodiment.

Note that the same reference numerals as inFIG. 14orFIG. 20denote the same functional components inFIG. 29, and a description thereof will be omitted.

A depth sensor & HMD2920includes a depth sensor and an optical see-through HMD. Note that the depth sensor and the HMD may be separate but are preferably integrated. The depth sensor is formed from an infrared projector2921and an infrared camera2922, and acquires a depth image (distance image) of a surgery part during surgery. The distance image is equivalent to the 3D image of a surface.

An image receiver2911receives a depth image (distance image). A bone surface image collator2912performs collation with a characteristic surface image of a target bone image of preoperative preparation data1419using the depth image (distance image) as a marker. A CG image generator1414performs 3D coordinate conversion of the 3D data of the preoperative preparation data1419in correspondence with a change in the position and direction necessary for collation of the bone surface obtained from the bone surface image collator2912, thereby generating a CG image.

As described above, in this embodiment, the 3D image of the surgery target bone is used as a marker. This makes it possible to support surgery without separately creating a marker, as in the above-described embodiments.

Data Table of Bone Image Collator

FIG. 30is a view showing a data table3000used by the bone image collator2912according to this embodiment. The data table3000collates the depth image (distance image) that the depth sensor has acquired from the surface of the surgery target bone of the affected part of the patient with the surgery target bone stored as the preoperative preparation data1419, and determines the position and direction of the current surgery target bone.

The data table3000stores collated 3D bone data3002and a real space position and direction3003of the target bone determined from the collation result in association with a depth sensor image3001. The data table3000stores 3D bone data3004and 3D data3005of the positions and directions of a bone cutting plane, a bone hole, an implant, and each instrument, which are obtained by 3D coordinate conversion, in correspondence with the real space position and direction3003of the target bone.

Processing Procedure of Intraoperative Image Processing System

FIG. 31is a flowchart showing the processing procedure of the information processing apparatus2910in the intraoperative image processing system2900according to this embodiment. This flowchart is executed by the CPU of the information processing apparatus2910shown inFIG. 29using a RAM to implement the functional components shown inFIG. 29. Note that the same step numbers as inFIG. 28denote the same steps, and a description thereof will be omitted. Artificial joint replacement surgery will be described with reference toFIG. 31. However, this flowchart is also applicable to other techniques.

In step S3103, the information processing apparatus2910captures the elbow joint portion of the humerus by the depth sensor. In step S3105, the information processing apparatus2910performs matching between the humerus surface of the depth sensor image and stored 3D bone data of a humerus corresponding portion, and calculates the position and direction of the humerus. In step S3107, the information processing apparatus2910displays the positions and directions of a bone cutting plane, a bone hole, an artificial joint implant, and each instrument overlaid on the elbow joint portion in correspondence with the position and direction of the humerus. In step S3109, the information processing apparatus2910determines whether the processing of the humerus has ended. If the processing has not ended, the information processing apparatus2910returns to step S3103to process the humerus.

In step S3111, the information processing apparatus2910captures the elbow joint portion of the ulna by the depth sensor. In step S3113, the information processing apparatus2910performs matching between the ulna surface of the depth sensor image and stored 3D bone data of an ulna corresponding portion, and calculates the position and direction of the ulna. In step S3115, the information processing apparatus2910displays the positions and directions of a bone cutting plane, a bone hole, an artificial joint implant, and each instrument overlaid on the elbow joint portion in correspondence with the position and direction of the ulna. In step S3117, the information processing apparatus2910determines whether the processing of the ulna has ended. If the processing has not ended, the information processing apparatus2910returns to step S3111to process the ulna.

Note thatFIG. 31illustrates performing the ulna after the processing of the humerus. However, the processes may be performed in a reverse order or progress simultaneously.

According to this embodiment, the 3D image of the surface of the surgery target bone is used as a marker. This makes it possible to support surgery without separately creating a marker, as in the above-described embodiments.

Other Embodiments

Note that in the above embodiments, an affected part with marker instruments being fixed is captured by CT imaging. In preoperative preparation data generation processing, a set of a first target bone that is one of two divided surgery target bones and a reference bone is stored in association with a first marker, and a second target bone is stored in association with a second marker, thereby supporting determination of the disposition of the surgery target bone during surgery. However, if the fixing position of each marker instrument can accurately be determined, the marker instruments may be fixed after CT imaging or during surgery. In this case, associating the first target bone with the first marker and associating the second target bone with the second marker are performed during surgery. Instead of storing the set of the first target bone and the reference bone in association with the first marker, the first target bone and the second target bone may be matched with the reference bone during surgery. In this case, for example, during surgery, the first target bone may be matched with the reference bone, and the set of the first target bone and the reference bone may be stored in association with the first marker. After that, the second target bone may be manipulated so as to match the reference bone.

In the above embodiments, the surgical operation support system according to the present invention has been described using nonunion surgery as an example. However, the same effects as described above can be obtained by applying the system to fracture treatment, artificial joint replacement surgery, or the like. In the embodiments, a surgery target bone divided into two target bones has been described. If the surgery target bone is divided into three or more target bones, the embodiments can directly be expanded by fixing a marker to each of the separated target bones. For example, in artificial joint replacement surgery, three markers are supposed to be fixed to two bones on both sides of a joint and an artificial joint.

The present invention is applicable to a system including a plurality of devices or a single apparatus. The present invention is also applicable even when a surgical operation support program for implementing the functions of the embodiments is supplied to the system or apparatus directly or from a remote site. Hence, the present invention also incorporates a control program installed in a computer to implement the functions of the present invention by the computer, a medium storing the control program, and a WWW (World Wide Web) server that causes a user to download the control program. Especially, the present invention incorporates at least a non-transitory computer readable medium storing a control program that causes a computer to execute processing steps included in the above-described embodiments.

This application claims the benefit of Japanese Patent Application No. 2013-123209 filed on Jun. 11, 2013, which is hereby incorporated by reference herein in its entirety.