Patent Description:
Surgery is in general an invasive procedure; the more invasive it is, the greater the time that is needed for recovery and the greater the probability of post-surgery complications. Medical technology has always looked for improvements to limit the likelihood of post-surgery complications, and to date minimally-invasive surgeries are performed routinely. In this regard, the improvement of digital medical imaging techniques has provided a major boost to the diffusion of minimal invasive procedures. Today, physicians often rely on digital imaging as a tool to plan surgeries, as well as a visual guide during actual surgery, to access sites that would otherwise be hidden from view.

The inventors have thus discovered that a need exists for enhancements to digital imaging tools that provide visual guides during surgery. To this end, embodiments described herein illustrate methods, apparatus, and systems for utilizing augmented reality, based on the use of three dimensional (3D) medical imaging to enhance the use of surgical tools such as a neuronavigator (a machine able to track the position of surgical probes inside the brain or other operating areas of a patient, and visualize it on MRI or CT scans of the patient itself). As described herein, instead of using a remote screen displaying single plane (i.e., two dimensional or 2D) scans of a target area of a patient (e.g., a body part, a tumor, an area into which a foreign object has been lodged, or the like), a surgeon employing example embodiments described herein may utilize a head-mounted display to perceive the 3D model of a target area during a surgery. This 3D model can be extracted from a scan, and visualized directly as a superimposition onto the actual target area of the patient undergoing surgery using augmented reality glasses.

It will be appreciated that the scope of the invention encompasses many potential embodiments, some of which will be further described below.

Some embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. Indeed, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements.

Neuronavigation systems are unique tools used to access lesions or tumors or other target areas located far too deep inside the brain for traditional surgical methods and which commonly result in surgical sites that do not have clear visibility or access. In order to provide sufficiently accurate tracking to reach such small targets, the set-up of assistive machinery in the operating room (e.g., a neuronavigator or the like) requires many hours of work before starting the actual surgery. Also, the precision by which surgical tools associated with the neuronavigator can be tracked is limited by the fact that the system cannot take into account the natural movements and deformations (swelling) occurring in the target area (e.g., the brain) during surgery.

Given this background, the use of the assistive machine itself is not very comfortable because the surgeon needs to focus attention on a screen that illustrates the actual position of the probe, while the surgeon's hands are operating on an operatory field that is oriented differently from this field of view. It is not uncommon, for instance, for the orientation of the images on the screen to be the reverse of the physical orientation in the actual working area, a situation that requires an extra level of concentration and may limit a surgeon's operative capabilities. These issues limit the distribution and use of assistive devices such as neuronavigators.

The inventors have identified a need for simplifying the preparation of the operating room, improving the comfort of the surgeon by enabling the direct visualization of the position and the path of the surgical probes superimposed on the real field of view of the surgeon, and improving the tracking of both the surgical instruments and the target areas upon which surgery is performed, in order to proficiently operate on small targets.

To address these needs and others, example embodiments described herein utilize improved digital imaging tools that provide a three dimensional visual guide during surgery. In particular, a head-mounted display provides augmented reality visualizations that improve a surgeon's understanding of the relative locations of a target area within the context of the surgeon's field of view, and, in some embodiments, also illustrates a relative location of the surgeon's surgical tools.

Historical systems and applications for digitally-enhanced surgical applications have not taken advantage of augmented reality and 3D superimposition, and no historical mechanism for enhancing surgery has considered the use of augmented reality 3D glasses as described herein. Using augmented reality 3D glasses, example embodiments deploy the 3D tracking technology and the 3D visualization of the surgical field in a manner that allows the surgeon to focus his attention (sight and hands) on the operatory field and on the patient rather than on separate screen. Similarly, using example embodiments described below, the surgeon is not required to interpolate movements that appear on an external display in one orientation on the fly to properly respond with movements in the actual working area.

<FIG> shows an example system that may perform the operations described herein for utilizing three dimensional augmented reality visualizations to assist surgery. In this regard, it is contemplated that computing device <NUM> may be configured to perform various operations in accordance with example embodiments of the present invention, such as in conjunction with the operations described in conjunction with <FIG> below. It should be noted that the components, devices, and elements described herein may not be mandatory in every embodiment of the computing device <NUM>, and some may be omitted in certain embodiments. Additionally, some embodiments may include further or different components, devices or elements beyond those shown and described herein.

As shown in <FIG>, the computing device <NUM> may include or otherwise be in communication with a processing system including, for example, processing circuitry <NUM> that is configurable to perform actions in accordance with example embodiments described herein. The processing circuitry <NUM> may be configured to perform data processing, application execution and/or other processing and management services according to an example embodiment of the present invention. In some embodiments, the computing device <NUM> or the processing circuitry <NUM> may be embodied as a chip or chip set. In other words, the computing device <NUM> or the processing circuitry <NUM> may comprise one or more physical packages (e.g., chips) including materials, components and/or wires on a structural assembly (e.g., a baseboard). The computing device <NUM> or the processing circuitry <NUM> may, in some cases, be configured to implement an embodiment of the present invention on a single chip or as a single "system on a chip. " As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.

In an example embodiment, the processing circuitry <NUM> may include a processor <NUM> and memory <NUM> that may be in communication with or otherwise control a head-mounted display <NUM> and, in some embodiments, a separate user interface <NUM>. The processing circuitry <NUM> may further be in communication or otherwise control a scanner <NUM> (which may be external to the computing device <NUM> or, in some embodiments, may be included as a component of the computing device <NUM>). In these capacities, the processing circuitry <NUM> may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., with hardware or a combination of hardware and software) to perform or direct performance of operations described herein.

The processor <NUM> may be embodied in a number of different ways. For example, the processor <NUM> may be embodied as various processing means such as one or more of a microprocessor or other processing element, a coprocessor, a controller or various other computing or processing devices including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or the like. In an example embodiment, the processor <NUM> may be configured to execute program code instructions stored in the memory <NUM> or otherwise accessible to the processor <NUM>. As such, whether configured by hardware or by a combination of hardware and software, the processor <NUM> may represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to embodiments of the present invention while configured accordingly. Thus, for example, when the processor <NUM> is embodied as an ASIC, FPGA or the like, the processor may include specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processor <NUM> is embodied as an executor of software instructions, the instructions may specifically configure the processor <NUM> to perform the operations described herein.

The memory <NUM> may include one or more non-transitory memory devices such as, for example, volatile and/or non-volatile memory that may be either fixed or removable. The memory <NUM> may be configured to store information, data, applications, instructions or the like for enabling the computing device <NUM> to carry out various functions in accordance with example embodiments of the present invention. For example, the memory <NUM> could be configured to buffer input data for processing by the processor <NUM>. Additionally or alternatively, the memory <NUM> could be configured to store instructions for execution by the processor <NUM>. As yet another alternative, the memory <NUM> may include one of a plurality of databases that may store a variety of files, contents or data sets. Among the contents of the memory <NUM>, applications may be stored for execution by the processor <NUM> in order to carry out the functionality associated with each respective application. In some cases, the memory <NUM> may be in communication with the processor <NUM> via a bus for passing information among components of the computing device <NUM>.

The head-mounted display <NUM> includes one or more interface mechanisms for augmenting a user's view of the real-world environment. The head-mounted display <NUM> may, in this regard, comprise augmented reality glasses that augment a user's perceived view of the real-world environment. This effect can be achieved in a number of ways. For instance, in some embodiments, the augmented reality glasses may include a display (e.g., an LED or other similar display) in one lens (or both lenses) that produces a video feed that interleaves computer-generated sensory input with the view captured by a camera on the front of the lens (or lenses) of the augmented reality glasses. In such embodiments, the user cannot see directly through the lens (or lenses) providing this interleaved video feed, and instead only sees the interleaved video feed itself. In other embodiments, the user can see through the lenses of the augmented reality glasses, and a display is provided in a portion of one lens (or both) that illustrates computer-generated sensory input relating to the user's view. In yet other embodiments, the computer-generated sensory input may be projected onto a lens (or both lenses) by a projector located in the augmented reality glasses. In such embodiments, the lens (or lenses) may comprise a partially reflective material that beams the projected video into the eye(s) of the user, thus adding the computer-generated sensory input directly to the user's natural view through the lenses of the augmented reality glasses without disrupting the natural view.

The head-mounted display <NUM> may further include spatial awareness sensors (e.g., gyroscope, accelerometer, compass, or the like) that enable the computing system <NUM> to compute the real-time position and orientation of the head-mounted display <NUM>.

In some embodiments, the head-mounted display <NUM> may itself incorporate the rest of the computing device <NUM>, and thus the various elements of the computing device <NUM> may communicate via one or more buses. In other embodiments, the head-mounted display <NUM> may not be co-located with the rest of the computing device <NUM>, in which case the head-mounted display may communicate with the computing device <NUM>, by a wired connection, or by a wireless connection, which may utilize short-range communication mechanisms (e.g., infrared, Bluetooth™, or the like), a local area network, or a wide area network (e.g., the Internet). In embodiments utilizing wireless communication, the head-mounted display <NUM> may include, for example, an antenna (or multiple antennas) and may support hardware and/or software for enabling this communication.

A separate user interface <NUM> (if needed) may be in communication with the processing circuitry <NUM> and may receive an indication of user input and/or provide an audible, visual, mechanical or other output to the user. As such, the user interface <NUM> may include, for example, a keyboard, a mouse, a joystick, a display, a touch screen, a microphone, a speaker, or any other input/output mechanisms. The computing device <NUM> need not always include a separate user interface <NUM>. For example, in some embodiments the head-mounted display <NUM> may be configured both to provide output to the user and to receive user input (such as, for instance, by examination of eye motion or facial movements made by the user). Thus, a separate communication interface <NUM> may not be necessary. As such, the communication interface <NUM> is shown in dashed lines in <FIG>.

Communication interface <NUM> includes means such as a device or circuitry embodied in either hardware, or a combination of hardware and software that is configured to receive and/or transmit data between any device or module of the computing device <NUM> and any external device, such as by a wired connection, a local area network, or a wide area network (e.g., the Internet). In this regard, the head-mounted display <NUM> may include, for example, an antenna (or multiple antennas) and may support hardware and/or software for enabling communications with a wireless communication network and/or a communication modem or other hardware/software for supporting this communication.

The scanner <NUM> may be a tomograph or any other device that can use penetrating waves to capture data regarding an object. For instance, scanner <NUM> may comprise a computed tomography (CT) scanner, a positronic emission tomography (PET) scanner, a single-photon emission computed tomography (SPECT) scanner, or a magnetic resonance imaging (MRI) scanner. The scanner <NUM> is configured to produce a tomogram, which constitutes a two dimensional image illustrating a slice, or section, of the imaged object. In a typical scenario, however, the scanner <NUM> may produce a series of tomograms (referred to herein as an image stack) that, when analyzed in combination, can be used to generate a three dimensional mesh representative of the scanned object.

Having described above some example components that might be employed in the present invention, a description of a high-level procedure utilized by example embodiments will be provided in connection with a discussion of <FIG> and <FIG>.

Turning first to <FIG>, a series of images are shown illustrating an order of operations used to generate a 3D reconstruction of a target area (in these illustrations, the example used is a human brain). Starting first with <FIG>, a patient must undergo a scan (using scanner <NUM> described above) to generate an image stack of one part of the patient's body. In this regard, <FIG> illustrates a patient in a MRI scanner, although any other tomographic scanner (or alternative type of scanner) is contemplated, so long as it can produce an image stack (i.e., a series or sequence of images showing an object of interest) with mm resolution that can subsequently be converted into a 3D mesh. <FIG> depicts a series of images forming an image stack that can subsequently be used to generate the 3D mesh.

<FIG> illustrates the importation of the image stack into application software that can be used to generate a 3D reconstruction of the target area via image segmentation and mesh generation. In one example, a 3D reconstruction of a target area can be performed by loading the image stack into iLastik (a software application for biological image segmentation). The image stack can thus be segmented using the carving module of iLastik, which allow fast, accurate, semi-automated segmentation of the z stack, and to extract a mesh of the 3D object.

In turn, <FIG> illustrates a visual representation of a 3D mesh generated in accordance with example embodiments described in greater detail below. Once this mesh is extracted, its size (in meters or centimeters) may need to be adjusted. Also, eventual compression on the z axis might need to be taken into account, if the voxel size of the 3D mesh is not isotropic (due, for instance to possible difference between the thickness of the optical slides and the xy resolution). This can be achieved using a Blender add-on (e.g., Neuromorph, a free collection of add-ons for Blender 3D modeling software, which allows importing 3D meshes and resizing them if the voxel size is known).

At this point, the 3D object of the target area can thus comprise a simple file in standard. obj format, which can then be loaded into augmented reality (AR) glasses (e.g., via a USB cable, SD card, or the like). The surgeon can then load the 3D model into augmented reality software within augmented reality glasses.

Turning next to <FIG>, a sequence of images are illustrates that demonstrate the connection between various elements of the present invention utilized in example embodiments described herein. In <FIG>, a surgeon is illustrated utilizing traditional assistive machinery for surgery. In this regard, <FIG> demonstrates the use of a 2D representation of a target area (in this case, a brain) that requires the surgeon to alternately view the surgical site and then an external display illustrating the target area. In contrast, the surgeon can utilize augmented reality glasses, such as those shown in <FIG>, to address this deficiency of traditional assistive machinery. In this regard, the augmented reality glasses of example embodiments of the present invention project the three dimensional model of the target area onto the field of view of the surgeon. <FIG> illustrates an example augmented reality display enabled using augmented reality glasses.

<FIG> illustrate flowcharts containing a series of operations performed by example embodiments described herein for utilizing augmented reality visualization to assist surgery. The operations shown in <FIG> are performed in an example embodiment using a computing device <NUM> that may be embodied by or otherwise associated with processing circuitry <NUM> (which may include a processor <NUM> and a memory <NUM>), a head-mounted display <NUM>, and in some embodiments, a separate user interface <NUM> and scanner <NUM>.

Turning first to <FIG>, a flow diagram illustrates example operations for utilizing three dimensional augmented reality visualizations to assist surgery.

In optional operation <NUM>, the computing device <NUM> may include means, such as scanner <NUM> or the like, for scanning a target area to generate an image stack representing the target area. As described previously, the scanner <NUM> may be a tomograph or any other device that can use penetrating waves to capture image data regarding an object. For instance, scanner <NUM> may comprise a computed tomography (CT) scanner, a positronic emission tomography (PET) scanner, a single-photon emission computed tomography (SPECT) scanner, or a magnetic resonance imaging (MRI) scanner. The scanner <NUM> is configured to produce a tomogram, which constitutes a two dimensional image illustrating a slice, or section, of the imaged object. In a typical scenario, however, the scanner <NUM> may produces a series of tomograms (referred to herein as an image stack) that, when analyzed in combination, can be used to generate a three dimensional mesh representative of the scanned object.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for generating a three dimensional reconstruction of an image stack representing a target area. The image stack may be received from a scanner <NUM> or from a local memory (e.g., memory <NUM>) or from a remote memory via a communication interface (not shown in <FIG>). Generation of the 3D reconstruction is described in greater detail below in conjunction with <FIG>.

In operation <NUM>, a head-mounted display <NUM> (or other similar wearable device), superimposes a 3D projection of the 3D reconstruction onto the field of view of the user (e.g., surgeon). The head-mounted display <NUM> may in some embodiments be an element of the computing device <NUM>, although in other embodiments, the head-mounted display <NUM> may be a separate apparatus connected to the computing device <NUM> via wireless communication media, as described previously. Specific aspects of superimposing the 3D projection are described in greater detail below in connection with <FIG>.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM>, head-mounted display <NUM>, user interface <NUM>, or the like, for maintaining alignment between the projection and the user's actual view of the target area using a plurality of fiducial markers associated with the target area. In this regard, these fiducial markers are deployed into the environment in advance of surgery and provide reference points that enable accurate relative positioning of the surgeon, surgical tools, and provide a better understanding of the location, orientation, and swelling of target areas. To this end, a subset of the fiducial markers may be pre-existing within the operating room environment while another subset of the fiducial markers may be inserted or otherwise disposed on different portions of the patient. In the latter respect, a subset of the plurality of fiducial markers may be located directly on the target area that is the subject of surgery (e.g., the brain), or on related tissue, such as on secondary target areas whose movement and swelling are intended to be tracked.

In addition to placing these fiducial markers in the operatory environment, the location of these markers must also be digitally placed into the 3D model during the preprocessing step so that the physical locations exactly match the corresponding positions in the virtual world. The camera in the augmented reality glasses can then recognize the fiducial markers in the real world, and match them with their digital representation. This will allow superimposing the real and the virtual world correctly. At least one of these fiducial markers should be placed directly on the organ of interest (i.e. brain). Because the organ can move and swell, this marker will be moving and follow the deformation of the organ, allowing the system to compute a deformation that can be applied to the 3D mesh loaded into the system. In this latter regard, specific operations for maintaining alignment of the 3D projection and the user's actual view of the target area are described in greater detail below in connection with <FIG>.

In optional operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM>, communication interface <NUM>, or the like, for receiving relative position and orientation information regarding a surgical tool. To this end, assistive machinery (e.g., a neuronavigator or the like) utilizes surgical tools that are not directly visible during operations. Such assistive machinery generally utilizes its own tracking mechanisms to identify relative locations and orientations of the surgical tools and which can accordingly provide the spatial information of the position of the surgical tools inside that are not visible during a surgical procedure.

Example embodiments can therefore interface with the existing neuronavigation technologies that already provide feedbacks about the position of surgical tools to visualize their respective location. In this fashion, optional operation <NUM> enables the retrieval of this location and orientation from the assistive machinery via communication interface <NUM> or from a local memory (e.g., memory <NUM> of the processing circuitry <NUM>).

Finally, in optional operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM>, head-mounted display <NUM>, or the like, for superimposing a projection of the surgical tool onto the field of view of the user. Having received the location and orientation of surgical tools from the assistive machinery, the surgical tools can be projected onto the field of view of the surgeon in the same manner as described above in connection with operation <NUM> and below in connection with <FIG>.

Turning now to <FIG>, example operations are illustrated for generating a three dimensional reconstruction of a target area based on an image stack received from a scan of the target area, in accordance with some example embodiments of the present invention.

From optional operation <NUM> above (in embodiments in which operation <NUM> is performed) or in response to retrieval of an image stack from a memory store, the procedure may turn to operation <NUM>, in which the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for pre-processing the image stack. In this regard, pre-processing the image stack may in some embodiments include aligning images of the image stack. In addition, pre-processing the image stack may in some embodiments include filtering images of the image stack to increase the SNR (signal-to-noise ratio) of the images.

Subsequently, in operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for performing image segmentation on all images of the image stack to identify structures within the image stack. In this regard, image segmentation may utilize a seeded watershed algorithm. The algorithm can detect the edges of each image to be segmented based on that image's gradient, then afterwards each edge can be followed on the <NUM>rd dimension (z) by following the spatial direction of the edge on the sequence of images. The procedure may, in some embodiments, be iterated for the both a target area (e.g., an organ) and a lesion (previously recognized by the physician) that appears within the reconstructed organ, removal of which may be the primary purpose of the surgical procedure.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for generating a 3D mesh defining boundaries of the identified structures. This 3D mesh may comprise a collection of points and surfaces in a 3D space that approximate the shape of the segmented structure. In this regard, the 3D reconstruction discussed above in connection with <FIG> may comprise this 3D mesh.

Scanner <NUM> typically provides information about pixel size and z thickness of an image stack. Thus, by reference to this information, the relative thickness of the images in the image stack and the XY resolution of the images of the image stack can be evaluated to determine if the generated 3D mesh includes isotropic voxels, in which case the 3D mesh can be utilized to present a 3D projection to the user. However, when the voxel size of the 3D mesh is not isotropic, a common result is that the Z dimension of the 3D mesh may be compressed or elongated.

In such circumstances, optional operation <NUM> may be performed, in which the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for rescaling the three dimensional mesh to render its voxels isotropic (correcting the compression or elongation caused by the asymmetry between the Z axis thickness and the XY-axis resolution of the images in the image stack.

Turning now to <FIG>, example operations are illustrated for superimposing a three dimensional augmented reality projection using augmented reality glasses, in accordance with some example embodiments of the present invention. It should be understood that while the operations of <FIG> illustrate one example by which a 3D projection can be displayed, other embodiments may be contemplated, such as those discussed above.

In operation <NUM>, the computing device <NUM> includes means, such as head-mounted display <NUM> or the like, for displaying a first projection of the three dimensional reconstruction to one eye of the user.

In operation <NUM>, the computing device <NUM> includes means, such as head-mounted display <NUM> or the like, for displaying a second projection of the three dimensional reconstruction to the other eye of the user. In this regard, it should be understood that the differences between the first projection and the second projection are designed to generate the three dimensional projection of the three dimensional reconstruction.

The 3D augmented reality projection may change over time with movement of the head-mounted display. In this regard, it should be understood that in order for the model to be superimposed correctly to the target area and to change perspective according to the relative position of the surgeon with respect to the target area, the augmented reality glasses utilize a front camera configured to recognize fiducial markers placed in the operating room that the computing device <NUM> can recognize as fixed points in the environment whose position relative to the patient is known. This enables correct alignment of the 3D model, as well as the tracking of the surgeon's head position. Similarly, tracking of the augmented reality glasses can be improved using sensors disposed on the augmented reality glasses (e.g., gyroscope(s), accelerometer(s), or the like) embedded in the augmented reality glasses.

Accordingly, turning now to <FIG>, example operations are illustrated for ensuring accurate alignment of a three dimensional projection based on relative locations and orientations of objects within a user's field of view, in accordance with some example embodiments of the present invention.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for identifying, by a camera associated with the head-mounted display, a location of each of the plurality of fiducial markers.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for calculating a relative location of the target area from the perspective of the head-mounted display.

In operation <NUM>, the computing device <NUM> includes means, such as head-mounted display <NUM> or the like, for generating the projection of the three dimensional reconstruction based on the calculated relative location of the target area.

In optional operation <NUM>, the computing device <NUM> includes means, such as head-mounted display <NUM>, user interface <NUM>, or the like, for presenting an interface requesting user adjustment of the alignment between the projection and the user's actual view of the target area.

In optional operation <NUM>, the computing device <NUM> includes means, such as head-mounted display <NUM>, user interface <NUM>, or the like, for receiving one or more responsive adjustments. In this regard, the use of the fiducial markers should be as accurate as possible, in order to grant a perfect tracing of the position of the 3D model relative to the real world. Nevertheless, because the model will be superimposed on the eyes of the surgeon, the surgeon may be provided with the ability to finely adjust the position of the model by using a graphical interface, to correct millimetric adjusting errors (on the x, y, or z axes).

Thus, in optional operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM>, head-mounted display <NUM>, or the like, for updating the projection of the three dimensional reconstruction in response to receiving the one or more responsive adjustments. These adjustments may be made u.

In addition to the operations described above in connection with <FIG>, maintenance of the alignment of the projection can also take into account fiducial markers located in or near the target area can be examined to enable the model to follow the natural displacements and deformation of the brain itself. The 3D projection, which is based on the 3D mesh, can thus follow the deformation and modify its volume according to the actual movement of the brain. In some example embodiments, this spatial tracking can be achieved utilizing an existing SDK (software development kit), provided by Vuforia, wikitude, or Metaio. Those SDKs are able to track the spatial position and orientation of the camera in the augmented reality glasses relative to the known fiducial markers.

Turning now to <FIG>, example operations are illustrated for maintaining alignment of a three dimensional projection based on changes in objects within the field of view, in accordance with some example embodiments of the present invention.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for detecting a change in location or orientation of at least two fiducial markers disposed on the target area, wherein movement of the at least two fiducial markers disposed on the target area indicates a change in location or shape of the target area.

In operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for computing a deformation of the three dimensional reconstruction based on the detected change. Subsequently, in operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for applying the deformation to the three dimensional reconstruction. Finally, in operation <NUM>, the computing device <NUM> includes means, such as processing circuitry <NUM> or the like, for updating the projection of the three dimensional reconstruction in response to applying the deformation to the three dimensional reconstruction.

In addition, it should be understood that this alignment can be improved through the use of input coming from the neuronavigation system, which provides an additional spatial reference to the target area with respect to the surgical tools utilized during surgery.

Accordingly, the operations illustrated in <FIG> provide a procedure that utilizes augmented reality visualization to assist surgery. In particular, utilizing augmented reality 3D glasses, example embodiments deploy 3D tracking and 3D visualization of the surgical field in a manner that allows the surgeon to focus attention (sight and hands) on the operatory field and on the patient rather than on separate screen. Similarly, using example embodiments, the surgeon is not required to interpolate movements that appear on an external display in one orientation on the fly to properly respond with movements in the actual working area. Accordingly, example embodiments described herein provide significant benefits for surgical applications having little or no direct visibility and surgical applications utilizing assistive machinery.

The above-described flowcharts illustrate operations performed by an apparatus (which include the hardware elements of computing device <NUM> of <FIG>), in accordance with some example embodiments of the present invention. It will be understood that each block of the flowchart, and combinations of blocks in the flowchart, may be implemented by various means, such as hardware, firmware, processor, circuitry and/or other device associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by program code instructions. In this regard, the program code instructions which embody the procedures described above may be stored by a memory <NUM> of the computing device <NUM> employing an embodiment of the present invention and executed by a processor <NUM> of the computing device <NUM>. As will be appreciated, loading these program code instructions onto a computing device <NUM> produces a specially programmed apparatus configured to implement the functions specified in the respective flowchart blocks. The program code instructions may also be stored in a non-transitory computer-readable storage medium that may direct a computer or other programmable apparatus to function in the prescribed manner, such that storing the program code instructions in the computer-readable storage memory produce an article of manufacture which can be accessed by an computing device <NUM> to implement the functions specified in the flowchart blocks. Accordingly, the operations illustrated in the flowchart define algorithms for configuring a computer or processing circuitry <NUM> (e.g., a processor) to perform example embodiments described above. When a general purpose computer stores the algorithms illustrated above, the general purpose computer is transformed into a particular machine configured to perform the corresponding functions.

Blocks of the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions. It will be understood that one or more blocks of the flowchart, and combinations of blocks in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.

Claim 1:
A system for utilizing augmented reality visualization in surgery, the system comprising:
a plurality of fiducial markers comprising:
a first subset of the plurality of fiducial markers configured to be placed in an operating room as fixed points whose position relative to a patient is known, and
a second subset of the plurality of fiducial markers configured to be placed on a target area of a patient;
an apparatus (<NUM>) configured to generate (<NUM>) a three dimensional reconstruction of an image stack representing the target area of the patient by
performing (<NUM>) image segmentation on all images of the image stack to identify structures within the image stack, and
generating (<NUM>) a three dimensional mesh defining boundaries of the identified structures, wherein
the three dimensional reconstruction comprises the three dimensional mesh, wherein, when the fiducial markers have been placed, the apparatus is additionally configured to digitally place the corresponding locations of a digital representation of the plurality of fiducial markers into the three dimensional reconstruction of the image stack;
a head-mounted display (<NUM>) configured to superimpose (<NUM>) a three dimensional projection of the three dimensional reconstruction onto a field of view of a user; and
a camera in the head-mounted display configured to identify (<NUM>) a location of each of the plurality of fiducial markers, and to match the plurality of fiducial markers with their digital representation, so that the apparatus is configured to maintain (<NUM>) alignment between the projection and the user's actual view of the target area using the plurality of fiducial markers,
wherein the camera is configured to detect (<NUM>) a change in the location of the second subset of the plurality of fiducial markers indicating a change in location or
shape of the target area, so that the apparatus is configured to compute (<NUM>) a deformation of the three dimensional reconstruction, and the head-mounted display is configured to update (<NUM>) the projection of the three dimensional reconstruction.