Patent Description:
CT or MRI scans, which are inherently 3D, are often visualized as a set of three orthogonal slices along the anatomical axes of the patient or along the axis of an instrument. Only recently have 3D images been rendered on displays in operating rooms, often in addition to the slice visualizations. These are often volume renderings from viewpoints, which can be defined and controlled by the user. Many surgeons prefer 2D images rather than 3D graphic renderings as they have been well trained in interpreting them and thus this defines the current clinical standard.

Navigation in surgery has been introduced since the early <NUM> and takes advantage of different visualization techniques. Many systems show slices of pre- or intraoperative data with additional annotations. These annotations relate to a surgical plan, which needs to be carried out in surgery. Such annotations are often an attempt to visualize deep seated anatomical targets and safe paths for the surgeon to reach them based on pre-operative images. In such systems, when rendering images, the position of the surgeon and location of displays are not taken into consideration. The position of the display therefore does not affect the visualization nor does the position of observer.

A series of related work is often referred to as Augmented Reality windows (AR-windows). Prior techniques tracked semi-transparent display for medical in-situ augmentation, which incorporated a head tracker and stereo glasses. Such semi-transparent displays included a half-silvered glass pane, which reflected the image from a computer display. Others have addressed the same problem of creating an AR-window on patient anatomy using a semi-transparent display between the patient and the surgeon. However, such prior techniques replaced the half-silvered glass pane with an active matrix LCD. Later attempts approached the problem again with a half-silvered glass pane, which rejected the projection of two DLP projectors generating high contrast images. Other systems comprised of a tracked mobile opaque screen in which the position of the screen affected the visualization. Once again, such prior systems were placed between the surgeon and the patient showing a slice view of the anatomy. However, in this system, the user's perspective is not considered, i.e. the image on the screen is merely two-dimensional, independent from the viewpoint of the surgeon.

Others have presented an AR visualization inspired by the dentists' approach for examining the patient's mouth without changing their viewpoints. Such techniques identified that in some AR applications, rotating the object or moving around it is impossible. It was suggested to generate additional virtual mirroring views to offer secondary perspectives on virtual objects within an AR view. Spatially tracked joysticks were utilized to move a virtual mirror which reflected the virtual data like a real mirror within the AR view of a head mounted device (HMD).

<CIT> discloses methods and systems for performing computer-assisted image-guided surgery, including robotically-assisted surgery. A method of displaying image data includes displaying image data of a patient on a handheld display device, tracking the handheld display device using a motion tracking system, and modifying the image data displayed in response to changes in the position and orientation of the handheld display device. Further embodiments include a sterile case for a handheld display device, display devices on a robotic arm, and methods and systems for performing image-guided surgery using multiple reference marker devices fixed to a patient.

This Summary introduces a selection of concepts in a simplified form that are further described below in the Detailed Description below. This Summary is not intended to limit the scope of the claimed subject matter and does not necessarily identify each and every key or essential feature of the claimed subject matter.

In a first aspect, a system for aiding in interaction with a physical object is provided, the system comprising: a display device defining a plane and being located on a first side of the physical object; and a navigation system coupled to a control system and being configured to: register computer images to the physical object; track a pose of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera that has a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field-of-view that faces the plane of display device, and wherein the virtual position of the virtual camera updates automatically in response to adjustment to the pose of the display device.

In a second aspect, a navigation system of the system of the first aspect is provided.

In a third aspect, a method of operating a system for aiding in interaction with a physical object is provided, the system comprising: a display device defining a plane and being located on a first side of the physical object; and a navigation system coupled to a control system, the method comprising: registering computer images to the physical object; tracking a pose of the physical object and the display device in a common coordinate system; and controlling the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera having a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field-of-view facing the plane of display device, and automatically updating the virtual position of the virtual camera in response to adjusting the pose of the display device.

In a fourth aspect, a computer program product is provided for aiding in interaction with a physical object, the computer program product usable with a system comprising a display device defining a plane and being located on a first side of the physical object, and a navigation system coupled to a control system, the computer program product comprising instructions, which when executed by one or more processors, are configured to: register computer images to the physical object; track a pose of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera that has a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field-of-view that faces the plane of display device, and wherein the virtual position of the virtual camera updates automatically in response to adjustment to the pose of the display device.

In a fifth aspect, a system for aiding in interaction with a physical object is provided, the system comprising: a display device defining a plane, wherein the physical object is located in front of the plane; a navigation system coupled to a control system and being configured to: register computer images to the physical object; track a pose of the physical object, the display device, and a user's viewpoint in a common coordinate system; and control the display device for rendering the registered computer images according to the tracked pose of the physical object, the display device, and the user's viewpoint; and wherein a perspective of the rendering is based on a field-of-view of a virtual camera that has a virtual position behind the plane, and wherein the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the user's viewpoint.

In a sixth aspect, a navigation system of the system of the fifth aspect is provided.

In a seventh aspect, a method of operating a system for aiding in interaction with a physical object is provided, the system comprising a display device defining a plane, wherein the physical object is located in front of the plane, and a navigation system coupled to a control system, the method comprising: registering computer images to the physical object; tracking a pose of the physical object, the display device, and a user's viewpoint in a common coordinate system; and controlling the display device for rendering the registered computer images according to the tracked pose of the physical object, the display device, and the user's viewpoint; and wherein a perspective of the rendering is based on a field-of-view of a virtual camera having a virtual position behind the plane, and automatically updating the virtual position of the virtual camera in response to movement of the tracked pose of the user's viewpoint.

In an eighth aspect, a computer program product is provided for aiding in interaction with a physical object, the computer program product usable with a system comprising a display device defining a plane, wherein the physical object is located in front of the plane, and a navigation system coupled to a control system, the computer program product comprising instructions, which when executed by one or more processors, are configured to: register computer images to the physical object; track a pose of the physical object, the display device, and a user's viewpoint in a common coordinate system; and control the display device for rendering the registered computer images according to the tracked pose of the physical object, the display device, and the user's viewpoint; and wherein a perspective of the rendering is based on a field-of-view of a virtual camera that has a virtual position behind the plane, and wherein the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the user's viewpoint.

In a ninth aspect, a system, method, or computer program product is provided for aiding in interaction with a physical object, comprising: a display device defining a plane; and a navigation system coupled to a control system and being configured to: register computer images to the physical object; track a pose of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer images according to the tracked pose of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera that has a field-of-view that faces the plane of display device, and wherein the virtual position of the virtual camera updates automatically in response to adjustment to the pose of the display device.

In a tenth aspect, a system, method, or computer program product is provided for aiding in interaction with a physical object, comprising: a display device defining a plane; a navigation system coupled to a control system and being configured to: register computer images to the physical object; track a pose of the physical object, the display device, and a user's viewpoint in a common coordinate system; and control the display device for rendering the registered computer images according to the tracked pose of the physical object, the display device, and the user's viewpoint; and wherein a perspective of the rendering is based on a field-of-view of a virtual camera, wherein the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the user's viewpoint.

Any of the aspects above can be combined in part, or in whole.

Any of the aspects above can be combined with any of the following implementations, whether individually or in combination:.

In one implementation, the computer images of the physical object are derived from a 3D model. In one implementation, the control system is configured to control the display device to display one or more slices of the 3D model. In one implementation, the slices are displayed depending upon the tracked pose of the physical object and/or the display device. In one implementation, the one or more slices are sliced at planes aligned with the plane of the display device. In one implementation, the planes are parallel to the display device. In one implementation, the control system is configured to control the display device to automatically change the one or more slices to other slices in response to at least one or more of the following: the tracked pose of the display device; the tracked pose of the physical object; a tracked pose of a surgical instrument; and/or a tracked pose of a user's viewpoint.

In one implementation, the virtual position of the virtual camera is located at a predetermined distance from the plane of display device. In one implementation, the predetermined distance updates automatically. In one implementation, the predetermined distance updates automatically in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user's viewpoint; a type of surgical procedure; and/or a certain step of a surgical procedure. In one implementation, the predetermined distance is manually updated in response to user input. In one implementation, the predetermined distance is fixed.

In one implementation, an X-Y placement of the virtual position of the virtual camera relative to the plane of the display device updates automatically. In one implementation, the X-Y placement updates automatically. In one implementation, the X-Y placement updates automatically in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user's viewpoint; a type of surgical procedure; and a certain step of a surgical procedure. In one implementation, an X-Y placement of the virtual position of the virtual camera relative to the plane of the display device is manually adjustable based on user input. In one implementation, an X-Y placement of the virtual position of the virtual camera relative to the plane of the display device is fixed.

In one implementation, the pose of the display device is manually adjustable. In one implementation, the virtual position of the virtual camera updates automatically in response to manual adjustment to the pose of the display device. In one implementation, one or more actuators are coupled to the display device. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device. In one implementation, an input device is coupled to the control system and the control system is configured to receive a command from the input device and control the one or more actuators to adjust the pose of the display device in accordance with the command. In one implementation, a surgical instrument includes one or more trackable features, and the navigation system is configured to track a pose of the surgical instrument in the common coordinate system and control the display device to display an image of the surgical instrument. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument. In one implementation, a viewpoint tracking system is coupled to the navigation system and configured to track a pose user's viewpoint in the common coordinate system. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the user's viewpoint.

In one implementation, a line defined between the virtual position of the virtual camera and the plane of the display device. In one implementation, the line is transverse to the plane of the display device. In one implementation, the line is orthogonal to the plane of the display device. In one implementation, the line strikes a geometrical center of the display device. In one implementation, the line strikes at a location offset from a geometrical center of the display device.

In one implementation, the perspective of the rendering is automatically rotated to align with the tracked pose of the user's viewpoint relative to the plane of the display device. In one implementation, the rotation of the rendering is computed according to the described projection matrix. In one implementation, the perspective of the rendering is computed according to the following: MirrorTWorld · Mflip ·WorldTMirror ·p.

In one implementation, the field-of-view of the virtual camera is automatically updated in response to movement of the tracked pose of the user's viewpoint. In one implementation, the display device comprises fixed features and wherein the field-of-view of the virtual camera comprises boundary features. In one implementation, the control system is configured to fit the boundary features to coincide with the fixed features. In one implementation, the control system fits the same for any given virtual position of the virtual camera that is automatically updated in response to movement of the tracked pose of the user's viewpoint.

In one implementation, the common coordinate system comprises an X, Y and Z axis, and wherein the virtual position of the virtual camera and the pose of the user's viewpoint are equidistant from the plane of the display device relative to each of the X, Y and Z axes.

In one implementation, the virtual position of the virtual camera is automatically updated to move towards the plane of the display device in response to movement of the tracked pose of the user's viewpoint towards the plane of the display device. In one implementation, the virtual position of the virtual camera is automatically updated to move away from the plane of the display device in response to movement of the tracked pose of the user's viewpoint away the plane of the display device.

In one implementation, the control system renders the computer images to increase in size in response to movement of the tracked pose of the user's viewpoint towards the plane of the display device. In one implementation, the control system renders the computer images to decrease in size in response to movement of the tracked pose of the user's viewpoint away from the plane of the display device.

In one implementation, the computer images of the physical object are derived from a 3D model. In one implementation, the control system is configured to control the display device to display one or more slices of the 3D model. In one implementation, the control system does so depending on the tracked pose of the physical object, the display device, and/or the user's viewpoint.

In one implementation, a perspective of the one or more slices is rendered based on the field-of-view of the virtual camera that has the virtual position automatically updated in response to movement of the tracked pose of the user's viewpoint.

In one implementation, the control system is configured to control the display device to automatically change the one or more slices to other slices. In one implementation, the control system does so in response to one or more of the following: the tracked pose of the user's viewpoint; the tracked pose of the display device; the tracked pose of the physical object; and/or a tracked pose of a surgical instrument.

In one implementation, the navigation system comprises a head-mounted device worn by the user. In one implementation, the navigation system is configured to track the pose of the user's viewpoint by being configured to track a pose of the head-mounted device. In one implementation, the navigation system comprises a camera. In one implementation, the camera is configured to be directed towards the user. In one implementation, the navigation system is configured to track the pose of the user's viewpoint by being configured to track a pose of the user's head, face, or eyes using the camera. In one implementation, the camera is mounted to the display device. In one implementation, the camera to another device.

Referring to <FIG>, a surgical system <NUM> is illustrated which can be utilized with the spatially-aware displays described in Section II below. The system <NUM> is useful for treating a target site or anatomical volume A of a patient <NUM>, such as treating bone or soft tissue. In <FIG>, the patient <NUM> is undergoing a surgical procedure. The anatomy in <FIG> includes a femur F, pelvis PEL, and a tibia T of the patient <NUM>. The surgical procedure may involve tissue removal or other forms of treatment. Treatment may include cutting, coagulating, lesioning the tissue, other in-situ tissue treatments, or the like. In some examples, the surgical procedure involves partial or total knee or hip replacement surgery, shoulder replacement surgery, spine surgery, or ankle surgery. In some examples, the system <NUM> is designed to cut away material to be replaced by surgical implants, such as hip and knee implants, including unicompartmental, bicompartmental, multicompartmental, or total knee implants, acetabular cup implants, femur stem implants, screws, anchors, other fasteners, and the like. Some of these types of implants are shown in <CIT>, entitled, "Prosthetic Implant and Method of Implantation," the disclosure of which is hereby referenced. The system <NUM> and techniques disclosed herein may be used to perform other procedures, surgical or non-surgical, or may be used in industrial applications or other applications.

The system <NUM> may include a robotic manipulator <NUM>, also referred to as a surgical robot. The manipulator <NUM> has a base <NUM> and plurality of links <NUM>. A manipulator cart <NUM> supports the manipulator <NUM> such that the manipulator <NUM> is fixed to the manipulator cart <NUM>. The links <NUM> collectively form one or more arms of the manipulator <NUM> (e.g., robotic arms). The manipulator <NUM> may have a serial arm configuration (as shown in <FIG>), a parallel arm configuration, or any other suitable manipulator configuration. In other examples, more than one manipulator <NUM> may be utilized in a multiple arm configuration.

In the example shown in <FIG>, the manipulator <NUM> comprises a plurality of joints J and a plurality of joint encoders <NUM> located at the joints J for determining position data of the joints J. For simplicity, only one joint encoder <NUM> is illustrated in <FIG>, although other joint encoders <NUM> may be similarly illustrated. The manipulator <NUM> according to one example has six joints J1-J6 implementing at least six-degrees of freedom (DOF) for the manipulator <NUM>. However, the manipulator <NUM> may have any number of degrees of freedom and may have any suitable number of joints J and may have redundant joints.

The manipulator <NUM> need not require joint encoders <NUM> but may alternatively, or additionally, utilize motor encoders present on motors at more than one or each joint J. Also, the manipulator <NUM> need not require rotary joints, but may alternatively, or additionally, utilize one or more prismatic joints. Any suitable combination of joint types is contemplated.

The base <NUM> of the manipulator <NUM> is generally a portion of the manipulator <NUM> that provides a fixed reference coordinate system for other components of the manipulator <NUM> or the system <NUM> in general. Generally, the origin of a manipulator coordinate system MNPL is defined at the fixed reference of the base <NUM>. The base <NUM> may be defined with respect to any suitable portion of the manipulator <NUM>, such as one or more of the links <NUM>. Alternatively, or additionally, the base <NUM> may be defined with respect to the manipulator cart <NUM>, such as where the manipulator <NUM> is physically attached to the cart <NUM>. In one example, the base <NUM> is defined at an intersection of the axes of joints J1 and J2. Thus, although joints J1 and J2 are moving components in reality, the intersection of the axes of joints J1 and J2 is nevertheless a virtual fixed reference pose, which provides both a fixed position and orientation reference and which does not move relative to the manipulator <NUM> and/or manipulator cart <NUM>.

In some examples, the manipulator <NUM> can be a hand-held manipulator where the base <NUM> is a base portion of a tool (e.g., a portion held free-hand by the user) and the tool tip is movable relative to the base portion. The base portion has a reference coordinate system that is tracked and the tool tip has a tool tip coordinate system that is computed relative to the reference coordinate system (e.g., via motor and/or joint encoders and forward kinematic calculations). Movement of the tool tip can be controlled to follow the path since its pose relative to the path can be determined. Such a manipulator <NUM> is shown in <CIT>, entitled, "Surgical Instrument Including Housing, A Cutting Accessory that Extends from the Housing and Actuators that Establish the Position of the Cutting Accessory Relative to the Housing".

The manipulator <NUM> and/or manipulator cart <NUM> house a manipulator controller <NUM>, or other type of control unit. The manipulator controller <NUM> may comprise one or more computers, or any other suitable form of controller that directs the motion of the manipulator <NUM>. The manipulator controller <NUM> may have a central processing unit (CPU) and/or other processors, memory (not shown), and storage (not shown). The manipulator controller <NUM> is loaded with software as described below. The processors could include one or more processors to control operation of the manipulator <NUM>. The processors can be any type of microprocessor, multi-processor, and/or multi-core processing system. The manipulator controller <NUM> may additionally, or alternatively, comprise one or more microcontrollers, field programmable gate arrays, systems on a chip, discrete circuitry, and/or other suitable hardware, software, or firmware that is capable of carrying out the functions described herein. The term processor is not intended to limit any implementation to a single processor. The manipulator <NUM> may also comprise a user interface UI with one or more displays and/or input devices (e.g., push buttons, sensors, switches, keyboard, mouse, microphone (voice-activation), gesture control devices, touchscreens, joysticks, foot pedals, etc.).

A surgical tool <NUM> couples to the manipulator <NUM> and is movable relative to the base <NUM> to interact with the anatomy in certain modes. The tool <NUM> is or forms part of an end effector <NUM> supported by the manipulator <NUM> in certain embodiments. The tool <NUM> may be grasped by the user. One possible arrangement of the manipulator <NUM> and the tool <NUM> is described in <CIT>, entitled, "Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes". The manipulator <NUM> and the tool <NUM> may be arranged in alternative configurations. The tool <NUM> can be like that shown in <CIT>, entitled, "End Effector of a Surgical Robotic Manipulator". Separate hand-held surgical tools can be utilized in addition to or alternative to the manipulator <NUM> and tool <NUM>.

The tool <NUM> includes an energy applicator <NUM> designed to contact the tissue of the patient <NUM> at the target site. In one example, the energy applicator <NUM> is a bur <NUM>. The bur <NUM> may be spherical and comprise a spherical center, radius (r) and diameter. Alternatively, the energy applicator <NUM> may be a drill bit, a saw blade <NUM> (see alternative tool in <FIG>), an ultrasonic vibrating tip, or the like. The tool <NUM> and/or energy applicator <NUM> may comprise any geometric feature, e.g., perimeter, circumference, radius, diameter, width, length, volume, area, surface/plane, range of motion envelope (along any one or more axes), etc. The geometric feature may be considered to determine how to locate the tool <NUM> relative to the tissue at the target site to perform the desired treatment. In some of the embodiments described herein, a spherical bur having a tool center point (TCP) and a sagittal saw blade having a TCP will be described for convenience and ease of illustration, but is not intended to limit the tool <NUM> to any particular form.

The tool <NUM> may comprise a tool controller <NUM> to control operation of the tool <NUM>, such as to control power to the tool <NUM> (e.g., to a tool drive such as a rotary motor of the tool <NUM>), control movement of the tool <NUM>, control irrigation/aspiration of the tool <NUM>, and/or the like. The tool controller <NUM> may be in communication with the manipulator controller <NUM> or other components. The tool <NUM> may also comprise a user interface UI with one or more displays and/or input devices (e.g., push buttons, triggers, sensors, switches, keyboard, mouse, microphone (voice-activation), gesture control devices, touchscreens, joysticks, foot pedals, etc.) that are coupled to the tool controller <NUM>, manipulator controller <NUM>, and/or other controllers described herein. The manipulator controller <NUM> controls a state (e.g., position and/or orientation) of the tool <NUM> (e.g., of the TCP) with respect to a coordinate system, such as the manipulator coordinate system MNPL. The manipulator controller <NUM> can control velocity (linear or angular), acceleration, or other derivatives of motion of the tool <NUM>.

The tool center point (TCP), in one example, is a predetermined reference point defined at the energy applicator <NUM>. The TCP has a known, or able to be calculated (i.e., not necessarily static), pose relative to other coordinate systems. The geometry of the energy applicator <NUM> is known in or defined relative to a TCP coordinate system. The TCP may be located at the spherical center of the bur <NUM> of the tool <NUM> or at the distal end of the saw blade <NUM> such that only one point is tracked. The TCP may be defined in various ways depending on the configuration of the energy applicator <NUM>. The manipulator <NUM> could employ the joint/motor encoders, or any other non-encoder position sensing method, to enable a pose of the TCP to be determined. The manipulator <NUM> may use joint measurements to determine TCP pose and/or could employ techniques to measure TCP pose directly. The control of the tool <NUM> is not limited to a center point. For example, any suitable primitives, meshes, etc., can be used to represent the tool <NUM>.

The system <NUM> further includes a navigation system <NUM>. One example of the navigation system <NUM> is described in <CIT>, entitled, "Navigation System Including Optical and Non-Optical Sensors". The navigation system <NUM> tracks movement of various objects. Such objects include, for example, the manipulator <NUM>, the tool <NUM> and the anatomy, e.g., femur F, pelvis PEL, and tibia T. The navigation system <NUM> tracks these objects to gather state information of the objects with respect to a (navigation) localizer coordinate system LCLZ. Coordinates in the localizer coordinate system LCLZ may be transformed to the manipulator coordinate system MNPL, to other coordinate systems, and/or vice-versa, using transformations.

The navigation system <NUM> may include a cart assembly <NUM> that houses a navigation controller <NUM>, and/or other types of control units. A navigation user interface UI is in operative communication with the navigation controller <NUM>. The navigation user interface includes one or more displays <NUM>. The navigation system <NUM> is capable of displaying a graphical representation of the relative states of the tracked objects to the user using the one or more displays <NUM>. The navigation user interface UI further comprises one or more input devices to input information into the navigation controller <NUM> or otherwise to select/control certain aspects of the navigation controller <NUM>. Such input devices include interactive touchscreen displays. However, the input devices may include any one or more of push buttons, a keyboard, a mouse, a microphone (voice-activation), gesture control devices, foot pedals, and the like.

The navigation system <NUM> also includes a navigation localizer <NUM> coupled to the navigation controller <NUM>. In one example, the localizer <NUM> is an optical localizer and includes a camera unit <NUM>. The camera unit <NUM> has an outer casing <NUM> that houses one or more optical sensors <NUM>. The localizer <NUM> may comprise its own localizer controller <NUM> and may further comprise a video camera VC.

The navigation system <NUM> includes one or more trackers. In one example, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker <NUM>, a second patient tracker <NUM>, and a third patient tracker <NUM>. In the illustrated example of <FIG>, the manipulator tracker is firmly attached to the tool <NUM> (i.e., tracker 52A), the first patient tracker <NUM> is firmly affixed to the femur F of the patient <NUM>, the second patient tracker <NUM> is firmly affixed to the pelvis PEL of the patient <NUM>, and the third patient tracker <NUM> is firmly affixed to the tibia T of the patient <NUM>. In this example, the patient trackers <NUM>, <NUM>, <NUM> are firmly affixed to sections of bone. The pointer tracker PT is firmly affixed to a pointer P used for registering the anatomy to the localizer coordinate system LCLZ. The manipulator tracker 52A, 52B may be affixed to any suitable component of the manipulator <NUM>, in addition to, or other than the tool <NUM>, such as the base <NUM> (i.e., tracker 52B), or any one or more links <NUM> of the manipulator <NUM>. The trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT may be fixed to their respective components in any suitable manner. For example, the trackers may be rigidly fixed, flexibly connected (optical fiber), or physically spaced (e.g., ultrasound), as long as there is a suitable (supplemental) way to determine the relationship (measurement) of that respective tracker to the object with which it is associated.

Any one or more of the trackers may include active markers <NUM>. The active markers <NUM> may include light emitting diodes (LEDs). Alternatively, the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT may have passive markers, such as reflectors, which reflect light emitted from the camera unit <NUM>. Other suitable markers not specifically described herein may be utilized.

The localizer <NUM> tracks the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT to determine a state of the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT, which correspond respectively to the state of the object respectively attached thereto. The localizer <NUM> may perform known triangulation techniques to determine the states of the trackers <NUM>, <NUM>, <NUM>, <NUM>, PT, and associated objects. The localizer <NUM> provides the state of the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT to the navigation controller <NUM>. In one example, the navigation controller <NUM> determines and communicates the state the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT to the manipulator controller <NUM>. As used herein, the state of an object includes, but is not limited to, data that defines the position and/or orientation of the tracked object or equivalents/derivatives of the position and/or orientation. For example, the state may be a pose of the object, and may include linear velocity data, and/or angular velocity data, and the like.

The navigation controller <NUM> may comprise one or more computers, or any other suitable form of controller. Navigation controller <NUM> has a central processing unit (CPU) and/or other processors, memory (not shown), and storage (not shown). The processors can be any type of processor, microprocessor or multi-processor system. The navigation controller <NUM> is loaded with software. The software, for example, converts the signals received from the localizer <NUM> into data representative of the position and orientation of the objects being tracked. The navigation controller <NUM> may additionally, or alternatively, comprise one or more microcontrollers, field programmable gate arrays, systems on a chip, discrete circuitry, and/or other suitable hardware, software, or firmware that is capable of carrying out the functions described herein. The term processor is not intended to limit any implementation to a single processor.

Although one example of the navigation system <NUM> is shown that employs triangulation techniques to determine object states, the navigation system <NUM> may have any other suitable configuration for tracking the manipulator <NUM>, tool <NUM>, and/or the patient <NUM>. In another example, the navigation system <NUM> and/or localizer <NUM> are ultrasound-based. For example, the navigation system <NUM> may comprise an ultrasound imaging device coupled to the navigation controller <NUM>. The ultrasound imaging device images any of the aforementioned objects, e.g., the manipulator <NUM>, the tool <NUM>, and/or the patient <NUM>, and generates state signals to the navigation controller <NUM> based on the ultrasound images. The ultrasound images may be <NUM>-D, <NUM>-D, or a combination of both. The navigation controller <NUM> may process the images in near real-time to determine states of the objects. The ultrasound imaging device may have any suitable configuration and may be different than the camera unit <NUM> as shown in <FIG>.

In another example, the navigation system <NUM> and/or localizer <NUM> are radio frequency (RF)-based. For example, the navigation system <NUM> may comprise an RF transceiver coupled to the navigation controller <NUM>. The manipulator <NUM>, the tool <NUM>, and/or the patient <NUM> may comprise RF emitters or transponders attached thereto. The RF emitters or transponders may be passive or actively energized. The RF transceiver transmits an RF tracking signal and generates state signals to the navigation controller <NUM> based on RF signals received from the RF emitters. The navigation controller <NUM> may analyze the received RF signals to associate relative states thereto. The RF signals may be of any suitable frequency. The RF transceiver may be positioned at any suitable location to track the objects using RF signals effectively. Furthermore, the RF emitters or transponders may have any suitable structural configuration that may be much different than the trackers 52A, 52B, <NUM>, <NUM>, <NUM>, PT shown in <FIG>.

In yet another example, the navigation system <NUM> and/or localizer <NUM> are electromagnetically based. For example, the navigation system <NUM> may comprise an EM transceiver coupled to the navigation controller <NUM>. The manipulator <NUM>, the tool <NUM>, and/or the patient <NUM> may comprise EM components attached thereto, such as any suitable magnetic tracker, electro-magnetic tracker, inductive tracker, or the like. The trackers may be passive or actively energized. The EM transceiver generates an EM field and generates state signals to the navigation controller <NUM> based upon EM signals received from the trackers. The navigation controller <NUM> may analyze the received EM signals to associate relative states thereto. Again, such navigation system <NUM> examples may have structural configurations that are different than the navigation system <NUM> configuration shown in <FIG>.

The navigation system <NUM> may have any other suitable components or structure not specifically recited herein. Furthermore, any of the techniques, methods, and/or components described above with respect to the navigation system <NUM> shown may be implemented or provided for any of the other examples of the navigation system <NUM> described herein. For example, the navigation system <NUM> may utilize solely inertial tracking or any combination of tracking techniques, and may additionally or alternatively comprise, fiber optic-based tracking, machine-vision tracking, and the like. While in our certain implementation, optical IR tracking is utilized, the concepts and techniques described herein may be utilized to work with any sufficiently accurate 6D tracking technology. Furthermore, we assume that existing surgical navigation systems already contain appropriate methods for registering pre-operative image data and surgical plans to the patient before surgery.

The system <NUM> includes a control system <NUM> that comprises, among other components, any one or more of the manipulator controller <NUM>, the navigation controller <NUM>, and the tool controller <NUM>. The control system <NUM> includes one or more software programs and software modules. The software modules may be part of the program or programs that operate on the manipulator controller <NUM>, navigation controller <NUM>, tool controller <NUM>, or any combination thereof, to process data to assist with control of the system <NUM>. The software programs and/or modules include computer readable instructions stored in non-transitory memory <NUM> on the manipulator controller <NUM>, navigation controller <NUM>, tool controller <NUM>, or a combination thereof, to be executed by one or more processors <NUM> of the controllers <NUM>, <NUM>, <NUM>. The memory <NUM> may be any suitable configuration of memory, such as RAM, non-volatile memory, etc., and may be implemented locally or from a remote database. Additionally, software modules for prompting and/or communicating with the user may form part of the program or programs and may include instructions stored in memory <NUM> on the manipulator controller <NUM>, navigation controller <NUM>, tool controller <NUM>, or any combination thereof. The user may interact with any of the input devices of the navigation user interface UI or other user interface UI to communicate with the software modules. The user interface software may run on a separate device from the manipulator controller <NUM>, navigation controller <NUM>, and/or tool controller <NUM>.

The control system <NUM> may comprise any suitable configuration of input, output, and processing devices suitable for carrying out the functions and methods described herein. The control system <NUM> may comprise the manipulator controller <NUM>, the navigation controller <NUM>, or the tool controller <NUM>, or any combination thereof, or may comprise only one of these controllers. These controllers may communicate via a wired bus or communication network, via wireless communication, or otherwise. The control system <NUM> may also be referred to as a controller. The control system <NUM> may comprise one or more microcontrollers, field programmable gate arrays, systems on a chip, discrete circuitry, sensors, displays, user interfaces, indicators, and/or other suitable hardware, software, or firmware that is capable of carrying out the functions described herein.

Described herein are systems, methods, and techniques related to spatially-aware displays for computer assisted interventions. A novel display and visual interaction paradigm is presented, which aims at reducing the complexity of understanding the spatial transformations between the user's (e.g., surgeon) viewpoint, a physical object (e.g., a patient), 2D and 3D data (e.g., the pre/intra-operative patient data), and tools during computer-assisted interventions. The interventional display, for example in surgical navigation systems, can be registered both to the patient and to the surgeon's view. With this technique, the surgeon can keep his/her own direct view to the patient independent of any need for additional display or direct view augmentation. In some implementations, the monitor used in the operating room is registered to the patient and surgeon's viewpoint. This enables the physicians to effortlessly relate their view of tools and patient to the virtual representation of the patient data. The direct view of the surgeon onto the patient and his/her working space can remain unchanged. The position and orientation of the display plays an integral part of the visualization pipeline. Therefore, the pose of the display is dynamically tracked relative to other objects of interest, such as the patient, instruments, and in some implementations, the surgeon's head. This information is then used as an input to the image-guided surgery visualization user interface.

At least two implementations are presented. The first one uses a "Fixed View Frustum" relating the pose of the display to the patient and tools. For the Fixed View Frustum technique, the display is tracked for example by attaching a tracking marker and calibrating the spatial relation between the physical display and the tracking marker.

The second one is built upon the mirror metaphor and extends the first technique to also integrate the pose of the surgeon's head as an additional parameter within the visualization pipeline, in which case the display will be associated to a "Dynamic Mirror View Frustum". The surgeon's viewpoint is tracked for visualization techniques for the Dynamic Mirror View Frustum. Estimation of the surgeon's viewpoint can be achieved either by using a head tracking target or by mounting a camera to the surgical display in combination with existing video-based head-pose estimation algorithms. Once the tracking information is available in a common global co-ordinate system, we can compute the spatial relationship between the patient, the surgical display, the tools and the surgeon's viewpoint by deriving the relevant transformations from the spatial relations of the tracked entities.

These novel display and visual exploration paradigms aim at reducing the complexity of understanding spatial transformations between a user's viewpoint, the physical object (O), the pre/intra-operative 2D and 3D data, and surgical tools <NUM>, <NUM> during computer assisted interventions with minimal change in the current setups. Any surgical tracking system can be used to track the display, tool and user's head supporting the integration into computer assisted intervention systems. The solutions presented allow physicians to effortlessly relate their view of tools and the patient to the virtual data on surgical monitors. The users gain the possibility of interacting with the patient data just by intuitively moving their viewing position and observing it from a different perspective in relation to the patient position independent from the need for an interaction device like a mouse or joystick.

With reference to <FIG>, one example visualization method that can be utilized with the surgical system <NUM> comprises a Fixed View Frustum (FVF) for aiding in user interaction with a physical object (O). The physical object (O) can be a physical anatomy, as shown in the Figures, or any other object requiring interaction, setup, or intervention, such as a surgical device, robotic device, surgical training model of an anatomy, and the like. For simplicity, the physical anatomy is shown as the object, however, the concept is not limited to such.

One or more external screen(s) or display device(s) <NUM> are provided. These display devices <NUM> can be those displays <NUM> on the navigation cart <NUM> assembly or any other external display that is spaced apart from and not worn by the user. The display device <NUM> can be any suitable type of display, including, but not limited to: LED, LCD, OLED, touchscreen, holographic; and can display any type of imagery. The display device <NUM> can also take any suitable geometric shape, including rectangular, square, circular, or the like.

During use, the display device (s) <NUM> is located on a side (S1) of the physical object (O) opposite to a side (S2) where the user is located. In other words, the physical object (O) is between a user's viewpoint (V) and the display device <NUM>. Hence, the display device <NUM> utilized herein is distinguished from head-mounted displays or tablet screens that are between the user's viewpoint (V) and the object (O). Here, the physical object (O) is shown as a virtual representation merely for illustrative purposes.

Of course, the user is not prohibited from moving between the physical object (O) and the display device <NUM>. However, implementation of the spatially-aware display takes into account the practical reality that a surgeon desires to visualize the physical object (O) on the side which he/she is presently located and that typically the side of the physical object (O) that is opposite to the surgeon would provide an inverted perspective. This will be understood further below in context of the field of view of the virtual camera which forward-faces the display device <NUM>.

The display device <NUM> defines a plane (P). The plane (P) is defined parallel to or coincident with the actual front face of the display device <NUM>. Here, the plane (P) is a virtual object utilized for computational purposes, as will be described below. The control system <NUM>, which includes the navigation system <NUM> comprising any one or more controllers described herein, is configured to register computer images (R) to the physical object (O). Registration of the physical object (O) is not limited to any technique and can occur according to any suitable method including digitization of the physical object (O), touchless registration (e.g., ultrasonic), 2D/3D image registration utilizing an imaging device (e.g., CT, X-Ray), or the like. The computer images (R) can be represented as a virtual model (VM) of any portions of the physical object (O). These computer images (R) can be rendered on the display device <NUM>. These computer images (R) can be static or dynamic and can include actual/real video graphic images, mixed reality, augmented reality, virtual reality images or models, or any combination thereof.

The control system <NUM>, by way of the localizer <NUM> tracks a pose of the physical object (O) utilizing any of the patient tracking techniques described above, including but not limited to the patient trackers <NUM>, <NUM>, <NUM>. In some implementations, the localizer <NUM> can also track a pose of the display device <NUM> in a common coordinate system with the physical object (O). The display device <NUM> can be tracked using a display tracker <NUM>, or by any other suitable technique such as those described above. For instance, the display device <NUM> can be tracked using machine vision (e.g., with or without trackers), tracked using optical or non-optical techniques. Alternatively, or additionally, when equipped with a motorized adjustment system, the pose of the display device <NUM> can be determined based on kinematic data, and independent of the localizer <NUM>, as will be described below.

The control system <NUM> controls the display device <NUM> to render the registered computer images (R) according to at least the tracked pose of the physical object (O) and the display device <NUM>. Hence, the rendering of the computer images (R) is dependent upon at least the pose the physical object (O) and the pose of the display device <NUM>. Specifically, referring to <FIG>, a perspective of the rendering is based on a point or field of view (FOV) of a virtual camera (VC) that faces the plane (P) of display device <NUM>. The field of view (FOV) can be based on a pinhole camera model projected towards the display device <NUM>. The field of view (FOV) is also known as the viewing frustum (F), which will be described below.

A virtual position (VP) of the virtual camera (VC) is located on a side (S1) of the physical object (<NUM>) that is opposite from the side (S2) of the display device <NUM>. Since the virtual camera (VC) is located in front of the plane (P) of the display device <NUM> in <FIG>, the computer image (R) is showing the side of the physical object (O) at side (S1), rather than a mirrored view at side (S2).

The virtual position (VP) may be located at a predetermined distance (d) from the plane (P) of display device <NUM>. The distance (d) is based on a Z-axis line drawn between the virtual position (VP) of the virtual camera (VC) and the plane (P). The Z-axis may be defined with reference to the virtual position (VP) or the display device <NUM>. The virtual position (VP) can be anywhere on the virtual camera (VC) and can be defined by a single point or points, or any type of surface or volumetric geometry. In some examples, the virtual position (VP) remains fixed at a predetermined distance (d). Alternatively, the predetermined distance (d) can change in response to certain conditions. For example, the user can specify to the control system <NUM> a preference for the distance (d). The distance (d) can change automatically depending upon conditions such as, but not limited to: a type of surgical procedure, a certain step of the procedure, a tracked pose of the user's viewpoint (V), a tracked pose of the object, a tracked pose of the display device <NUM>, a tracked pose of a tool <NUM>, <NUM>, or the like. In one example, the distance (d) can be defined at any suitable distance and within any suitable range, including, but not limited to between <NUM> meter and <NUM> meters, or any distance inclusively therebetween.

In one example, the virtual position (VP) of the virtual camera (VC) is transverse to the plane (P) of the display device <NUM>. In other words, a Z-axis line drawn between the virtual position (VP) and the plane (P) of the display device <NUM> defines an angle relative to the plane (P) wherein the angle is in a range between <NUM> and <NUM> degrees. In a more specific implementation, the virtual position (VP) of the virtual camera (VC) is orthogonal to the plane (P) of the display device <NUM>. In other words, a line drawn between the virtual position (VP) and the plane (P) of the display device <NUM> is <NUM> degrees normal to the plane (P). In other words, in this example, computer images (R) are rendered with an on-axis perspective projection orthogonal to the display device <NUM> from the given distance (d).

The virtual position (VP) of the virtual camera (VC) also has X and Y coordinate locations relative to the plane (P) of the display device <NUM>. The X and Y coordinates may be defined with reference to the virtual position (VP) or the display device <NUM>. In one example, the X and Y coordinates are defined in a plane that is parallel to the plane (P) of the display device <NUM>. However, the X and Y coordinate plane need not be parallel to the plane (P), for example, if the virtual camera (VC) is projecting towards the display device <NUM> at a non-orthogonal angle.

Furthermore, whether projecting transverse or orthogonal to the display device <NUM>, the line drawn between the virtual position (VP) and the plane (P) may be directed towards the geometrical center of the display device <NUM>. Alternatively, the line drawn between the virtual position (VP) and the plane (P) may be offset from geometrical center of the display device <NUM>. For example, the offset location can be towards a corner or edge of the display device <NUM>. The X-Y placement of virtual position (VP) relative to the display device <NUM> can change in response to certain conditions. For example, the user can specify to the control system <NUM> a preference for X-Y placement of virtual position (VP). The X-Y placement of virtual position (VP) can change automatically depending upon conditions such as, but not limited to: a type of surgical procedure, a certain step of the procedure, a tracked pose of the user's viewpoint (V), a tracked pose of the object, a tracked pose of the display device <NUM>, a tracked pose of a surgical tool <NUM>, or the like. In one example, the distance (d) can be defined at any suitable distance and within any suitable range, including, but not limited to between <NUM> meter and <NUM> meters, or any distance inclusively therebetween.

Rendering of the computer images (R) based on the virtual camera (VC) is implemented using a viewing frustum (F) originating from the virtual camera (VC) and passing through an object, which in this case is the plane (P) of the display device <NUM>. The viewing frustum (F) may be a region of space in a 3D modeled world that may appear on the display device <NUM> and may considered as the field of view of the virtual camera (VC). An apex of the frustum (F) is the zero-point originating from the virtual camera (VC) and may also be regarded as the virtual position (VP) of the virtual camera (VC). In one example, the plane (P) of the display device <NUM> may be located at a base (B) or far-clipping plane (FCP) of the viewing frustum (F). The viewing frustum (F) may comprise a near clipping plane (NCP) proximate to the virtual camera (VC). The far-clipping plane (FCP) and near-clipping planes (NCP) virtually cut the frustum (F) perpendicular to the viewing direction such that objects closer to the camera than the near-clipping planes (NCP) or beyond the far-clipping plane (FCP) are not rendered on the computer image (R). Objects that lie partially or completely outside of the viewing frustum (F) can be removed from the rendering process for processing efficiency. The computer images (R) on the display device <NUM> are projected based on the viewing frustum (F). The viewing frustum (F) can be based off any plane truncation or any suitable <NUM>-shape, including a pyramid, cone, or the like. The viewing frustum (F) can take any configuration other than that shown in the Figures or described herein.

In the FVF technique, the virtual position (VP) of the virtual camera (VC) is automatically updated by the control system <NUM> in response to (manual or motorized) adjustment to the pose of the display device <NUM>. If the user moves or rotates the display device <NUM>, the position of the virtual camera (VC) is automatically updated. Therefore, in contrast to the standard visualization displays within surgical navigation systems, the mental mapping of the real object to its image on the screen is simplified.

Examples of such Fixed View Frustum display visualizations are shown in <FIG>. An illustrative real-world example is shown in <FIG> and two simulations are shown in <FIG>, respectively. <FIG> shows an operator moving the display device <NUM> by hand to visualize the internal virtual model (VM) of the physical object (O). As the user adjusts the display device <NUM>, the computer rendering (R) on the display device <NUM> updates accordingly. <FIG> provide simulation illustrations of the FVF technique from two different viewpoints of the same configuration of the virtual camera (VC). As comparatively shown, the display device <NUM> changes pose between <FIG> and the virtual position (VP) of the virtual camera (VC) updates accordingly. The computer image (R) also changes perspective in accordance with the relative positioning between the viewing frustum (F) and the physical object (O).

Referring now to <FIG>, a Screen Parallel Slice Visualization (SPSV) sub-technique of the FVF method is described. While the FVF visualization is intuitive to use and can display 3D data, the technique can be further implemented using slice visualization. Accordingly, one can incorporate the slice view into the spatially-aware visualization concept described.

With reference to <FIG>, this can be achieved in one implementation by slicing a 3D virtual model (VM) of the physical object (O) into a plurality of slices (SL1, SL2, SL3. In one implementation, the slices (SL) are made at planes parallel to the plane (P) of the display device <NUM>. The virtual model (VM) can be a CT model, X-ray model, MRI model or a model created using any other type of imaging technique. Alternatively, instead of slicing a 3D virtual model (VM), the imaging data of the physical object (O) may comprise a plurality of slices that can be obtained from computer-memory with or without having combined the same into a 3D model.

In one implementation, and with reference to <FIG>, the displayed slice (SL) can be based on a designated plane (p) sliced through the viewing frustum (F). The designated plane (p) can be fixed relative to the virtual camera (VC) position, or the designated plane (p) can dynamically change based on any of the conditions described herein. When the field of view of the virtual camera (VC) passes through the physical object (O), the control system <NUM> can immediately obtain the slice (SL) that corresponds to the intersection of the designated plane (p) with the physical object (O).

The orientation of the computer image (R) of the slice (SL) can be adjusted based on the pose of the display device <NUM>. As shown comparatively between <FIG>, the user is given the interaction method to rotate the display device <NUM>, which in turn also rotates the slice (SL) and a virtual model (T) of the tool <NUM>, <NUM>. The physical object (O) and tool <NUM>, <NUM> are in the same relative position in <FIG>.

Alternatively, or additionally, as shown in <FIG>, the orientation of the slice (SL) can be set based on a tool <NUM>, <NUM> brought within the viewing frustum (F) of the virtual camera (VC). A pose of the tool <NUM>, <NUM> can be tracked by the localizer <NUM> using any suitable means, such as those described herein. The computer images (R) that are rendered can include images of the tool <NUM>, <NUM> and slice (SL) using the Fixed View Frustum method as discussed above. Alternatively, or additionally, the orientation of the slice (SL) can be set based on a user's perspective as defined by a tracked pose of a viewpoint tracking system (VTS), such as an external camera (C) facing the user or a head-mounted device <NUM> (described in detail below).

The displayed slice (SL) can be dynamically changed to different slices (SL) during use of these techniques. The slice (SL) can change depending on the pose of the physical object (O), the display device <NUM>, or any combination thereof. Additionally, or alternatively, the slice (SL) can change using any of the input devices described herein. Also, as shown in <FIG>, the tool <NUM>, <NUM> can be utilized to change the slice (SL). For example, the tool's <NUM>, <NUM> pose, position, and/or distance or orientation relative to either the physical object (O) or the display device <NUM>, can automatically cause the control system <NUM> to change the slice (SL). In some implementations, portions of the slice (SL) can be displayed based on a (2D or 3D) virtual boundary (VB) that is associated with a tip <NUM> of the tool <NUM>, <NUM>, as shown in <FIG>. For instance, the virtual boundary (VB) can be defined by any shape (e.g., rectangle, box, circle or sphere) having any suitable dimension (e.g., a <NUM> diameter) with the tool tip <NUM> at the center of the boundary (VB). If the tool <NUM>, <NUM> is moved towards the object (O) such that the object (O) intersects the virtual boundary (VB), the control system <NUM> can dynamically present or change the displayed slice (SL) that corresponds to the intersection. The slice (SL) can be presented in whole (as shown in <FIG>) or can be cropped (<FIG>) depending upon the location of the virtual boundary (VB).

With reference to <FIG>, the control system <NUM> is configured to render the computer images (R) in a contextually accurate manner. In other words, the computer image (R) rendering takes into account spatial layering of the object (O), tool <NUM>, <NUM>, and/or slice (SL) relative to one other in in the common coordinate system so as to mimic the actual spatial location of the same in the real coordinate system in which these items exist. As shown in <FIG>, the tool <NUM>, <NUM> is physically inserted into the object (O) and the display device <NUM> presents the virtual model (T) of the tool <NUM>, <NUM> and the corresponding slice (SL) relative to a 3D model (VM) of the object (O). However, as shown, these renderings (R) layered based on the perspective of the virtual camera (VC). In other words, the virtual model (T) of the tool <NUM>, <NUM> deliberately obstructed by corresponding portions of the 3D model (VM) of the object (O). In this case, the rib cage of the anatomy is obstructing the tool <NUM>, <NUM> shaft. Similarly, the slice (SL) is displayed such that portions of the slice (SL) are obstructed by corresponding portions of the 3D model (VM) of the object (O). In this case, the slice (SL) is partially obstructed by several ribs. Although this technique obstructs portions of the visualization, it benefits the user to visualize the environment in a context that closely resembles real-world spatial positioning of the object (O), slice (SL) and tool <NUM>, <NUM>. This visualization technique can be utilized for any of the techniques described herein, including the FVF and SPSV techniques, as well as for any of the specific implementations, conditions, and/or situations described herein.

As shown in <FIG>, the display device <NUM> may optionally be connected to an adjustment system <NUM>. The adjustment system <NUM> can be passive or active and can comprise any features or possible configurations of the robotic manipulator <NUM> described above. For example, the adjustment system <NUM> may comprise a plurality of links (L), joints (J), and actuators (M) for driving the joints (J). The adjustment system <NUM> can be manually or automatically controlled to adjust a pose of the display device <NUM>. The pose of the display device <NUM> can be moved in up to six-degrees of freedom (three translational and three rotational). Sensors (S) may be provided on any of the components of the adjustment system <NUM> to detect movement of the adjustment system <NUM>. The control system <NUM> can utilize measurements from the sensors (S) to kinematically derive the pose of the display device <NUM>. This can be done in addition to or alternatively from using the localizer <NUM>. The sensors (S) can be any suitable configuration, including, but not limited to: motor current sensors, joint position sensors or encoders (rotary, absolute, incremental, virtual-absolute, etc.), optical sensors, inertial sensors (accelerometer, inclinometer, gyroscope, etc.) or the like. Any of the sensors (S) can also be provided on the display device <NUM> itself.

In some instances, as shown in <FIG>, one or more user input devices <NUM> can be wired or wirelessly connected to the control system <NUM> to enable the user to control the pose of the display device <NUM>. The input devices <NUM> include but are not limited to: a foot pedal 106a, a hand-held pendant 106b, display input 106c such as a tablet, smartphone, or display of the navigation system, the tool <NUM>, <NUM>, and/or a viewpoint tracking system (VTS), such as an external camera (C) facing the user or a head-mounted device <NUM>. The input devices can also be any of those described above, including but not limited to: push buttons, sensors, switches, keyboard, mouse, microphone (voice-activation), gesture control devices, touchscreens, joysticks, etc. The input device <NUM> receives a command from the user and the control system <NUM> directs the one or more actuators M to adjust the joints J and ultimately the pose of the display device <NUM> in accordance with the command.

The tracked tool <NUM>, <NUM> can also be utilized for triggering the control system <NUM> to control the adjustment system <NUM>. In one implementation, the pose of the tracked tool <NUM>, <NUM>, whether it be position and/or orientation, or any derivate thereof (velocity, acceleration, etc.) can cause the control system <NUM> to adjust the pose of the display device <NUM>. For example, the tracked pose tool <NUM>, <NUM> can be compared to the virtual position (VP) of the virtual camera (VC), the viewing frustum (F), the plane (P) of the display device <NUM>, or any combination thereof. The control system <NUM> can assess this comparison relative to a threshold condition or measurement. If the control system <NUM> determines that the tool <NUM>, <NUM> has moved in a manner that meets or exceeds the threshold condition, the control system <NUM> can command the adjustment system <NUM> to adjust the display device <NUM>. This may be beneficial for various situations, including, but not limited to: the tool <NUM>, <NUM> moving towards or away from the object (O), the tool <NUM>, <NUM> moving in/out of the viewing frustum (F), keeping the tool <NUM>, <NUM> within view of the virtual camera (VC), or the like.

With continued reference to <FIG>, the head-mounted device <NUM> can also be utilized for triggering the control system <NUM> to control the adjustment system <NUM>. In one implementation, the pose of the head-mounted device <NUM> is tracked using any tracking technique described herein, or equivalents thereof. The tracked pose of the head-mounted device <NUM>, whether it be position and/or orientation, or any derivate thereof (velocity, acceleration, etc.) can cause the control system <NUM> to adjust the pose of the display device <NUM>. For example, the tracked pose of the head-mounted device <NUM> can be compared to the virtual position (VP) of the virtual camera (VC), the viewing frustum (F), the plane (P) of the display device <NUM>, or any combination thereof. The control system <NUM> can assess this comparison relative to a threshold condition or measurement. If the control system <NUM> determines that the head-mounted device <NUM> has moved in a manner that meets or exceeds the threshold condition, the control system <NUM> can command the adjustment system <NUM> to adjust the display device <NUM>. This may be beneficial for various situations, including, but not limited to: the head-mounted device <NUM> moving towards or away from the object (O), the head-mounted device <NUM> moving in/out of the viewing frustum (F), keeping the head-mounted device <NUM> within view of the virtual camera (VC), or the like.

Additionally, or alternatively, any other viewpoint tracking system (VTS), such as an external camera (C) facing the user can be utilized to track the user's viewpoint (V) for triggering the control system <NUM> to control the adjustment system <NUM>.

The adjustment system <NUM> can be utilized for any of the techniques described herein, including the FVF and SPSV techniques, as well as for any of the specific implementations, conditions, and/or situations described herein.

With reference to <FIG>, another example visualization method that can be utilized with the surgical system <NUM> comprises a Dynamic Mirror View Frustum (DMVF) for aiding in user interaction with the physical object (O).

As will be understood from the description below, there are technical similarities between the FVF and DMVF techniques. Accordingly, any and all of the above description related to the system <NUM>, the FVF technique, and any physical, computational, and/or any techniques associated therewith are fully incorporated by reference for use with and can be applied by the DMVF technique described herein and hence are not repeated for simplicity of description.

For the DMVF technique, and with reference to <FIG>, the display device <NUM> defines the plane (P). The physical object (O) is located in front of the plane (P). In other words, the physical object (O) is located in front of the displayed screen of the display device <NUM>.

The user viewing the display device <NUM> is also located in front of the plane (P). The user's viewpoint (V) is tracked utilizing a viewpoint tracking system (VTS). In one implementation, the viewpoint tracking system (VTS) includes a camera (C) facing the user. The viewpoint tracking system (VTS) can the navigation system itself or can be part of or separate from the navigation system. The camera (C) can be the localizer <NUM> or a separate device. The camera (C) can be mounted to the display device <NUM>, or elsewhere. The control system <NUM> can receive signals from the camera (C) and recognize changes to position and/or orientation of the user's face, eyes, head, or other feature using pose estimation algorithms.

In another implementation, the viewpoint tracking system (VTS) additionally or alternatively includes a head-mounted device <NUM> that is provided directly on the user. The head-mounted device <NUM> is located in front of the plane (P) and in front of the displayed screen of the display device <NUM>. In a one configuration, the head-mounted device <NUM> is located on a first side (S1) of the physical object (O) and the display device <NUM> is located on a second opposing side (S2) of the physical object.

The head-mounted device <NUM> comprises one or more trackable features (HT) such that its pose can be tracked by the navigation system <NUM> and/or control system <NUM>. The trackable features (HT) can be any type described above or any equivalents thereof. In this technique, the head-mounted device <NUM> is any device that is configured to enable tracking the pose of the user's point of view. Here, pose means position of the head-mounted device <NUM>, and optionally, position and orientation. The user's point of view can be defined by the general field of view of the user's vision, the direction in which the user turns his/her head, and/or the gaze of the user's eyes. The head-mounted device <NUM> can be or include one or more trackers attached to any part of a user's head and/or eye or head wear such as glasses (such as those shown in <FIG> for example), goggles, head-band helmet, contact lenses, or the like. In one example, the head-mounted device <NUM> is an optical-see-through head-mounted-display. Other examples of the head-mounted device <NUM> are contemplated. In <FIG>, the viewpoint derived from tracking the head-mounted device <NUM> is shown as a sphere for simplicity and the viewpoint is facing the display device <NUM>.

Just as with the FVF technique, the navigation system <NUM> and/or control system <NUM> registers computer images (R) to the physical object (O). The navigation system <NUM> and/or control system <NUM> track a pose of the physical object (O), the display device <NUM>, and the user's viewpoint (V) relative to one another in a common coordinate system. The display device <NUM> is controlled for rendering the registered computer images (R) according to the tracked pose of the physical object (O), the display device <NUM>, and the user's viewpoint (V). In other words, the pose of each of these items can affect how the computer images (R) are displayed.

A perspective of the computer image (R) is based on a field of view or viewing frustum (F) of a virtual camera (VC) that has a virtual position (VP) behind the plane (P), shown as side (S3) in <FIG>, which is behind the display device <NUM>. The virtual camera (VC) faces the rear of the display device <NUM> and faces the viewpoint of the user as derived from the viewpoint tracking system (VTS). The physical object (O), the display device <NUM>, and the user's viewpoint (V) are at least partially within the viewing frustum (F) in <FIG>. Since the virtual camera (VC) is located behind the plane (P) of the display device <NUM>, the rendering (R) provides visualization as though the display device <NUM> were a mirror relative to the viewpoint of the user. In <FIG>, the computer image (R) is showing the side of the physical object (O) at side (S2).

The virtual position (VP) of the virtual camera (VC) is automatically updated in response to adjustment to the tracked pose of the user's viewpoint (V). The user can move left and right or back and forth and see the physical object (O) and any tool <NUM>, <NUM> moving correspondingly on the mirror-like rendering (R) of the display device <NUM>. By knowing the poses of the physical object (O), the display device <NUM>, and the user's viewpoint (V), the control system <NUM> is able to create the computer images (R) such that they obey the same laws as a real mirror.

<FIG> illustrate examples of the DMVF technique. Specifically, simulated mirror views are shown in <FIG> and real-world illustrative examples are shown in <FIG>. Comparing the third person views, as shown in <FIG> and <FIG>, the virtual position (VP) of the virtual camera (VC) changes depending on user's viewpoint (V) (shown as a sphere) as derived from viewpoint tracking system (VTS). <FIG> shows the first-person viewpoint of what would be seen on the display device <NUM> in the scenario of <FIG>. <FIG> shows the first-person viewpoint of what would be seen on the display device <NUM> in the scenario of <FIG>.

The display device <NUM> shows the objects in front of the display just like a mirror would do, taking the poses of the screen, objects, and the user viewpoint into account. This paradigm works can contribute to a more intuitive visualization of the physical object (O) by using motion parallax and observing structures from additional desired viewpoints. The user gains the freedom to interact with the data just by looking, without necessarily requiring an interaction device like a mouse or joystick. As the users do not constantly need to redefine their view on the object (O), the user is enabled to toggle the interactivity on or off for example with any input device, such as a foot pedal. The user can change the visualization in a hands-free manner. Since the natural human visual system is adapted to looking at real mirrors, the DMVF visualization provides an intuitive interface. DMVF also facilitates exploring and defining the best slice (SL) or rendering (R) for a given navigation task. The slice (SL) or rendering (R) can remain fixed during the surgical action until further data exploration is desired.

The virtual position (VP) of the virtual camera (VC) dynamically changes in the DMVF method depending upon at least the tracked pose of the user's viewpoint (V). In one implementation, the viewpoint of the virtual camera (VC) can be derived as described herein. In one example, the control system <NUM> moves the virtual camera (VC) according to a projection matrix that is adjusted so that the viewing frustum (F) is fitted such that boundary features (BF) of the frustum (F) coincide with the features (DF) of the display device <NUM>. The features (DF) of the display device <NUM> comprise points or other geometry which encode a size, aspect ratio, position and orientation of the display device <NUM> in the common or world coordinate system. In the example of <FIG>, wherein the viewing frustum (F) comprises a truncated four-sided pyramid and the display device <NUM> is rectangular, the boundary features (BF) of the viewing frustum (F) are points on the edges of the sides of the pyramid and the features (DF) are the fixed corners of the display device <NUM>. Hence, the points on the edges of the viewing frustum (F) are made to coincide with the corners of the display device <NUM> for any positional movement of the virtual camera (VC) in the coordinate system. Of course, this configuration may change depending on the geometry of the viewing frustum (F) and the display device <NUM>. This results in a warped image (R), which appears in correct perspective as a mirror reflection from the tracked pose of the user's viewpoint (V).

From the virtual position (VP) of the virtual camera (VC), a mirrored world position of the viewpoint pmirrored can be calculated as follows: define the normal to the mirror plane (P) as (<NUM>; <NUM>; <NUM>). In its local coordinate system, the mirroring corresponds to multiplication with the matrix Mflip which is expressed in [<NUM>] and which considers the fixed features (DF) of the display device <NUM> (i.e., in this case the four corners).

This is equivalent to scaling by -<NUM> in z-direction. Let us assume, according to one implementation, that to provide the mirror view, the rendering depends on the position, and not the orientation, of the user's viewpoint (V) with respect to the mirror plane (P). A first transform is performed to transform a world position of the virtual camera (VC) viewpoint p into local coordinates of the mirror. This causes rotation of the frustum (F) to the viewpoint of the user. Then mirroring is performed with matrix Mflip which provides perspective projection relative to the display device <NUM>. Lastly, the mirrored local coordinates of the mirror are transformed back to the world position of the virtual camera (VC) viewpoint p which translates the user's viewpoint (V) to the apex of the viewing frustum (F). The result is a projection matrix expressed by:
<MAT>.

In the equation expressed in [<NUM>], the notation ATB is utilized to represent a transformation from coordinate system A into B. This computation enables re-alignment or rotation of the base (B) or far-clipping plane (FCP) of the frustum (F) from an otherwise XY coordinate aligned position relative to the display device <NUM> to an alignment that is positioned relative to the user's viewpoint (V). Thereafter, the user viewpoint-aligned base (B) or far-clipping plane (FCP) is re-aligned back to the XY coordinates of the display device <NUM> such that an off-axis projection matrix can be applied to apply perspective rendering of the computer images (R).

To enable the computer images (R) on the display device <NUM> to render mirror views of the physical object (O) or tools <NUM>, <NUM> in front of the display device <NUM> in relation to the user's pose, the off-axis projection matrix can be employed in one implementation. Here off-axis means that a line drawn from the user's viewpoint (V) to the display device <NUM> would not be at the geometrical center of the display device <NUM>, but rather off-center. In such situations, the viewing frustum (F) becomes asymmetric and will change shape as the user's viewpoint (V) changes. The location at which the user's viewpoint (V) relative to the geometry of the plane (P) of the display device <NUM> can be monitored by the control system <NUM>. As the frustum (F) shape changes depending upon the user's viewpoint (V), the boundary features (BF) of the viewing frustum (F) change. In one implementation, the viewing frustum (F) can be recomputed and/or updated at frequent intervals, such as, e.g., at every frame, as the pose of the display device <NUM> and/or the viewpoint of the user are likely to change.

The off-axis projection matrix has the mirrored viewpoint p at the apex of the viewing frustum (F) and the base (B) matching the surface or plane (P) of the display device <NUM>. The viewing frustum (F) is rotated to align with the user's viewpoint (V). This takes the position of the viewpoint into account and warps the image so that it appears like a projection onto a tilted plane relative to the plane (P) of the display device <NUM>. The effect of the changed viewpoint of the user on the frustum (F) together with the warped images, can be seen in <FIG> as compared with <FIG>. This implementation relies on a graphic API to set the projection matrix with the frustum (F) defined by the near clipping plane (NCP) coordinates in view space, which is then rotated to be non-perpendicular and moved to be located at the mirrored viewpoint. In other words, the base (B) of the frustum (F) is dynamically modified rotate out of the X-Y plane of the plane (P) of the display device <NUM> and positioned corresponding to the angle of the user's viewpoint (V) as derived from the viewpoint tracking system (VTS).

The calculation provided above provides one implementation of virtual camera (VC) setup for DMVF. However, there may be alternative manners of dynamically changing the virtual position (VP) of the virtual camera (VC) depending upon at least the user's viewpoint (V), without departing from the scope of the concept. For example, it is possible to align the base (B) of the frustum (F) to the display device <NUM>. Boundary features (BF) of the viewing frustum (F) may exceed the geometry of the display device <NUM>. In such instances, the control system <NUM> may limit the boundary features (BF) of the viewing frustum (F) relative to the geometry of the display device <NUM>.

<FIG> compare, in various views of the X,Y,Z coordinate system, the relative positioning between the virtual camera (VC) and frustum (F), user's viewpoint (V), display device <NUM> and object (O) among three example scenarios. In these examples, the position of the virtual camera (VC) and the user's viewpoint (V) can be mirrored relative to the plane (P) of the display device <NUM> for each of the X, Y, and Z axes individually. For instance, the virtual camera (VC) and the user's viewpoint (V) are substantially equal distance apart from the display device <NUM> in each of the X, Y, and Z-directions. When visualized from the X-Y plane, the position of the virtual camera (VC) and the user's viewpoint (V) appear to coincide as result of this mirroring effect. However, perfect mirroring is not required in all implementations and the distance of (VC) and (V) to the display device <NUM> may not be equal for each of the axes and such distances can be customized or changed.

In <FIG>, the user's viewpoint (V) is on-axis at given position relative to the display device <NUM>. The base (B) of the frustum (F) of virtual camera (VC) aligns with the X-Y plane of the display device <NUM>. The projection renders computer images (R) that are centered and aligned to the user's centered viewpoint.

In <FIG>, the user's viewpoint (V) is moved to the left of center (from the user's perspective) and moved away from the display device <NUM> further than the given position of <FIG>. This relative change in motion of the user's viewpoint (V) causes the virtual camera (VC) to change correspondingly. In other words, the virtual camera (VC) moves to the right and away from the display device <NUM> (from the virtual camera's perspective). This correctly positions the virtual camera (VC) relative to the user's viewpoint (V). The viewing frustum (F) is rotated (counter-clockwise as seen from above) out of the X-Y plane of the display device <NUM> to the align to the user's viewpoint (V). In some instances, modification of the frustum (F) causes the far-clipping plane (FCP) to move further from the apex of the frustum (F) in response to the position of the virtual camera (VC) moving away from the display device <NUM>. The projection renders computer images (R) that are rotated to align to the user's off-center viewpoint. The computer images (R) also render objects, such as the physical object (O) within the frustum (F) to be smaller in size as compared to <FIG> since the position of the user's viewpoint (V) is further away from the display device <NUM>.

In <FIG>, the user's viewpoint (V) is moved to the right (from the user's perspective) and moved closer to the display device <NUM> than the given position of <FIG>. This relative change in motion of the user's viewpoint (V) causes the virtual camera (VC) to move to the left and closer to the display device <NUM> (from the virtual camera's perspective). This correctly positions the virtual camera (VC) relative to the user's viewpoint (V). The viewing frustum (F) is rotated (clockwise as seen from above) out of the X-Y plane of the display device <NUM> to the align to the user's viewpoint (V). In some instances, modification of the frustum (F) causes the far-clipping plane (FCP) to move closer to the apex of the frustum (F) in response to the position of the virtual camera (VC) moving away from the display device <NUM>. The projection renders computer images (R) that are rotated to align to the user's off-center viewpoint. The computer images (R) also render objects, such as the physical object (O) within the frustum (F) to be larger in size as compared to <FIG> since the position of the user's viewpoint is closer to the display device <NUM>.

The dimensioning of the components of the <FIG> are merely provided for illustrative purposes and may not be exactly to scale. Furthermore, although the X-Z plane is shown illustrating lateral motion among the user's viewpoint and virtual camera (VC), the principles described herein may apply fully to vertical motion between the same in the X-Y plane.

Referring now to <FIG>, a Viewpoint Facing Slice Visualization (VFSV) sub-technique of the DMVF method is described. While the DMVF visualization is intuitive to use and can display 3D data, it is contemplated to further incorporate the slice view into the spatially-aware DMVF visualization concept described.

The slicing of the data can occur according to any of the techniques described above with respect to the SPSV technique and as shown with respect to <FIG>, and hence, are not repeated herein for simplicity. Furthermore, the selection of the displayed slice (SL) can occur according to any of the techniques described above with respect to the SPSV technique and as shown with respect to <FIG>.

In one implementation, the tool <NUM>, <NUM> can be utilized to change the slice (SL). For example, the tool's <NUM>, <NUM> pose, position, and/or distance or orientation relative to either the physical object (O) or the display device <NUM>, can automatically cause the control system <NUM> to change the slice (SL). In some implementations, portions of the slice (SL) can be displayed based on a (2D or 3D) virtual boundary (VB) that is associated with a tip <NUM> of the tool <NUM>, <NUM>, as shown in <FIG>. For instance, the virtual boundary (VB) can be defined by any shape (e.g., rectangle, box, circle or sphere) having any suitable dimension (e.g., a <NUM> diameter) with the tool tip <NUM> at the center of the boundary (VB). If the tool <NUM>, <NUM> is moved towards the object (O) such that the object (O) intersects the virtual boundary (VB), the control system <NUM> can dynamically present or change the displayed slice (SL) that corresponds to the intersection. The slice (SL) can be presented in whole (as shown in <FIG>) or can be cropped (<FIG>) depending upon the location of the virtual boundary (VB).

Additionally, or alternatively, the slice (SL) may be changed depending on the pose of the physical object (O), the display device <NUM>, user's viewpoint or any combination thereof. The slice (SL) may be changed using any of the input devices <NUM> described herein.

The orientation of the slice (SL) can be manipulated according to several different techniques. In the VFSV technique, the control system <NUM> tracks the display device <NUM> and the user's viewpoint (V) according to the DMVF technique. A position of the slice (SL) is based on the tracked position of the tool <NUM>, <NUM>. The orientation of the slice (SL), however, is chosen to point towards the mirror viewpoint. This guarantees that the slice (SL) faces the user when viewed through the mirror. That is, the slice (SL) is oriented to face the user's viewpoint (V) as derived by the viewpoint tracking system (VTS). The orientation of the slice (SL) rendered using the DMVF technique described above. That is, to accurately account for the user's viewpoint (V) relative to the display device <NUM>, the slice (SL) is rotated relative to the plane (P) of the display device <NUM> and projected to be smaller or larger.

An example of the VFSV technique from two different first-person viewpoints can be seen in <FIG>, wherein the tool <NUM>, <NUM> remains stationary between <FIG>. In <FIG>, the user's viewpoint (V) is moved slightly to the left, and closer to the tool <NUM>, <NUM>. The distance of the user's viewpoint (V) to the display device <NUM> remains about the same. In this example, the displayed slice (SL) appears to be the same to the user since the slice (SL) has rotated to correspond to the lateral motion of the user's viewpoint (V). If the user in <FIG> were to approach the display device <NUM>, the slice (SL) will appear correspondingly larger, and vice-versa.

While the user's viewpoint (V) is utilized to implement the VFSV technique, the orientation of the computer image (R) of the slice (SL) can additionally be adjusted based on the pose of the display device <NUM>, the pose of the object (O) and/or the pose of the tracked tool <NUM>, <NUM> brought within the viewing frustum (F) of the virtual camera (VC).

Additionally, the control system <NUM> is configured to render the computer images (R) in a contextually accurate manner for the VFSV technique. In other words, the computer image (R) rendering takes into account spatial layering of the object (O), tool <NUM>, <NUM>, and/or slice (SL) relative to one other in in the common coordinate system so as to mimic the actual spatial location of the same in the real coordinate system in which these items exist. Hence, any of the description above related to the contextually accurate rendering for the SPSV technique, including the visualization of <FIG> can be applied in its entirety to the VFSV technique and is not repeated for simplicity in description.

The control system <NUM> can enable switching between any of the visualization modes described herein at any time and in response to any input or condition. For example, the control system <NUM> can enable switching between any of the following: FVF and DMVF visualizations; SPSV and the VFSV visualizations; 3D model visualization (without slices) using FVF visualization and using the SPSV technique (with or without the 3D model); 3D model visualization (without slices) using the DMVF technique and visualization using the VFSV technique (with or without the 3D model); static visualization and any of the FVF, SPSV, DMVF and VFSV techniques.

The switching between visualization modes can occur depending on user input. For example, the control system <NUM> can receive a command from any of the input devices <NUM> described herein, which include but are not limited to: a foot pedal 106a, a hand-held pendant 106b, display input 106c such as a tablet, smartphone, display device <NUM> or display of the navigation system, the tool <NUM>, <NUM>, and/or a head-mounted device <NUM>, push buttons, sensors, switches, keyboard, mouse, microphone (voice-activation), gesture control devices, touchscreens, joysticks, etc..

The switching between visualization modes can occur automatically depending upon conditions such as, but not limited to: a type of surgical procedure, a certain step of the procedure, a tracked pose of the user's viewpoint (V) or head-mounted device <NUM>, a tracked pose of the object (O), a tracked pose of the display device <NUM>, a tracked pose of a tool <NUM>, <NUM>, or any combination thereof.

Several implementations have been discussed in the foregoing description. However, the implementations discussed herein are not intended to be exhaustive or limit the invention to any particular form. The terminology which has been used is intended to be in the nature of words of description rather than of limitation. Many modifications and variations are possible in light of the above teachings and the invention may be practiced otherwise than as specifically described. The invention is solely defined by the appended claims.

Claim 1:
A system (<NUM>) for aiding in interaction with a physical object (O), the system (<NUM>) comprising:
a display device (<NUM>) defining a plane (P) and being located on a first side (S1) of the physical object (O); and
a navigation system (<NUM>) coupled to a control system (<NUM>) and being configured to:
register computer images (R) to the physical object (O);
track a pose of the physical object (O) and the display device (<NUM>) in a common coordinate system; and
control the display device (<NUM>) to render the registered computer images (R) according to the tracked pose of the physical object (O) and the display device (<NUM>); and
wherein a perspective of the rendering is based on a virtual camera (VC), and wherein the virtual camera (VC) has a field-of-view (FOV) that faces the plane (P) of display device (<NUM>), and wherein the virtual position (VP) of the virtual camera (VC) updates automatically in response to adjustment to the pose of the display device (<NUM>); and
characterised in that the virtual camera has a virtual position located on a second side (S2) of the physical object opposite to the first side.