Patent Description:
The technology disclosed herein relates to methods and systems for controlling or otherwise interacting with equipment, tools and/or the like in a medical (e.g. surgical) environment.

There is a general desire for medical practitioners (e.g. surgeons, interventional radiologists, nurses, medical assistants, other medical technicians and/or the like) to control or otherwise interact with medical equipment, tools and/or the like in a medical (e.g. surgical) environment.

By way of non-limiting example, the PCT Applications describe the desirability for medical practitioners to interact with information systems which provide medical information (e.g. images of the patient's body and/or organs) that may be germane to the procedure being performed. Such desired medical information may include, by way of non-limiting example, radiological images, angiography images, other forms of images of the patient's body, other information relevant to a patient undergoing the medical procedure, other information relevant to the procedure itself, other information related to the condition being treated and/or the like. Such desired medical information may be procured prior to performing the procedure and/or during performance of the procedure and may allow medical practitioners to formulate or alter their therapeutic plan during image-guided medical procedures.

However, the desirability of controlling or otherwise interacting with medical equipment, tools and/or the like in a medical (e.g. surgical) environment is not limited to information systems. There is a desire to control other types of medical equipment, tools and/or the like in surgical environments. By way of non-limiting example, it can be desirable to control the pose (i.e. orientation and position) of an adjustable patient bed (e.g. to tilt the patient's body); the brightness of a light source; the directionality of a spotlight or working light; the information displayed by diagnostic equipment (vital signs monitors); the rate of infusion of an intra-venous drug delivery system and/or the like.

One aspect of the invention provides a method according to claim <NUM> for touchless control of one of more medical equipment devices in an operating room. The method comprises providing a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; displaying an interaction display unit (IDU) image corresponding to the three-dimensional control menu, the IDU image providing indicia of any one or more selected menu items; estimating a line of sight of a practitioner; when the estimated line of sight is directed within a first spatial range around a first medical equipment device, determining that the practitioner is looking at the first medical equipment device and wherein, after determining that the practitioner is looking at the first medical equipment device: providing the three-dimensional control menu comprises providing a first device-specific three-dimensional control menu comprising first device-specific menu items which, when selected, result in delivering corresponding operational commands to the first medical equipment device to control operation of the first medical equipment device; and displaying the IDU image corresponding to the three-dimensional control menu comprises displaying a first device-specific IDU image comprising graphics or text corresponding to the first device-specific menu items.

Another aspect of the invention provides a system according to claim <NUM> for touchless control of one or more medical equipment devices. The system comprises a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor; a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three- dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item and detected by the controller based on the 3D optical data; an IDU display for displaying an IDU image corresponding to the three-dimensional control menu, the IDU image providing indicia of any one or more selected menu items. The controller is further configured, based on input from one or more sensors, to estimate a line of sight of a practitioner. When the estimated line of sight is directed within a first spatial range around a first medical equipment device, the controller is configured to determine that the practitioner is looking at the first medical equipment device and wherein, after determining that the practitioner is looking at the first medical equipment device, the controller is configured to: provide a first device-specific three-dimensional control menu comprising first device-specific menu items which, when selected, result in delivering corresponding operational commands to the first medical equipment device to control operation of the first medical equipment device; and cause the IDU display to display a first device-specific IDU image comprising graphics or text corresponding to the first device-specific menu items.

An aspect of the disclosure provides a method for touchless control of one or more medical equipment devices in an operating room. The method comprises providing a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; displaying an IDU image corresponding to the three-dimensional control menu, the IDU image providing indicia of any one or more selected menu items; wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices; estimating at least one of a location of a head of the practitioner and an orientation of the head of the practitioner; and adjusting display of the IDU image based at least in part on the at least one of the estimated location of the head of the practitioner and the estimated orientation of the head of the practitioner.

Another aspect of the disclosure provides a system for touchless control of one or more medical equipment devices in an operating room. The system comprises a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor; a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; and an IDU display for displaying an IDU image corresponding to the three-dimensional control menu, the IDU image providing indicia of any one or more selected menu items. The controller is configured to determine selection of any particular one of the menu items and to deliver a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices. The controller is configured, based on input from one or more sensors, to estimate at least one of a location of a head of the practitioner and an orientation of the head of the practitioner and to adjust the display of the IDU image by the IDU display based at least in part on the at least one of the estimated location of the head of the practitioner and the estimated orientation of the head of the practitioner.

Another aspect of the disclosure provides a method for touchless control of one or more medical equipment devices in an operating room. The method comprises providing a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by a practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; projecting an IDU image corresponding to the three- dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items; wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices; obtaining an estimate of a profile of the non-planar projection surface; estimating a viewing vector of the practitioner to the projection surface; and pre-adjusting the IDU image prior to projecting the IDU image, the pre-adjustment based at least in part on the estimated profile of the non-planar projection surface and the estimated viewing vector.

One aspect of the disclosure provides a system for touchless control of one or more medical equipment devices in an operating room. The system comprises a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor; a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item and detected by the controller based on the 3D optical data; an IDU display for displaying an IDU image corresponding to the three-dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items, wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices. The controller is configured, based on input from one or more sensors, to estimate a profile of the non-planar projection surface. The controller is configured, based on input from one or more sensors, to estimate a viewing vector of the practitioner to the non-planar projection surface. The IDU display is configured to pre-adjust the IDU image prior to projecting the IDU image, the pre-adjustment based at least in part on the estimated profile of the non-planar projection surface and the estimated viewing vector.

One aspect of the disclosure provides a method for touchless control of one or more medical equipment devices in an operating room (OR). The method comprises providing a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; projecting an IDU image corresponding to the three- dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items; wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices; providing one or more 3D optical sensors which are mounted to a robotic positioning system for at least one of moving and orienting the one or more 3D optical sensors; and performing at least one of moving and orienting the robotic positioning system and capturing 3D optical data corresponding to a region of interest in the OR and processing the captured 3D optical data to locate and identify the one or more medical equipment devices in the operating room that are controllable using the three-dimensional control menu.

One aspect of the disclosure provides a system for touchless control of one or more medical equipment devices in an operating room (OR). The system comprises one or more 3D optical sensors connected for detecting one or more gestures made by a practitioner in a sensing volume of the one or more 3D optical sensors, the one or more 3D optical sensors mounted on a robotic positioning system for at least one of moving or orienting the one or more 3D optical sensors; a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three- dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item and detected by the controller based on the 3D optical data; an IDU display for displaying an IDU image corresponding to the three-dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items, wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices. The one or more 3D optical sensors are configured to capture 3D optical data corresponding to a region of interest in the OR and process the captured 3D optical data to locate and identify the one or more medical equipment devices in the OR that are controllable using the three- dimensional control menu.

One aspect of the disclosure provides a method for touchless control of one or more medical equipment devices in an operating room (OR). The method comprises providing a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item; projecting an IDU image corresponding to the three- dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items; wherein selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices; providing an IDU display for projecting the IDU image, the IDU display mounted to a robotic positioning system for at least one of moving and orienting the IDU display; performing at least one of moving and orienting the robotic positioning system, and projecting the IDU image onto a first surface; and after receiving an indication that the first surface is undesirable or determining that a practitioner has moved within the OR, performing at least one of moving and orienting the robotic positioning system, and projecting the IDU image onto a second surface.

One aspect of the disclosure provides a system for touchless control of one or more medical equipment devices in an operating room (OR). The system comprises a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor; a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item and detected by the controller based on the 3D optical data; an IDU display for displaying an IDU image corresponding to the three-dimensional control menu onto a non-planar projection surface, the IDU image providing indicia of any one or more selected menu items, the IDU display mounted to a robotic positioning system for at least one of moving and orienting the IDU display. Selection of any particular one of the menu items results in delivering a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices. The IDU display is configured to project the IDU image onto a first surface. The controller is configured to receive an indication that the first surface is undesirable or determining that a practitioner has moved within the OR, and upon receiving such an indication or making such a determination, the IDU display is configured to project the IDU image onto a second surface.

Further aspects of the invention and features of specific embodiments of the invention are described herein.

The accompanying drawings illustrate non-limiting example embodiments of the invention.

Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive sense.

Aspects of the invention provide systems and methods for touchless control or other interaction with medical equipment, tools and/or the like in a medical (e.g. surgical) environment using hand motions (e.g. gestures and/or the like). Medical practitioners interact with one or more adjustable menus defined in volumetric spatial regions located near to (e.g. within arm's reach of) the practitioner. The space in which the one or more menus are located may be referred to as the workspace and the one or more menus may be referred to as the 3D control menus. A practitioner may interact with the 3D control menu (e.g. to select a menu item or to otherwise interact with the 3D control menu) using hand motions based on the configuration (gestures like pointing, finger-tapping, etc.), location, or movement of a practitioner's hand(s) and/or finger(s). The 3D control menu comprises a plurality of volumetric spatial regions (each such spatial region corresponding to a menu item, for example) within which the practitioner's hand(s) may perform suitable hand motion(s) for interaction with the 3D control menu. The 3D control menu may be implemented by a suitably configured controller which receives data from at least one 3D optical sensor and performs a machine vision algorithm that processes optical data received from the 3D optical sensor and interprets that optical data as interaction of the practitioner with the 3D control menu. The controller may be additionally connected, in communication with or otherwise configured to control medical equipment based on the practitioner's interaction with the 3D control menu.

Typically, a practitioner's interaction with the 3D control menu will involve selection of one or more menu items which may in turn result in the delivery of suitable control commands to medical equipment. As discussed above, menu items may correspond to volumetric spatial regions (rather than to physical objects). Consequently, it can be desirable for the practitioner to receive some feedback to indicate that they have effectively selected or otherwise interacted with a menu item. To aid the practitioner's interaction with the 3D control menu, particular embodiments of the invention comprise an interaction display unit (IDU) which displays a visual depiction of the 3D control menu's current configuration (including, for example, displayed indications of any selection, or other interaction, with menu items) to provide useful real-time feedback to notify the practitioner about selections and menu interactions. The IDU may be embodied in a number of different ways. By way of non- limiting example the IDU may comprise a projector which may be used to project the 3D control menu on a suitable surface (e.g. an operating table which may or may not have a patient located thereon, the practitioner's hand, an operating room side table and/or the like). As another non-limiting example, the IDU may comprise a physical display with a depiction of the 3D control menu and a depiction of the practitioner's hand or some other suitable pointing device graphic over the menu. Such a display may be integrated within an existing display used in the medical procedure being performed or may be separately implemented on an independent display. The display may optionally be implemented in wearable technology such as smart glasses, smart watches, augmented/virtual reality headsets and/or the like. In a still further non-limiting example, the IDU comprises one or more projectors that are part of an augmented reality headset for virtual depictions of the 3D control menu. Each of these exemplary IDU formats can be configured to provide indicia which inform the practitioner of selected menu items and/or positioning of their hand(s) and/or finger(s) within or relative to the workspace.

In some embodiments, the IDU may additionally or alternatively display medical image data, such as radiological images, angiography images, or other images of the patient's body. Such medical image data could alternatively be displayed on a display separate from the IDU (e.g. on a separate projector screen, television, monitor, or the like). System Overview.

<FIG>schematically depicts a system <NUM> for touchless control of medical equipment 12A, 12B, 12C (collectively and individually, medical device <NUM> or equipment <NUM>) according to a particular embodiment. System <NUM> comprises one or more 3D optical sensor unit(s) <NUM> which may be used to obtain 3D optical information about objects in workspace <NUM>, including, in particular the body parts of practitioner <NUM> (e.g. hand(s) 25A, finger(s) and/or head 25B of practitioner <NUM>). By way of non- limiting example, 3D optical sensor unit(s) <NUM> may comprise stereo cameras, Time of Flight (ToF) cameras, LIDAR sensors and/or the like. System <NUM> also comprises an IDU display device <NUM> which provides a visual guide and feedback to practitioner <NUM>, as discussed above. In the particular example embodiment shown in <FIG>, IDU display device <NUM> comprises a projector 16A, which projects an IDU image <NUM> corresponding to the 3D control menu onto the surface <NUM> of an operating table <NUM> on which a patient <NUM> is located. In other embodiments, other forms of IDU display device may display IDU image in different manners (e.g. in a physical display or as a virtual reality object) as described above.

System <NUM> also comprises a system controller <NUM> (also referred to as controller <NUM>). System controller <NUM> may be implemented by, or may otherwise comprise, one or more programmable data processes suitably configured using applicable software, as described elsewhere herein. In the illustrated <FIG>embodiment, IDU display device <NUM>, 3D optical sensor(s) <NUM> and controller <NUM> work together to provide a 3D control menu <NUM> in workspace <NUM> located just above surface <NUM> of operating table <NUM>. Control menu <NUM> comprises one or more menu items 19A, 19B. 19n (collectively, and individually menu items <NUM>), each menu item <NUM> corresponding to a volumetric spatial region in workspace <NUM>. Practitioner <NUM> may interact with 3D control menu <NUM> (e.g. to select a menu item <NUM> or to otherwise interact with the 3D control menu <NUM>) using hand motions based on the configuration (gestures like pointing, finger-tapping, etc.), location, or movement of a practitioner's hand(s) and/or finger(s).

System controller <NUM> also uses 3D optical data received from 3D optical sensor(s) <NUM> to control medical equipment <NUM> via control modules 32A, 32B, 32C (collectively and individually, control modules <NUM>). Control modules <NUM> perform the task of interfacing between controller <NUM> and various types of medical equipment <NUM>. Specifically, control modules <NUM> receive commands from controller <NUM> and use various forms of communications interface(s) and protocol(s) to provide particular commands to equipment <NUM> to thereby control or otherwise interact with equipment <NUM>. By way of non-limiting example, control modules <NUM> may comprise Bluetooth communications protocols, USB dongles, LAN communications interfaces, WiFi communications interfaces, data communication protocols/means (e.g. serial com interfaces) and/or the like. In some embodiments, some or all of control modules <NUM> may be implemented in whole or in part by controller <NUM>.

System <NUM> (and methods implemented by system <NUM>) allow practitioner <NUM> to touchlessly interact with and control multiple devices and equipment <NUM> located inside operating room <NUM> or remote to operating room <NUM>. Non-limiting examples of such devices and equipment <NUM> include a medical image display device such as a Picture Archiving and Communication System (PACS) workstation, intraoperative radiology image workstation, surgical lights, a patient bed (operating table), patient diagnostic equipment, a radiology image acquisition system, fluoroscopy equipment (e.g. a C- Arm), other types of medical imaging systems, drug delivery systems, robotic surgical equipment, robotic surgical assistance equipment, control panels for the control of other medical equipment and/or the like. One specific type of equipment <NUM> which may be controlled by system <NUM> is a control interface (e.g. a GUI, a touch panel interface and/or the like) for controlling other medical equipment. Currently, many such devices <NUM> are outside the direct control of practitioner <NUM> during a procedure, because they are not operable while remaining sterile. Using system <NUM>, medical practitioner <NUM> would no longer need to scrub out of the sterile environment in which the procedure is being performed or communicate with technicians located outside of the sterile environment to control such devices <NUM> nor would medical practitioner <NUM> need to communicate with or delegate tasks to a technician or nurse inside the sterile environment. Removing such distractions may thus aid practitioner <NUM> in maintaining focus on the procedure being performed.

<FIG>schematically depicts a system <NUM><NUM> for touchless control of medical equipment <NUM> according to another example embodiment. System <NUM><NUM> is similar in many respects to system <NUM> of <FIG>and similar reference numerals are used to refer to similar components. System <NUM><NUM> comprises two 3D optical sensors 14A, 14B, one of which is configured to sense 3D information in workspace <NUM> (including the locations of the hands 25A of practitioner <NUM>) and the other one of which is configured to sense 3D information corresponding to the location of the head 25B of practitioner <NUM>. System <NUM><NUM> also comprises three IDU display devices 16A, 16B, 16C (collectively and individually IDU display devices <NUM>) which display IDU images 18A, 18B, 18C (collectively and individually IDU images <NUM>). In practice, not all of IDU display devices <NUM> are necessary, but a number of IDU display devices <NUM> and their corresponding IDU images <NUM> are shown in <FIG>for the purpose of explanation. In the example embodiment shown in <FIG>, IDU display device 16A comprises a projector 16A, which projects an IDU image 18A corresponding to the 3D control menu <NUM> and menu items <NUM> onto the surface <NUM> of an operating table <NUM> on which a patient <NUM> is located. In the example embodiment shown in <FIG>, IDU display device 16B comprises a display 16B (which is actually a piece of medical equipment <NUM> used to display medical images), wherein an IDU image 18B corresponding to the 3D control menu is overlaid on the display image. In the example embodiment shown in <FIG>, IDU display device 16C comprises a dedicated IDU display 16C which displays an IDU image 18C corresponding to the 3D control menu.

To properly locate and orient IDU images <NUM>, it may be desirable to establish a world coordinate frame in physical space. Such a coordinate frame may provide the coordinate frame to which all position and orientation data are referenced. The world coordinate frame may be provided by placing one of 3D optical sensors 14A or 14B, or another suitable sensor or marker, in a known physical position such as at the base of robotic arm <NUM>, on a camera projector mount, at the base of operating table <NUM>, and/or the like. This physical position may then be defined as the origin of the coordinate frame, for proper position referencing of all IDU images, 3D control menu components, and medical equipment in the room.

System <NUM><NUM> also comprises a system controller <NUM> (also referred to as controller <NUM>), which receives 3D optical data from 3D optical sensors 14A, 14B and uses such data to control medical equipment <NUM> via control modules <NUM>. In the illustrated example embodiment of <FIG>, medical equipment <NUM> being controlled includes surgical light 12A, C-Arm 12B, bed 12C and medical image display 12D, which are respectively interfaced by control modules 32A, 32B, 32C, 32D.

<FIG> schematically depicts a system <NUM> for touchless control of medical equipment <NUM> according to another example embodiment. System <NUM> is similar in many respects to systems <NUM>, <NUM><NUM> of <FIG>, <FIG> and similar reference numerals are used to refer to similar components. System <NUM> differs from systems <NUM>, <NUM><NUM> in that a number of the components of system <NUM> are integrated into an augmented reality (AR) headset <NUM> - also referred to as AR goggles <NUM>, virtual reality (VR) headset <NUM> or VR goggles <NUM>. Specifically, referring to <FIG>, 3D optical sensor(s) <NUM>, IDU display device <NUM> and optionally system controller <NUM> and any components of control modules <NUM> implemented by system controller <NUM> may be integrated into AR headset <NUM>. The functionality of optical sensors <NUM> integrated into AR headset <NUM> may be similar to that of optical sensors <NUM> described elsewhere herein. Specifically, such optical sensors <NUM> may collect 3D optical data, except such 3D optical data may be from the perspective of practitioner <NUM>, which may include the locations and/or orientations of the hands 25A, fingers and/or head 25B of practitioner <NUM>. The 3D optical data may be transformed to the world coordinate frame described above in order to locate and orient the data within OR <NUM>.

IDU display device <NUM> integrated into AR headset <NUM> may provide similar functionality to IDU display devices <NUM> described elsewhere herein. However, IDU display devices <NUM> integrated into AR headset <NUM> may project IDU images <NUM> directly into the eyes of practitioner <NUM>. Such IDU images <NUM> may, but need not necessarily, comprise three-dimensional images. Because IDU display devices <NUM> integrated into AR headset <NUM> project IDU images <NUM> directly into the eyes of practitioner <NUM>, the corresponding IDU images <NUM> may appear wherever practitioner <NUM> is looking and/or adjacent to that location. Similarly, because IDU display devices <NUM> integrated into AR headset <NUM> project images directly into the eyes of practitioner <NUM>, workspaces <NUM>, 3D control menus <NUM> and corresponding menu items <NUM> may be located wherever practitioner <NUM> is looking. In some embodiments, one or more 3D control menus <NUM> may remain fixed in arbitrary positions, either in 3D space or in practitioner's field of view, regardless of where practitioner <NUM> is looking. For example, a 3D control menu <NUM> may remain fixed in a corner of the IDU image <NUM> projected into the eyes of practitioner <NUM>. Practitioner <NUM> may reposition these 3D control menus <NUM> as desired. In the particular case of the illustrated example embodiment of system <NUM> shown in <FIG>, AR headset <NUM> is shown as creating a 3D control menu 17A and IDU image 18A in workspace 15A for controlling surgical light 12A, a 3D control menu 17C and IDU image 18C in workspace 15C for controlling operating table 12C and a 3D control menu 17D and IDU image 18D is workspace 15D for controlling medical image display 12D. In other embodiments, system <NUM> can create additional or alternative 3D control menus <NUM> and IDU images <NUM> in workspaces <NUM> for controlling additional or alternative medical equipment <NUM>. In some embodiments as is the case in <FIG>, 3D control menus <NUM> and IDU images <NUM> are created in space. In some embodiments, 3D control menus <NUM> and IDU images <NUM> may be projected onto suitable surfaces within OR <NUM>. Techniques for interacting with 3D control menus <NUM> which are created by AR headset <NUM> may be similar to interacting with projected 3D control menus <NUM> described elsewhere herein.

In the illustrated embodiment, system <NUM> is shown as comprising one or more optional additional 3D optical sensors <NUM> located external to AR headset <NUM>. Such additional 3D optical sensors <NUM> can be used to locate practitioner <NUM> within OR <NUM> and/or the head 25B of practitioner <NUM> in OR <NUM>. Such optional additional optical sensors <NUM> can also be used to locate and/or identify medical equipment <NUM> in OR <NUM>, to track humans in OR <NUM> or to otherwise construct a 3D model of OR <NUM> or a relevant portion of OR <NUM>, as discussed in more detail below.

In some embodiments, AR headset <NUM> alone or in combination with additional optical sensor <NUM> detects the location and/or orientation of the head 25B of practitioner <NUM> and system controller <NUM> may determine a particular 3D control menu <NUM> to display to practitioner <NUM> based on the head 25B of practitioner <NUM> being oriented toward a particular component of medical equipment <NUM>. For example, if system controller <NUM> determines (based on information from AR headset <NUM> and/or additional optical sensor <NUM>) that the head 25B of practitioner <NUM> is oriented toward medical image display 12D, then controller <NUM> may elect to display 3D control menu 17D (which may be specific to controlling medical image display 12D) and a corresponding IDU image 18D into a suitable workspace 15D, but if system controller <NUM> determines (based on information from AR headset <NUM> and/or additional optical sensor <NUM>) that the head 25B of practitioner <NUM> is oriented toward light 12A, then controller <NUM> may elect to display 3D control menu 17A (which may be specific to controlling light 12A) and a corresponding IDU image 18A into a suitable workspace 15A. In some embodiments, a toggle may be provided for each 3D control menu <NUM> and IDU image <NUM>, so that practitioner <NUM> may elect whether or not to have such 3D control menu and IDU image <NUM> presented. In some embodiments, system controller <NUM> may elect not to display any 3D control menu <NUM> (based on information from AR headset <NUM> and/or additional optical sensor <NUM>). For example, system controller <NUM> may elect not to display any 3D control menu <NUM> when the head 25B of practitioner <NUM> is oriented toward a body part being operated on.

In some embodiments, AR headset <NUM> may additionally or alternatively comprise suitable hardware and software to implement gaze tracking and such gaze tracking techniques may also be used in electing to display particular 3D control menus and IDU images <NUM> based on gaze direction toward a particular component of medical equipment or toward a body part that is being operated on. One, among many, suitable gaze tracking techniques is described, for example, in <CIT>, which is hereby incorporated herein by reference. Gaze tracking may permit more granular control than would be possible by just tracking head location and orientation. Implementation of eye tracking may additionally or alternatively allow estimate of gaze depth. Gaze depth estimation can be used for knowing when practitioner is looking at 3D control menu <NUM> located in a workspace <NUM> between practitioner <NUM> and a component of medical equipment <NUM> or to the medical equipment <NUM> itself.

In some embodiments, AR headset <NUM> may additionally or alternatively comprise suitable hardware and/or software to implement orientation control. For example, AR headset <NUM> may comprise one or more sensors for sensing and/or interpreting the movement or orientation of hand(s) 25A of practitioner <NUM>. A particular movement/orientation may correspond to the selection of a particular item <NUM> on 3D control menu <NUM>, or may manipulate (e.g. rotate or translate) an IDU image <NUM>. The orientation sensors may additionally or alternatively be provided on a handheld device or gloves worn by practitioner <NUM>. In some embodiments, practitioner <NUM> may move between selections of items <NUM> on 3D control menu <NUM> by rotating their hand(s) 25A in a suitable direction. Such rotation may avoid the need for practitioner <NUM> to move their hand(s) 25A between volumetric spatial regions of 3D control menu <NUM>. As a consequence of controlling the menu, orientation control may be used to control any of the mentioned equipment in the OR. In specific cases where the controlled device is an imaging workstation, this orientation control modality can be used to directly manipulate the displayed image (eg. 3D Rotation, transformation, etc. of the image).

By means of 3D optical sensing via one or more 3D optical sensors <NUM>, the systems described herein determine the real-time position and orientation of objects of interest within the fields of view of sensors <NUM>. Primarily, the practitioner's hand 25A (for menu interactions) and obstructions within the workspace <NUM> (for menu configuration) are of interest. By increasing the field of view of sensor <NUM> or providing additional 3D optical sensor(s) <NUM>, the torso and/or head 25B of practitioner <NUM> may also be tracked to determine the location of practitioner <NUM> and the practitioner's head location and/or orientation.

System controller <NUM> comprises one or more processing units which connect to 3D optical sensors <NUM>, control modules <NUM> and IDU display device <NUM>. System controller <NUM> processes 3D data from 3D optical sensors <NUM> and determines the location and orientation of objects within workspace <NUM>, which may include the hand(s) 25A, torso and/or head 25B of practitioner <NUM>. Based on this 3D optical information, system controller <NUM> may send commands (corresponding to the practitioner's interactions with 3D control menu <NUM>) to medical equipment <NUM> via appropriate control module(s) <NUM>.

Control modules <NUM> interface with medical equipment <NUM> to pass on commands from system controller <NUM>. Specifically, control modules <NUM> may receive electronic commands from system controller <NUM> via any suitable wired or wireless communication protocol, translate such commands into specific control commands corresponding to medical equipment <NUM> and communicate these specific control commands to medical equipment <NUM>. That is, control modules <NUM> may be retrofitted to legacy medical equipment <NUM> (in addition or in the alternative to the existing control/communication interface(s) of the legacy medical equipment <NUM>). In some embodiments, some portions or all of control modules <NUM> may be implemented by system controller <NUM>. In some embodiments, some portions of, or all of, control modules <NUM> may be implemented within medical equipment <NUM>. Where all of a control module <NUM> is implemented within a medical device, system controller <NUM> can interface natively with such medical equipment <NUM>.

A non-limiting example of a control module <NUM> is a USB dongle, which may plug into a radiology image workstation 12D. The dongle may receive commands wirelessly from system controller <NUM> and may translate these commands into mouse and keyboard commands, which are sent to radiology image workstation 12D to manipulate images displayed thereon.

In some embodiments, control modules <NUM> may comprise displays which may be located relatively proximate to their respective components of medical equipment <NUM>. Such displays may be part of the corresponding medical equipment <NUM> or may be retrofitted onto legacy medical equipment <NUM>. In such embodiments, the displays of control modules <NUM> can be used as IDU display devices <NUM> for displaying IDU images <NUM> back to practitioner <NUM>, so that the practitioner <NUM> can relatively easily control a component of medical equipment <NUM> when looking at the medical equipment <NUM> (or more precisely at the display of the control module <NUM> corresponding to the medical equipment <NUM>).

A number of exemplary and non-limiting types of medical equipment <NUM> and interfacing control modules <NUM> include the following.

Medical imaging devices <NUM>, such as the C-Arm and/or other intra-op imaging devices.

Medical imaging devices <NUM> (such as the C-Arm and/or other medical imaging devices <NUM>) may comprise their own control panel <NUM> (e.g. C-Arm control panel 13B and bed control panel 13C as shown in <FIG> that may be connected to the device or hanging down from a ceiling mount and moveable within the operating room. A control module <NUM> can be interfaced with the control panels <NUM> of these imaging devices <NUM> to enable the systems described herein to control the position of the imaging device <NUM> and to activate the device <NUM> for procuring images of patient <NUM> during surgery. o Robotic C-Arm manipulation - Robotic C-Arms used to support medical imaging equipment are provided by a number of different medical device manufacturers, including by way of non-limiting example, the Artis Zeego™ by Siemens Healthcare, the Veradius Neo™ by Philips and/or the like.

C-Arm mounted imaging devices <NUM> may provide extended range and positioning capabilities and can also be controlled in the same or similar manner by the systems described herein. Many models allow for particular configurations to be set up prior to a surgery - allowing various types of image procurement of patient anatomy information.

The systems described herein could enable practitioner <NUM> to select from these configurations using a simple menu interface.

C-Arms and the imaging systems and/or other medical equipment mounted thereon may be permanent fixtures in an OR <NUM> or may be mobile, such that the C-Arm can be wheeled into (or otherwise moved into) OR <NUM>.

Such image navigation workstations <NUM> are usually controlled by either a standard or specialized keyboard. For these situations the corresponding control module <NUM> may take the form of a USB dongle which may control the image navigation workstations <NUM> via the USB HID protocol. Non-limiting examples of commands that the systems described herein could effect through such control modules <NUM> include: image manipulations (including brightness, contrast, pan, zoom, scroll, rotation and other angular orientations in the 2D and 3D space and/or the like), adjustments of viewing configurations, image- based measurements (geometric and spatial), user-drawn markings (for communication and referencing), and selection of various image-sets.

Some ORs have a composite screen which can display outputs from several devices on a single screen in a customizable configuration. In other ORs several hanging monitors can be switched to display output from various devices. The systems described herein can control the display configuration through a control module <NUM> interfaced with the control panels <NUM> for such displays. Through this control module <NUM> the systems described herein can facilitate presentation of desired data on the monitor of choice.

Sometimes, touch screen or computer panels inside the OR allow quick access to patient medical data. In some cases, it may be useful to allow practitioner <NUM> to have sterile control of such a panel to ascertain certain facts about patient <NUM>. The systems described herein could allow practitioner <NUM> to use the 3D control menu to navigate sections of the patient data and scroll through as desired. Such patient data could be displayed on a physical screen or a virtual screen as a projector or VR glasses.

Medical equipment <NUM> controllable by the systems described herein may include such medical equipment. The hand-operated input devices of such medical equipment may be bypassed using 3D control menus <NUM> and control modules <NUM> which generate commands (based on practitioner interaction with 3D control menus <NUM>) which replace the commands of the hand-operated input devices. For example, practitioner <NUM> may control any means of a floor- or ceiling-mounted articulated arm, including those of surgical robots, using the systems described herein.

The 3D control menu <NUM> that practitioner <NUM> interacts with is virtual/invisible. The purpose of the IDU (and specifically the IDU display device <NUM> and its corresponding IDU image <NUM>) is to visually inform practitioner <NUM> of the location and configuration of the 3D control menu <NUM>, as well as to provide feedback regarding the interaction of practitioner <NUM> with the 3D control menu <NUM>. This allows practitioner <NUM> to focus on the IDU, and not their hand(s) 25A. By way of non-limiting example, IDU image <NUM> may provide indicia indicative of the selection of a particular menu item <NUM> (e.g. the menu item may change color when the hand 25A of practitioner <NUM> hovers over the particular menu item). As discussed above, IDU image <NUM> may also comprise medical image data. IDU display device <NUM> and corresponding IDU images <NUM> may take various forms as discussed herein. Details of a number of embodiments are described further below.

One modality of the IDU comprises an IDU display device <NUM> which projects an IDU image <NUM> (comprising icons representing menu interface items <NUM> of 3D control menu <NUM>) on a surface within workspace <NUM>. This is the case, for example, with IDU display device <NUM> and IDU image <NUM> of system <NUM> shown in <FIG>and IDU display device 16A and IDU image 18A of system <NUM><NUM> shown in <FIG> The icons of projected IDU image <NUM> may indicate the locations of the volumetric spatial regions corresponding to menu interface items <NUM> of 3D control menu <NUM> and associated with corresponding controls. When practitioner <NUM> moves their hand 25A into the volumetric region corresponding to a particular menu item <NUM>, the corresponding icon may change color or otherwise change to provide some visual feedback indicator to practitioner <NUM> of the location of his or her hand 25A within the volumetric spatial region. A finger-tap gesture over, or otherwise proximate to, a given icon can then be detected to actuate controls associated with the menu item <NUM>. Other similar gestures may be used for other common controls (activation gesture, pause gesture, etc.).

In the illustrated embodiments of <FIG>and <FIG>, IDU display device <NUM> projects onto the surface <NUM> of an operating table <NUM> on which a patient <NUM> is located. This is not necessary. A projection type IDU display device <NUM> may additionally or alternatively project IDU image <NUM> on any given surface in OR <NUM>. Such a surface could be a flat panel next to one or more components of the equipment <NUM> under control. Such a surface may be covered with a drape or sterile covers during the surgery (i.e. projecting onto those drape regions). Where IDU display device <NUM> projects IDU image <NUM> onto an irregular surface (e.g. the surface <NUM> of an operating table <NUM>), the projected IDU image <NUM> may be augmented using a method for projection correction (described elsewhere herein), such that practitioner <NUM> sees an undistorted IDU image <NUM>.

In some embodiments, the color and/or pattern of IDU image <NUM> may be adjusted to enhance contrast from the surface onto which IDU image <NUM> is projected for better visibility. For example, if the surface has blood splatter on it, an alternating pattern or the like can be used in IDU image <NUM> to enhance the contrast over the non- uniformly-colored surface.

In some embodiments, the IDU is implemented using haptic feedback, for example by way of ultrasonic waves. In such embodiments, information from the IDU is relayed to practitioner <NUM> via their sense of touch, rather than their sense of vision.

Another modality of the IDU comprises an IDU display device <NUM> which projects an IDU image <NUM> onto the hand 25A of practitioner <NUM> within workspace <NUM>. This embodiment may, but does not typically, project a representation of the 3D control menu <NUM> onto a surface. Instead, this embodiment may involve projection of feedback only onto hand 25A of practitioner <NUM> when it moves through the volumetric spatial regions associated with the various menu items <NUM>. When practitioner <NUM> moves their hand 25A into a given menu region, an icon representing the corresponding control may be projected onto the hand 25A of the practitioner <NUM>. The projected icon may be augmented according to the curvature of hand 25A to appear undistorted to practitioner <NUM>. By sweeping their hand 25A laterally in front of them, practitioner <NUM> can move between volumetric spatial regions of the 3D control menu <NUM> and actuate the corresponding menu items <NUM> using suitable gestures (e.g. point, finger-tap and/or the like).

Another modality of the IDU comprises an IDU physical display <NUM> which displays an IDU image <NUM> comprising a 2D representation of the 3D control menu <NUM>. The display that displays IDU image <NUM> may comprise a dedicated display or the 2D representation of the 3D control menu <NUM> may be overlaid as a "GUI overlay" on some other display (such as a display for displaying medical images), which display itself might be medical equipment <NUM> controlled by the systems described herein. The GUI overlay (for example, IDU image <NUM> as shown in <FIG>) may allow for markers to be drawn or placed over a medical image in locations selected by practitioner <NUM>. A depiction of the hand 25A of practitioner <NUM> may also be displayed to inform practitioner <NUM> of the proximity and location of their hand 25A relative to the volumetric spatial regions in the 3D control menu <NUM>. The 2D representation of IDU image <NUM> can display special icons or animations to feedback information to practitioner <NUM> about hand motions (e.g. gestures) performed in real-time.

IDU physical display 16B and corresponding IDU image 18B of system <NUM><NUM> depicted in <FIG>represents an example of an embodiment where IDU image 18B comprises a GUI overlay on a medical image display 12D, which display 12D is itself medical equipment 12D controlled by the system <NUM><NUM>. In embodiments where the GUI overlay corresponding to IDU image <NUM> can be overlaid on the same display 12D where medical images are displayed in the OR, practitioner <NUM> will have the choice to look down at the projected 3D control menu for interactions or to use the GUI overlay corresponding to IDU image <NUM> on the medical image display to do the same. The IDU physical display <NUM> may display an IDU image <NUM> comprising a representation 18A of hand 25A of practitioner <NUM> (or any other object being used for interaction - e.g. a scalpel) and its relative position to the volumetric menu regions (as shown in <FIG>).

In such embodiments, practitioner <NUM> need not to look down to see the projected 3D control menu <NUM> while they are trying to focus on controlling OR equipment (e.g. navigating radiology images on a medical image display 12D or rotating the C-arm 12B). When practitioner <NUM> moves their hand 25A within workspace <NUM>, the GUI overlay would display the relative position of hand representation 18A within IDU image <NUM> in real-time. The same or similar visual feedback techniques used for the projected IDU image <NUM> can be employed for the physical display IDU image <NUM>. For example, highlighting of a particular icon upon selection could also be reflected on the IDU image <NUM> shown by the physical display <NUM>. It should be noted that in this case IDU image <NUM> and the projected 3D control menu <NUM> work independent of each other. The physically displayed IDU image <NUM> does not need projected 3D control menu <NUM> to function and vice versa.

As mentioned above, it is not necessary that the physical IDU display <NUM> be a display that is used for other purposes or that the IDU display <NUM> be a component of medical equipment <NUM> controlled by the systems described herein. Instead, in some embodiments, IDU display <NUM> may comprise a dedicated display for displaying IDU image <NUM>. This is the case, for example, with dedicated IDU display 16C and corresponding IDU image 18C of system <NUM><NUM> shown in <FIG> Such a dedicated IDU display <NUM> may be positioned in any convenient location in OR <NUM> for this purpose (ideally adjacent to medical equipment <NUM> under the control of the systems described herein for ease of control of such equipment <NUM>).

As discussed above, control modules <NUM> may comprise displays which may be located relatively proximate to their respective components of medical equipment <NUM> and such displays can be used as physical IDU displays <NUM> for displaying IDU images <NUM>, either as dedicated IDU displays <NUM> or as an IDU image <NUM> overlaid on a display <NUM> that also displays other information (e.g. medical image data).

<FIG> schematically depicts a practitioner <NUM> wearing AR headset <NUM> and a 3D control menu <NUM> and corresponding IDU image <NUM> according to an example embodiment. As discussed above, where practitioner <NUM> wears an AR headset <NUM>, IDU display device <NUM> is integrated into AR headset <NUM> and projects AR image <NUM> directly into the eyes of practitioner <NUM>. As such, IDU images <NUM> can be located wherever practitioner <NUM> is looking as shown in <FIG>. In some embodiments, two separate IDU images may be projected into the eyes of practitioner <NUM>, so that practitioner <NUM> sees 3D IDU images <NUM>. As also discussed above, the head orientation and/or gaze orientation of practitioner <NUM> can be used to select between 3D control menus <NUM> and corresponding IDU images <NUM> to display (or whether to display such menus/images at all). In some embodiments, practitioner <NUM> can "pin" 3D control menus <NUM> and corresponding IDU images <NUM> to particular locations, so that such menus/images only appear when the head orientation and/or gaze orientation of practitioner <NUM> is directed to that location.

A system like system <NUM> which makes use of an AR headset <NUM> may be advantageous in some circumstances, because suitable 3D control menus <NUM> and corresponding IDU images <NUM> can be located in the direction of corresponding medical equipment <NUM> which is intuitive and easy for practitioners to use. However, physical 3D control menus <NUM> and corresponding IDU images <NUM> can be advantageous where multiple practitioners <NUM> are working cooperatively. If a projected or physical form of 3D control menu <NUM> and corresponding IDU image <NUM> are used, it is easier to communicate and collaborate, because one practitioner <NUM> can see what the other is doing without needing any extra equipment. Projection Correction.

As discussed above, in some embodiments, IDU display device <NUM> projects an image corresponding to a 3D control menu <NUM> and/or a corresponding IDU image <NUM> onto a surface, such as, for example, the surface <NUM> of an operating table <NUM> on which a patient <NUM> may be located. A typical operating table <NUM> in a busy OR <NUM> is hardly an ideal environment to act as a projection surface - due to geometrical irregularities and deformations (e.g. drape wrinkles, the presence of patient <NUM>, the presence of surgical tools, etc.). Such an irregular surface may cause geometric distortions in any image projected thereupon, making the projected image appear warped from the perspective of practitioner <NUM>. Such warping of images corresponding to control menus <NUM> and/or IDU images <NUM> can hinder system usage and user clarity. For example, a practitioner <NUM> may be performing a surgical procedure on a patient's chest area. The upper half of the patient's body may accordingly be surrounded by various surgical tools and devices. In this situation, the system would be most optimally placed such it that can project 3D control menu <NUM> and/or IDU image <NUM> over the patient's lower torso/legs. However, whenever projection is done on irregular or curved surfaces, the projected IDU image <NUM> may be warped and difficult to see and, consequently, the corresponding 3D control menu <NUM> may be difficult to use.

Aspects of the invention provide methods and systems for homographic correction of such warping of projected images. Using such techniques, 3D control menu <NUM> and/or IDU image <NUM> may be pre-adjusted such that projected graphic content (projected adjusted images) appear undistorted from the perspective of practitioner <NUM>. These techniques compensate for deformations and irregularities on the projection surface. While these techniques are applicable to any projection surface, these techniques are described herein for the case where the projection surface is the surface <NUM> of an operating table <NUM> on which patient <NUM> may be located without loss of generality. Similarly, these techniques assume that the projector that is projecting the image is an IDU display device <NUM> of the type shown in <FIG> A and at 16A in <FIG>, without loss of generality.

When projecting 3D control menu <NUM> and/or IDU image <NUM> on an irregular projection surface <NUM>, no matter how the projector (e.g. IDU display device) <NUM> is positioned and oriented with respect to surface <NUM>, the resulting image will look distorted from the practitioner's point of view, in the absence of pre-adjustment. However, there is one point in space from which the projected image looks perfectly linear and undistorted (i.e. non-warped) - the position of projector <NUM>. To see a non- distorted projection image (of 3D control menu <NUM> and/or IDU image <NUM>) from an arbitrary viewpoint (which may be considered to be the viewpoint of practitioner <NUM>), the original image may be pre-adjusted to provide an adjusted image and the adjusted image may be projected such that it appears as if it was projected from the arbitrary viewpoint.

Projection surface <NUM> may be scanned using 3D optical sensor(s) <NUM> and, based on the 3D data relating to the projection surface <NUM> obtained from sensors <NUM>, the curvature of projection surface <NUM> may be characterized. Parts of the original image that align with regions of positive curvature on projection surface <NUM> may be calibrated to the same degree of distortion caused by equivalent negative curvature, and vice-versa for regions of projection surface <NUM> exhibiting negative curvature. The resulting adjusted image, when projected onto projection surface <NUM> appears linear or otherwise non-distorted and non-warped from the perspective of practitioner <NUM>.

The location of practitioner <NUM> and the point-of-view of practitioner <NUM> are parameters used to determine how to adjust the original image and to thereby provide the adjusted image. Specifically, a projection from the viewer's perspective may be simulated to determine the adjustments to make to the original image. In some embodiments, information about the location and/or orientation of the head 25B of practitioner <NUM> may be ascertained by one or more optical sensors <NUM> described above. In some embodiments, the location of the head 25B of practitioner <NUM> may be ascertained by one or more optical sensors <NUM> described above and the orientation of the practitioner's head 25B may be estimated from the location. In some embodiments, one or more additional 3D cameras or orientation sensors (e.g. accelerometers, gyroscopes and/or the like) may be used to determine the location and/or orientation of the practitioner's head 25B. Additional details of techniques for head tracking are described below. It will be appreciated that head location and head orientation are in fact proxies for gaze orientation. In some embodiments, gaze tracking techniques such as those described elsewhere herein may be used to determine the perspective by which practitioner <NUM> views 3D control menu <NUM> and/or IDU image <NUM>.

In some embodiments, one or more of 3D optical sensors <NUM> and/or one or more dedicated 3D optical sensors similar to sensors <NUM> may be used to track practitioner <NUM> as he or she moves around OR <NUM> and/or workspace <NUM>, such that the projected 3D control menu <NUM> and/or IDU image <NUM> is always oriented towards practitioner <NUM>. One or more 3D optical sensors <NUM> might also be used to scan all of, or the relevant portions of, OR <NUM>. A suitable machine vision method (e.g. surface feature localization by surface normal estimation) can then be used to perform a curvature analysis on the scanned OR <NUM> for determining the best (e.g. flattest) projection surface <NUM>. Following this, an articulated robotic arm <NUM> might be employed (under the control of system processor <NUM>) to automatically position IDU display projector <NUM> in a suitable location for projecting on the desired projection surface <NUM>. This process of using articulated robotic arm <NUM> is described in more detail below.

The location of a practitioner <NUM> within OR <NUM> may be identified and/or tracked using point clouds generated by one or more 3D optical sensors <NUM> suitably mounted within OR <NUM>. Such 3D optical sensors <NUM> may or may not comprise optical sensors <NUM> used for detecting interactions with 3D control menus <NUM>. Many such methods can be employed in the medical environment. Such algorithmic processes utilize not only preprocessing techniques (for filtering and smoothing the point cloud), but also techniques for analyzing shapes and curvatures known to be apparent for the human head and upper body.

Methods for projection correction according to particular embodiments, which may be performed by system processor <NUM>, may be broken down into two component methods: surface reconstruction; and projection correction. As a part of surface reconstruction, system processor uses 3D data from 3D optical sensors (e.g. 3D optical sensors <NUM>) or any other suitable sensors to construct a mesh representation of projection surface <NUM>.

<FIG> schematically depicts a method <NUM> for surface reconstruction according to a particular embodiment. Method <NUM> may be performed by system controller <NUM>. Method <NUM> begins in block <NUM> which involves capturing raw 3D data using 3D optical sensors <NUM> of the type described herein. The output of block <NUM> comprises a plurality of points in 3D space (referred to as a point cloud or a 3D point cloud) 302A. <FIG> shows an example representation of a point cloud 302A in a particular example. As can be observed from <FIG> and as is typical in block <NUM>, point cloud 302A does not have uniform point density over the various regions of point cloud 302A. There is a desirability, from the perspective of the illustrated embodiment of surface reconstruction method <NUM> of <FIG>, for a relatively more uniform point density in the point cloud.

Method <NUM> then proceeds to block <NUM> which comprises smoothing the point cloud 302A obtained in block <NUM>. Although there are many available smoothing techniques (any of which can be used in block <NUM>), block <NUM> of the currently preferred embodiment uses a moving least squares (MLS) predictive smoothing technique. MLS predictive smoothing is a resampling method that helps to remove and smooth irregularities in point cloud 302A. Such irregularities may be caused, for example, by small distance measurement errors that come from 3D optical sensor(s) <NUM>. The block <NUM> MLS method comprises attempting to modify the points within 3D point cloud 302A to be more regularly distributed - by filling in points in low density regions with interpolations based on the surrounding points.

Method <NUM> then proceeds to block <NUM> which comprises reconstruction of the 3D point cloud which may help to fill in holes in some regions which were not captured with desired density by 3D optical sensors <NUM> in block <NUM>. One particular technique for implementing the block <NUM> reconstruction procedure is referred to as Poisson reconstruction. Poisson reconstruction helps fill in the holes in certain regions that were not captured with sufficient density by 3D optical sensors <NUM>. This block <NUM> procedure may comprise analyzing the curvature of surface regions surrounding any holes (absences of points within the point cloud), and then populating each hole with points such that the change in curvature is minimized. <FIG> schematically depicts an example of the output 3D point cloud 306A of block <NUM> after the predictive smoothing of block <NUM> and the reconstruction of block <NUM>.

Method <NUM> then proceeds to block <NUM> which involves voxel filtering and segmentation. Point cloud 306A output from block <NUM> is dense and is typically populated by more than 3x106 individual points in 3D space. Performing any processing on such a large dataset can be computationally expensive. Thus, from a computational expense perspective, it can be desirable to reduce the size of point cloud 306A, without losing important information about the shape of projection surface <NUM>. This reduction in the size of point cloud 306A may be performed in block <NUM> by voxel filtering. The block <NUM> voxel filtering process may involve the use of a voxel grid filter which returns a point cloud with a smaller number of points which optimally represents the input point cloud as a whole. The block <NUM> voxel grid filter may down-sample the data from point cloud 306A by taking a spatial average, median and/or the like of the points in point cloud 306A. <FIG> schematically depicts an example of the output 3D point cloud 308A of block <NUM> after the block <NUM> voxel filtering process.

Method <NUM> then proceeds to block <NUM> which involves performing an inquiry as to whether a surface can be detected within point cloud 308A output from block <NUM>. Block <NUM> may comprise evaluating the resultant point cloud to determine whether a surface can be ascertained from the resultant point cloud. If no surface is detected (typically because the main cluster of 3D points in the resultant point cloud is too sparse), then method <NUM> proceeds along the block <NUM> NO output back to block <NUM>, where further image data is acquired. On the other hand, if a surface is detected in block <NUM> then method <NUM> proceeds along the block <NUM> YES output to block <NUM>.

Block <NUM> involves removal of outlying points from the point cloud 308A. Due to the nature of some 3D optical sensors (e.g. time of flight sensors), accuracy typically drops along the edges of a captured scene. Therefore, in some embodiments, it can be desirable, for surface estimation, to remove statistically outlying points from point cloud 308A, where such points are sparse and not dense. In some embodiments, the block <NUM> outlier removal may be based on the computation of the distribution of point-to-neighbor distances in the point cloud 308A received in block <NUM>. For each point, block <NUM> may comprise computing the mean distance from the given point in the point cloud to its neighbors. Points whose mean distances are outside a configurable (e.g. user configurable) threshold interval may be removed from the point cloud. This block <NUM> outlier removal process is schematically illustrated in <FIG> which depicts both the outlying points removed from the point cloud in block <NUM> and the remaining point cloud 312A.

Method <NUM> then proceeds to block <NUM> which involves implementing a triangulation process (e.g. a greedy triangulation process or any other suitable triangulation process) to generate a surface mesh 314A from the block <NUM> point cloud 312A. The block <NUM> greedy triangulation process may comprise generating a virtual approximation 314A of the real projection surface <NUM> by connecting points in the point cloud 312A with triangles. The block <NUM> greedy triangulation process works by maintaining a list of points from which the mesh can be grown ("fringe" points) and extending the mesh until all possible points are connected, which results in a triangulated surface mesh 314A. <FIG> depicts an example triangulated surface mesh 314A output from block <NUM>.

Once a surface mesh 314A is obtained in block <NUM>, surface reconstruction method <NUM> is completed and the projection correction method of the illustrated embodiment proceeds to projection correction in block <NUM>. <FIG> illustrates a typical set-up of an exemplary system described herein which is useful for the purposes of explaining the block <NUM> projection correction method. In the <FIG> illustrative example a 3D optical sensor <NUM> and an IDU display device (e.g. projector) <NUM> are placed above projection surface <NUM> (e.g. an operating table <NUM>). Projector <NUM> placed above projection surface <NUM> projects the 3D control menu <NUM> and corresponding IDU image <NUM> of interest, while 3D optical sensor <NUM> (which is placed close to projector <NUM>) captures projection surface <NUM> in 3D.

<FIG> schematically depicts a method <NUM> for implementing projection correction block <NUM> according to a particular embodiment. Method <NUM> begins in block <NUM> which involves obtaining any available data about the location of the head 25B of practitioner <NUM> and the gaze direction of practitioner <NUM>. As discussed elsewhere herein, head orientation of practitioner <NUM> may be used as an estimation of user gaze direction. Head orientation may be detected using a variety of techniques, a number of which are described herein. In some embodiments, head orientation may be estimated based on head location. For example, a vector may be constructed between the estimated head location and the projection surface and it may be assumed that the head orientation of practitioner <NUM> is directed along this vector. In some embodiments, the vector may be determined by detecting (for example, using one or more optical sensors) Purkinje reflections (i.e. glints) in one or both eyes of practitioner <NUM>.

Blocks <NUM> and <NUM> of projection correction method <NUM> may perform a process referred to as a perspective geometry exchange. Once the practitioner's head location and at least an estimate of the practitioner's head orientation are determined in block <NUM> and a surface mesh 314A is determined in block <NUM>, a simulation of the real scene maybe created in a graphical engine using a suitable graphics simulation library. Examples of suitable graphics simulation libraries include OpenGL, DirectX, Unreal Engine and/or the like. It is important to appreciate that the steps of block <NUM> through <NUM> are performed in a computer-generated virtual scene that aims to approximate the real scene, but is not the same as the real scene. Blocks <NUM> through <NUM> involve computational simulations and corresponding steps in such simulations, but no physical projection or sensors are used in the performance of these processing steps.

Surface mesh 314A created in block <NUM> is placed in the virtual scene. A virtual camera <NUM> is placed at the same relative position to the simulated mesh as the real world projector is located relative to the real projection surface in the physical scenario. Then a process which may be referred to as a perspective geometry exchange may be performed in the virtual scene. In the perspective geometry exchange, a virtual projector <NUM> is placed at the viewer's known head location relative to the projection surface in the virtual scene. Simultaneously, a virtual camera <NUM> is placed at the location of the real projector relative to the projection surface in the virtual scene. Because the practitioner's (viewer's) perspective is replaced with a virtual projector <NUM>, this process is called perspective geometry exchange.

Block <NUM> involves placing a virtual projector <NUM> at the location of the practitioner's head in the virtual scene. As discussed above, when a projected image is viewed from the projector location, it will always appear non-distorted, regardless of the shape of the projection surface. The concept behind projection correction method <NUM> is to move, in simulation, a virtual projector <NUM> to the same position and orientation as the practitioner's (viewer's) head. When viewed from the practitioner's (viewer's) perspective, this virtual projection will appear undistorted, no matter the shape of the projection surface captured by the 3D optical sensor. It should be noted that the virtual projector's FOV (i.e. throw ratio) and optical characteristics are completely arbitrary and can be set to anything that suits the scene that has been set up.

Method <NUM> then proceeds to block <NUM> which involves placing a virtual camera <NUM> in the simulated scene at the same location relative to the simulated projection surface as the location of the real projector relative to the real projection surface. Placing the virtual camera <NUM> in the scene completes the process of perspective geometry exchange. It should be noted that the virtual camera's FOV and optical characteristics (such as throw of camera) are preferably the same as those of the real projector. At the conclusion of block <NUM>, the virtual scene has the configuration shown in <FIG>.

Method <NUM> then proceeds to block <NUM>, where, using the virtual projector <NUM> (at the location of the viewer in the simulated scene), the original, undistorted image (e.g. 3D control menu and the corresponding IDU image <NUM>) is projected on the virtual projection surface <NUM> in the simulated scene as shown in <FIG>. As the virtual projector <NUM> is placed at viewer location, from viewer's perspective, this projection would appear undistorted and linear (i.e. non-warped). From any other perspective (including from virtual camera's perspective), the same virtual projection would appear distorted. It should also be noted that <FIG> is shown from some arbitrary perspective that does not belong to either the virtual camera <NUM> or the viewer position. This is done so that reader can better understand the relative positioning of all the components in the simulation.

Method <NUM> then proceeds to block <NUM> which involves capturing the virtual scene from the perspective of the virtual camera <NUM> (and the real projector). The virtual camera's perspective may be captured in block <NUM> in the form of a screenshot. In this block <NUM> screenshot (which is taken from the perspective of the virtual camera <NUM> (and real projector)), the image projected by the virtual projector <NUM> will look distorted, as explained above. However, as this distorted block <NUM> image is the same image that is designed (in block <NUM>) to look correct from perspective of the viewer, this same block <NUM> image, when projected from the real projector, will look undistorted to the viewer. <FIG> depicts a representation of a block <NUM> screenshot showing the distorted image captured by the virtual camera <NUM>. As block <NUM> is the step where the final input for the pre-adjusted image (for real projection) is captured, the virtual camera's FOV and optical characteristics preferably match the real projector, as discussed above. Block <NUM> also completes the steps for simulated projection correction, which may be performed in a suitable graphics simulation environment, such as OpenGL, DirectX, Unreal Engine and/or the like. In some embodiments, a graphics simulation environment is not required and the perspective exchange process can be performed analytically using mathematical representations of the virtual camera <NUM> and projector. The process of translations and rotations (for exchanging camera and projector perspective) may be achieved by corresponding rotation and translation matrices.

Method <NUM> then proceeds to block <NUM> which involves projecting the distorted (adjusted) block <NUM> image using the real projector (e.g. IDU display device <NUM>). From the way the virtual scene is set up, both the virtual camera <NUM> and real projector are located in the same position and share optical characteristics. When the block <NUM> screenshot (the adjusted image) is projected on the real projection surface, it appears undistorted and linear (i.e. non-warped) from the visual perspective of the real viewer.

In some embodiments, a robotic positioning system (e.g. pan & tilt mount, spherical wrist, linear actuator(s), articulated robotic arm and/or the like) is provided which may move and/or re-orient IDU display device <NUM> to change the location of IDU images <NUM> depending on the location of practitioner <NUM>. One or more of sensors <NUM> may additionally or alternative be mounted on the robotic arm, so as to mitigate issues which arise due to sensor positioning and/or occlusion. In some embodiments, the robotic positioning system is provided in the form of a series of linear actuators or the like. In other embodiments, the robotic positioning system is provided as a combination of an articulated robotic arm and a series of linear actuators. For brevity, in this description, such a robotic positioning system may be referred to as a robotic arm without loss of generality.

In system <NUM><NUM> of the <FIG>embodiment, IDU display device 16A and 3D optical sensors 14A, 14B are mounted at the end of a robotic positioning system <NUM> which may move and/or re-orient IDU display device 16A and/or 3D optical sensors 14A, 14B. For example, robotic position system <NUM> may comprise an articulated robotic arm, a series of linear actuators, a combination of an articulated robotic arm and a series of linear actuators, or the like. While not explicitly shown in the block diagram illustration of system <NUM> in <FIG>, IDU display device <NUM> and 3D optical sensors <NUM> of system <NUM> may be mounted at the end of a substantially similar robotic positioning system <NUM>. While not required in system <NUM> of <FIG>, a substantially similar robotic positioning system <NUM> may be used to house or support additional 3D optical sensor(s) <NUM> - i.e. optical sensors <NUM> not included in AR headset <NUM>. In some embodiments, other components (e.g. system controller <NUM>) of any of the systems described herein may be mounted or enclosed in suitable enclosures on robotic positioning system <NUM>, although this is not necessary.

Robotic arm <NUM> permits the components of any of the systems described herein to be maneuvered using a robotic manipulator. <FIG> schematically depicts an articulated robotic arm <NUM> according to a particular embodiment. Arm <NUM> of the <FIG> embodiment is itself mounted on a rolling stand <NUM>. This is not necessary. In some embodiments, arm <NUM> may be mounted to a wall, a ceiling or a stationary floor mount. In the illustrated embodiment of <FIG>, 3D optical sensor <NUM> and IDU display device <NUM> are suitably mounted at or near an end <NUM> of arm <NUM>. Robotic arm <NUM> may itself be a component of medical equipment <NUM> which may be controlled using 3D control menus implemented by any of the systems described herein.

Robotic arm <NUM> allows automatic positioning of the components mounted thereon at suitable locations within the OR <NUM>. For example, robotic arm <NUM> may permit controllably positioning 3D optical sensors <NUM> and/or IDU display device <NUM> at a suitable location over surface <NUM> of operating table <NUM>. Robotic arm <NUM> may also be used to retract the components mounted thereon to suitable locations out of the way (e.g. away from operating table <NUM>, away from other medical equipment <NUM> and/or toward the base of arm <NUM>) when the systems described herein, or the components mounted on arm <NUM>, are not needed or desired during a procedure. This ability to retract may free up space for other surgical equipment.

Robotic arm <NUM> shown in the <FIG> embodiment has two degrees of freedom (<NUM> DOF) about pivot joints 56A, 56B. A motor or other suitable actuator may be mounted to each of pivot joints 56A, 56B to permit pivotal motion of corresponding links 58A, 58B relative to one another and/or relative to mount <NUM>. This illustrated embodiment of robotic arm <NUM> represents just one of many possibilities. In other embodiments, additional pivot joints and/or translational actuators (e.g. a linear height adjusting actuator) could be incorporated into robotic arm <NUM>. Robotic arm <NUM> may also comprise one or more additional movable joints <NUM> which may facilitate motion of the enclosure for IDU display device <NUM> and 3D optical sensor <NUM>. For example, such moveable joints <NUM> may permit adjustment of the yaw, pitch and/or roll of IDU display device <NUM> and/or 3D optical sensor <NUM>.

Robotic arm <NUM> may permit optical sensors to map all of, or a suitable portion of, OR <NUM> using 3D optical sensors <NUM> mounted thereon to determine the locations of people, equipment and/or the like in OR <NUM>. Mapping OR <NUM> may be useful for a number of reasons. By way of non-limiting example, C-Arm 12B of the <FIG>system <NUM><NUM> is a controllable arm which permits movement of medical tools (e.g. medical imaging tools) mounted thereon relative to operating table <NUM> and patient <NUM> thereon. However, it can be dangerous or destructive to move C-Arm 12B if the movement path is not clear. Mapping OR <NUM> using optical sensors <NUM> mounted to robotic arm <NUM> (or elsewhere) may be useful to ensure that the movement path of C- Arm 12B is free from obstruction by people or other equipment. In some embodiments, system controller <NUM> could use 3D optical data obtained from optical sensors <NUM> mounted on robotic arm <NUM> or elsewhere to locate and/or identify particular components of medical equipment <NUM> that may be controlled by the 3D control menus <NUM> of any of the systems described herein. Such location of medical equipment <NUM> may be used for head tracking as explained elsewhere herein. Such identification could be made using the shape of the medical equipment <NUM> and by comparing a library of such shapes to 3D optical data obtained by sensors <NUM>. In some embodiments, system controller <NUM> could use 3D optical data (e.g. point clouds) obtained from optical sensors <NUM> mounted on robotic arm <NUM> or elsewhere to map candidate projection surfaces available within OR <NUM> to allow (in combination with moveable robotic arm <NUM>) projection of IDU images <NUM> onto different projection surfaces depending on the location of practitioner <NUM> within OR <NUM>. For example, if practitioner <NUM> is proximate to operating table <NUM>, then controller <NUM> may cause arm <NUM> to move IDU display projector <NUM> over operating table <NUM> and project IDU image <NUM> onto the surface <NUM> of operating table <NUM>. However, if practitioner <NUM> moves closer to a side table (i.e. a different candidate projection surface), then controller <NUM> may cause arm <NUM> to move IDU display projector <NUM> over the side table and to project IDU image <NUM> onto the surface of the side table.

As discussed above, it can be desirable in some embodiments to know the location and orientation of head 25B of practitioner <NUM> (or at least obtain estimates thereof). For example, such practitioner head 25B location and orientation can be used for selection of 3D control menus <NUM> and corresponding IDU images <NUM> to be displayed to practitioner <NUM> (in system <NUM>) and/or for implementing the projection correction methods described herein. In some embodiments, information about the location and/or orientation of the head 25B of practitioner <NUM> may be ascertained by one or more optical sensors <NUM> described herein. In some embodiments, the location of the head 25B of practitioner <NUM> may be ascertained by one or more optical sensors <NUM> described above and the orientation of the practitioner's head 25B may be estimated from the location. In some embodiments, one or more additional 3D cameras or orientation sensors (e.g. accelerometers, gyroscopes and/or the like) may be used to determine the location and/or orientation of the practitioner's head 25B. In some embodiments, gaze tracking techniques such as those described elsewhere herein may be used to determine information similar to the head orientation.

In some embodiments, one or more 3D optical sensor(s), mounted at appropriate location(s) relative to practitioner <NUM> may be used to detect the full practitioner head 25B pose - i.e. position and orientation. Such 3D optical sensor(s) may or may not be the same 3D optical sensors <NUM> used to detect user interactions with 3D control menu <NUM>. In some embodiments, this head pose sensing 3D optical sensor is capable of using modulated infrared light to sense 3D locations for a 2D array of pixels and to form a corresponding point cloud. This point cloud data can then be processed by means of 3D machine vision method(s), whereby the practitioner's head is identified and localized relative to the sensor. A suitable non- limiting example of a machine vision method for identifying head pose is described in N. Ziraknejad, "Driver head pose sensing using a capacitive array and a time-of- flight camera," University of British Columbia, <NUM>, which is hereby incorporated herein by reference. The head pose sensing 3D optical sensor may be tuned for a detection range of up to several meters. In some embodiments, multiple people located in the range of the head pose sensing 3D optical sensor can be detected and delineated from the background. Additionally, a given person can be tracked and distinguished from other nearby persons within view. In some embodiments, suitable indicia may be used to identify one practitioner for whom 3D control menus <NUM> are presented. Such indicia can include indicia that can be detected by the head pose sensing 3D optical sensor (e.g. facial recognition indicia, uniquely shaped headwear and/or the like) or such indicia can include other indicia, such as a wearable RFID tag and/or the like.

System controller <NUM> may be involved in the estimation of head position or the head position may be communicated to system controller <NUM>. System controller <NUM> may then adjust 3D control menu <NUM> and/or IDU image <NUM> for an optimal size and location in front of the desired user (e.g. practitioner <NUM> described elsewhere herein). In addition, the head position estimate enables system controller <NUM> to determine a vector from the practitioner's head 25B to the location of the IDU image <NUM>. For example, the vector may be determined by detecting (e.g. by way of one or more optical sensors) Purkinje reflections (i.e. glints) in one or both of the eyes of practitioner <NUM>. This vector may provide a proxy or estimate for the orientation of the practitioner's head 25B. In some embodiments, where IDU display device <NUM> comprises a projector, the vector from the practitioner's head 25B to the projection surface <NUM> onto which IDU image <NUM> is projected can also be calculated. This vector may be used in the projection correction methods described elsewhere herein. An accurate vector leads to an optimally corrected projection of IDU image <NUM>.

In addition or in the alternative, point cloud data may be analyzed to estimate the orientation of the practitioner's head 25B. A 3D machine vision method can detect characteristics of a user's nose, cheeks and/or other facial structures to estimate the orientation of the practitioner's head 25B. One non-limiting example of such a machine vision method is described in <NPL>. The orientation of the practitioner's head 25B may be used as a proxy or estimate of the gaze direction of practitioner <NUM>. Based on this process, the orientation of the practitioner's head 25B may be used to estimate the line of sight of practitioner <NUM> using one or more 3D optical sensors - no wearables are needed for this technique. However, in some embodiments, the one or more 3D optical sensors may be integrated into AR headset <NUM> of the type described elsewhere herein. In some embodiments, other gaze tracking techniques, such as those described in <CIT> for example, could be used to directly estimate the line of sight of practitioner <NUM>. Such gaze tracking techniques could be implemented in an AR headset <NUM> or using separate components suitably mounted in the OR <NUM>. While this description provides a number of exemplary techniques for estimating the line of sight of practitioner <NUM>, it should be understood that any other suitable line-of-sight estimating technique known now or that becomes known in the future could be used in accordance with various embodiments of the invention. Such line-of-sight estimation techniques could be based on data from any suitable types of sensors, including without limitation 2D and 3D optical sensors, orientation sensors in an AR headset, accelerometers, gyroscopes, and/or the like).

As discussed above, line-of-sight information may be matched with the locations of particular components of medical equipment <NUM> (or locations of their control modules <NUM>) within the OR <NUM>. If the line of sight (i.e. gaze direction) of practitioner <NUM> is directed to within a threshold spatial region corresponding to a piece of controllable medical equipment <NUM> (or to its control module <NUM>, or to any location where a 3D control menu <NUM> has been previously pinned or arbitrarily positioned by practitioner <NUM>), then any of the systems described herein may activate a corresponding 3D control menu <NUM> and display a corresponding IDU image <NUM> suitable for controlling that given piece of medical equipment <NUM>. Based on the estimated line-of-sight information and the corresponding component of controllable medical equipment <NUM> (or the corresponding control module <NUM>), the 3D control menu <NUM> and corresponding IDU image <NUM> presented to practitioner <NUM> may present custom menu items <NUM> corresponding to the functions of the component of controllable medical equipment <NUM> that practitioner <NUM> is looking toward. This may mean a custom 3D control menu <NUM> and corresponding custom set of menu items <NUM> and corresponding custom IDU image <NUM> depending on the component of medical equipment that practitioner <NUM> is looking toward. As discussed above, control modules <NUM> corresponding to components of medical equipment <NUM> and/or the component of medical equipment <NUM> themselves may comprise physical displays which may act as IDU displays <NUM> described herein and upon which custom 3D control menus <NUM> and corresponding custom IDU images <NUM> may be displayed.

For example, if system controller <NUM> determines that the line-of-sight of practitioner <NUM> is oriented toward medical image display 12D (see <FIG>and <FIG>), then controller <NUM> may elect to display a specific 3D control menu <NUM> (which may be specific to controlling medical image display 12D) and a corresponding specific IDU image <NUM>. In some embodiments, a toggle may be provided for each 3D control menu <NUM> and IDU image <NUM>, so that practitioner <NUM> may elect whether or not to have such 3D control menu <NUM> and IDU image <NUM> presented. In some embodiments, system controller <NUM> may elect not to display any 3D control menu <NUM>. For example, system controller <NUM> may elect not to display any 3D control menu <NUM> when the head 25B of practitioner <NUM> is oriented toward a body part being operated on.

To simplify this process, certain user line-of-sight (i.e. gaze direction) regions may be defined and utilized to activate corresponding 3D control menus <NUM>. For example, the yaw of the practitioner's head orientation may be predictably calculated within a range of <NUM>°-<NUM>°, where <NUM>° is the gaze of practitioner <NUM> staring directly forward with a direction substantially perpendicular to a long dimension of the operating table <NUM>. This range may then be split into three sections (<NUM>°-<NUM>°, <NUM>°- <NUM>° <NUM>°-<NUM>°) to represent distinct regions for the activation of different 3D control menus <NUM>, which may in turn correspond to different components of medical equipment <NUM> (or their control modules <NUM>).

A first technique for tracking a practitioner <NUM> (or a portion of the body of a practitioner <NUM>) is described in <NPL>. This method is based on the 3D detection of the human nose - a facial feature with distinct curvature characteristics. By first delineating the head of the user, utilizing standard clustering and filtering techniques, a unique HK curvature analysis is performed to localize surface peaks and troughs - pinpointing the tip of the nose and spot between the eyes (at the base of the nose). Post-processing then structures a facial coordinate frame based on these two positions as well as the centroid of the head point cloud. Then, a standard geometric approach enables a comparison of the facial coordinate frame and camera coordinate frame to determine the user's head orientation - notably the pitch and yaw. This method is capable of robustly determining a practitioner's full head pose without the use of special markers or wearable items on the practitioner <NUM>. The practitioner's head orientation vector may be used as an estimate of the practitioner's line-of-sight, as discussed elsewhere herein.

Another example for identifying and tracking a practitioner's head 25B involves the identification of the practitioner's eyes. The eyes are a common feature that can be detected using standard 2D optical sensors. They can be identified by the reflectivity of the eye (e.g. Purkinje reflections or glints), detection of the iris, or by their shape. One or more aligned 3D optical sensors may be used to determine a 3D position for the eyes. If the field-of-view of both 2D and 3D optical sensors are appropriately mapped, generic 2D detection methods for eye detection and a mapping to 3D data could be utilized for practitioner identification and tracking.

Various additional features of the human head can also be detected in order to identify and track the practitioner's head position. Examples of such features include: eyebrow ridge, chin, cheeks, and lips. Each of these features has unique and generalized curvature properties which can be analyzed. Such methods may involve the detection of a combination of a plurality of the above features to provide a robust method for identification and tracking.

Other methods may be used to analyze optical sensor information to identify and track a practitioner's upper body. Some such methods may involve detecting the shape of the practitioner's head, neck, and/or shoulders. In fact, a 2D optical sensor could be utilized for this purpose and aligned with a 3D optical sensor to determine 3D coordinate data - similar to the eye detection example above. However, a well- filtered 3D point cloud is sufficient for this purpose. The typical shape of a human head is known, and can be deciphered from a plethora of point cloud clusters within the field-of-view of the 3D optical sensor. By removing noise and utilizing a Euclidean (or other suitable) clustering method (grouping points in the point cloud based on proximity and density), larger clusters could be evaluated for the outline shapes of a human head, neck, and/or shoulders.

Another example technique which may be used for practitioner identification and tracking based on a 3D point cloud comprises filtering and clustering the point cloud as described in the previous example. The highest cluster could be identified as a practitioner's head. The centroid of this cluster (average position) of points could be calculated to produce a 3D position of the practitioner's viewpoint. An alternative to the centroid calculation is to decipher the point with the greatest Euclidean distance to the edge of the cluster of 3D points.

In a typical OR <NUM> there will be more than one human. There can be a desire in some embodiments to identify one such human (the control practitioner <NUM>) to be in control of the systems described herein.

In some embodiments, the human closest to 3D optical sensor(s) <NUM> or closest to some other reference point in OR <NUM> may be selected to be the control practitioner <NUM>. The 3D positions of every visible human can be compared to the reference point and the human located at the minimum distance to the reference point can be selected to be the control practitioner <NUM>.

In some embodiments, a plurality of audio microphones can be arranged and utilized to identify the control practitioner <NUM>. By listening for a keyword, information from such microphones can use traditional triangulation techniques for localizing the control practitioner <NUM>. Each microphone will sense an input audio magnitude/amplitude. Based on a known configuration of microphones and the magnitude values, the 3D location of the control practitioner <NUM> can be triangulated. These techniques are referred to as methods for 3D sound localization.

Fiducial markers could be placed on suitable surgical clothing (e.g. on headgear, on the shoulders of surgical scrubs and/or the like) to detect and identify relevant humans within OR <NUM>. Based on differences between such markers, one human in OR <NUM> could be identified as the control practitioner <NUM>. By way of non- limiting example, such markers could comprise optical (e.g. infrared)-reflective markers, electromagnetic (e.g. RFID) tags and/or other indicators detectable by optical sensor or electromagnetic sensors, placed on the outer surface of surgical clothing so that such markers could be detected by 3D or 2D optical sensor(s). By identifying arrays or specific configurations of such markers, unique users could also be identified in real-time.

In some embodiments, a control practitioner <NUM> may identify themselves as the control practitioner by carrying out a specific hand gesture - e.g. a wave or finger configuration within view of 3D optical sensor <NUM>. By identifying various 3D point clusters within the scene (or only delineating the closest clusters), points can be analyzed for specific shapes to change control to a new control practitioner <NUM>.

Many methods exist for face detection and identification. Any of the systems described herein could utilize one or more of these facial recognition techniques for not only detecting human faces, but also identifying unique known users in the workspace, including identifying a control practitioner <NUM>. A human face detected by a 2D optical sensor, could be aligned with the 3D optical sensor's output to find control practitioner <NUM> in 3D. Also, certain users (e.g. surgeons) could be catalogued as authorized control practitioners <NUM> and certain other users could be catalogued as troubleshoot users (e.g. nurses).

Other sensor-based methods could also be utilized for allowing a particular human to become the control practitioner <NUM> and to assume control of the system. For example, a low-proximity sensor like a capacitive sensor could sense a human's specific gesture command to identify the control practitioner <NUM>. In such embodiments, it would be desirable for the 3D position of the capacitive sensor relative to the system or 3D optical sensor(s) <NUM>, in this case, to be known to register the control practitioner's 3D position in the system's world coordinate frame. In other embodiments, control practitioner <NUM> may be identified by way of voice recognition, body scan recognition, retinal scans, and/or the like.

Unless the context clearly requires otherwise, throughout the description and the claims:.

Words that indicate directions such as "vertical", "transverse", "horizontal", "upward", "downward", "forward", "backward", "inward", "outward", "vertical", "transverse", "left", "right", "front", "back", "top", "bottom", "below", "above", "under", and the like, used in this description and any accompanying claims (where present), depend on the specific orientation of the apparatus described and illustrated. The subject matter described herein may assume various alternative orientations. Accordingly, unless the context dictates otherwise, these directional terms are not strictly defined and should not be interpreted narrowly.

Embodiments of the invention may be implemented using specifically designed hardware, configurable hardware, programmable data processors configured by the provision of software (which may optionally comprise "firmware") capable of executing on the data processors, special purpose computers or data processors that are specifically programmed, configured, or constructed to perform one or more steps in a method as explained in detail herein and/or combinations of two or more of these. Examples of specifically designed hardware are: logic circuits, application-specific integrated circuits ("ASICs"), large scale integrated circuits ("LSIs"), very large scale integrated circuits ("VLSIs"), and the like. Examples of configurable hardware are: one or more programmable logic devices such as programmable array logic ("PALs"), programmable logic arrays ("PLAs"), and field programmable gate arrays ("FPGAs")). Examples of programmable data processors are: microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors, math coprocessors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors in a control circuit for a device may implement methods as described herein by executing software instructions in a program memory accessible to the processors.

Processing may be centralized or distributed. Where processing is distributed, information including software and/or data may be kept centrally or distributed. Such information may be exchanged between different functional units by way of a communications network, such as a Local Area Network (LAN), Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channel.

For example, while processes or blocks are presented in a given order, alternative examples may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times.

In addition, while elements are at times shown as being performed sequentially, they may instead be performed simultaneously or in different sequences. It is therefore intended that the following claims are interpreted to include all such variations as are within their intended scope.

Software and other modules may reside on servers, workstations, personal computers, tablet computers, image data encoders, image data decoders, PDAs, color-grading tools, video projectors, audio-visual receivers, displays (such as televisions), digital cinema projectors, media players, and other devices suitable for the purposes described herein. Those skilled in the relevant art will appreciate that aspects of the system can be practiced with other communications, data processing, or computer system configurations, including: Internet appliances, hand-held devices (including personal digital assistants (PDAs)), wearable computers, all manner of cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics (e.g., video projectors, audio-visual receivers, displays, such as televisions, and the like), set-top boxes, color-grading tools, network PCs, mini-computers, mainframe computers, and the like.

The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, non-transitory media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

In some embodiments, the invention may be implemented in whole or in part in software. For greater clarity, "software" includes any instructions executed on a processor, and may include (but is not limited to) firmware, resident software, microcode, and the like. Both processing hardware and software may be centralized or distributed (or a combination thereof), in whole or in part, as known to those skilled in the art. For example, software and other modules may be accessible via local memory, via a network, via a browser or other application in a distributed computing context, or via other means suitable for the purposes described above.

The scope of the invention is set out in the appended claims.

Claim 1:
A system (<NUM>) for touchless control of one or more medical equipment devices (<NUM>) in an operating room, the system (<NUM>) comprising:
a 3D optical sensor (<NUM>) connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor (<NUM>);
a controller (<NUM>) connected to receive 3D optical data from the 3D optical sensor (<NUM>) and configured to provide a three-dimensional control menu (<NUM>), the three-dimensional control menu (<NUM>) comprising a plurality of menu items (<NUM>), each menu item (<NUM>) selectable by the practitioner by one or more gestures made by the practitioner in a volumetric spatial region corresponding to the menu item (<NUM>) and detected by the controller based on the 3D optical data; and
an interactive display unit, IDU, display (<NUM>) for displaying an IDU image (<NUM>) corresponding to the three-dimensional control menu (<NUM>), the IDU image (<NUM>) providing indicia of any one or more selected menu items (<NUM>);
characterized in that:
the controller (<NUM>) is further configured, based on input from the 3D optical sensor (<NUM>), to estimate a line of sight of a practitioner;
wherein, when the estimated line of sight is directed within a first spatial range around a first medical equipment device (<NUM>), the controller is configured to determine that the practitioner is looking at the first medical equipment device (<NUM>) and wherein, after determining that the practitioner is looking at the first medical equipment device (<NUM>), the controller (<NUM>) is configured to:
provide a first device-specific three-dimensional control menu (<NUM>) comprising first device-specific menu items which, when selected, result in delivering corresponding operational commands to the first medical equipment device (<NUM>) to control operation of the first medical equipment device (<NUM>); and
cause the IDU display (<NUM>) to display a first device-specific IDU image (<NUM>) comprising graphics or text corresponding to the first device-specific menu items.