Patent Description:
<CIT> describes various systems and methods for surgical and interventional planning, support, post-operative follow-up, and functional recovery tracking. In general, a patient can be tracked throughout medical treatment including through initial onset of symptoms, diagnosis, non-surgical treatment, surgical treatment, and recovery from the surgical treatment. In one embodiment, a patient and one or more medical professionals involved with treating the patient can electronically access a comprehensive treatment planning, support, and review system. The system can provide recommendations regarding diagnosis, non-surgical treatment, surgical treatment, and recovery from the surgical treatment based on data gathered from the patient and the medical professional(s). The system can manage the tracking of multiple patients, thereby allowing for data comparison between similar aspects of medical treatments and for learning over time through continual data gathering, analysis, and assimilation to decisionmaking algorithms.

<CIT> describes an electromagnetic tracking method that includes generating an electromagnetic field in a region of interest. The electromagnetic field is subject to distortion in response to a presence of metal artifacts proximate the electromagnetic field. An array of reference sensors having a predefined known configuration are disposed proximate the region of interest. A first set of locations of the array of reference sensors is determined with respect to the electromagnetic field generator in response to an excitation of one or more of the reference sensors via the electromagnetic field. A second mechanism, other than the electromagnetic field, determines a first portion of a second set of locations of at least one or more sensors of the array of reference sensors with respect to the second mechanism, the second mechanism being in a known spatial relationship with the electromagnetic field generator. A remainder portion of the second set of locations of the reference sensors of the array of reference sensors is determined in response to (i) the first portion of the second set of locations determined using the second mechanism and (ii) the predefined known configuration of the array of reference sensors. The method further includes compensating for metal distortion of the electromagnetic field in the region of interest as a function of the first and second sets of reference sensor locations of the array of reference sensors.

<NPL>, describes combining real-time intra-operative echocardiography with a virtual reality environment providing the surgeon with a broad range of valuable information. Pre-operative images, electrophysiological data, positions of magnetically tracked surgical instruments, and dynamic surgical target representations are among the data that can be presented to the surgeon to augment intra-operative ultrasound images. The described augmented reality system is applicable to procedures such as mitral valve replacement and atrial septal defect repair, as well as ablation therapies for treatment of atrial fibrillation.

<CIT> describes a method for assisting cartilage diagnostic and therapeutic procedures and includes the steps of acquiring 3D osteocartilaginous parameters by using multimodal 3D tracked devices; incorporating these parameters into a volumic anatomic osteocartilaginous model from which a bone tracking virtual real-time environment is built; three-dimensionally computing an osteocartilaginous quality score from this multiparametric 3D osteocartilaginous model; providing real-time navigation in this 3D virtual environment in order to make ongoing therapeutic assessments and adjustments; and updating steps <NUM> to <NUM> according to the performed therapy.

The present invention provides a medical device-placing system according to claim <NUM>.

The present invention further provides a method of a medical device-placing system according to claim <NUM>.

Disclosed herein is a medical device-placing system including an ultrasound probe, an array of magnetic sensors, a console, and an alternative-reality headset. The ultrasound probe is configured to emit ultrasound signals into a limb of a patient and receive echoed ultrasound signals from the patient's limb by way of a piezoelectric sensor array. The array of magnetic sensors are embedded within a housing configured for placement about the patient's limb. The console has electronic circuitry including memory and a processor configured to transform the echoed ultrasound signals to produce ultrasound-image segments corresponding to anatomical structures of the patient's limb. The console is also configured to transform magnetic-sensor signals from the array of magnetic sensors into location information for a magnetized medical device within the patient's limb when the housing is placed about the patient's limb. The alternative-reality headset includes a display screen coupled to a frame having electronic circuitry including memory and a processor. The display screen is configured such that a wearer of the alternative-reality headset can see the patient's limb through the display screen. The display screen is configured to display over the patient's limb a virtual medical device in accordance with the location information for the medical device within objects of virtual anatomy corresponding to the ultrasound-image segments.

In some embodiments, the ultrasound probe is configured with a pulsed-wave Doppler imaging mode for emitting and receiving the ultrasound signals. The console is configured to capture ultrasound-imaging frames in accordance with the pulsed-wave Doppler imaging mode, stitch the ultrasound-imaging frames together with a stitching algorithm, and segment the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with an image segmentation algorithm.

The console is configured to transform the ultrasound-image segments into the objects of virtual anatomy with a virtualization algorithm. The console is configured to send both the virtual medical device and the objects of virtual anatomy to the alternative-reality headset for display over the patient's limb.

In some embodiments, the alternative-reality headset is configured to anchor the virtual medical device and the objects of virtual anatomy to the patient's limb over which the virtual medical device and the objects of virtual anatomy are displayed.

In some embodiments, the alternative-reality headset further includes one or more eye-tracking cameras coupled to the frame configured to capture eye movements of the wearer. The processor of the alternative-reality headset is configured to process the eye movements with an eye-movement algorithm to identify a focus of the wearer for selecting or enhancing the objects of virtual anatomy, the virtual medical device, or both corresponding to the focus of the wearer.

In some embodiments, the alternative-reality headset further includes one or more patient-facing cameras coupled to the frame configured to capture gestures of the wearer. The processor of the alternative-reality headset is configured to process the gestures with a gesture-command algorithm to identify gesture-based commands issued by the wearer for execution thereof by the alternative-reality headset.

In some embodiments, the alternative-reality headset further includes one or more microphones coupled to the frame configured to capture audio of the wearer. The processor of the alternative-reality headset is configured to process the audio with an audio-command algorithm to identify audio-based commands issued by the wearer for execution thereof by the alternative-reality headset.

In some embodiments, the housing is a rigid frame. Each magnetic sensor of the array of magnetic sensors is embedded within the frame and has a fixed spatial relationship to another magnetic sensor. The fixed spatial relationship is available to the console for transforming the magnetic-sensor signals from the array of magnetic sensors into the location information for the medical device.

In some embodiments, the medical device-placing system further includes a magnetic-field generator configured to generate a magnetic field. The console is configured to determine the spatial relationship of each magnetic sensor of the array of magnetic sensors to another magnetic sensor from the magnetic-sensor signals produced by the array of magnetic sensors while in the presence of the generated magnetic field. The determined spatial relationship is available to the console for subsequently transforming the magnetic-sensor signals from the array of magnetic sensors into the location information for the medical device.

Also disclosed herein is an anatomy-visualizing system including, in some example implementations, an ultrasound-imaging system and an alternative-reality headset. The ultrasound-imaging system includes an ultrasound probe and a console. The ultrasound probe is configured to emit ultrasound signals into a patient and receive echoed ultrasound signals from the patient by way of a piezoelectric sensor array. The console has electronic circuitry including memory and a processor configured to transform the echoed ultrasound signals to produce ultrasound-image segments corresponding to anatomical structures of the patient. The alternative-reality headset includes a display screen coupled to a frame having electronic circuitry including memory and a processor. The display screen is configured such that a wearer of the alternative-reality headset can see the patient through the display screen. The display screen is configured to display objects of virtual anatomy over the patient corresponding to the ultrasound-image segments.

In some example implementations, the ultrasound probe is configured with a pulsed-wave Doppler imaging mode for emitting and receiving the ultrasound signals. The console is configured to capture ultrasound-imaging frames in accordance with the pulsed-wave Doppler imaging mode, stitch the ultrasound-imaging frames together with a stitching algorithm, and segment the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with an image segmentation algorithm.

In some example implementations, the console is configured to transform the ultrasound-image segments into the objects of virtual anatomy with a virtualization algorithm. The console is configured to send the objects of virtual anatomy to the alternative-reality headset for display over the patient.

In some example implementations, the alternative-reality headset is configured to anchor the objects of virtual anatomy to the patient over which the objects of virtual anatomy are displayed.

In some example implementations, the alternative-reality headset further includes one or more eye-tracking cameras coupled to the frame eye movements of the wearer. The processor of the alternative-reality headset is configured to process the eye movements with an eye-movement algorithm to identify a focus of the wearer for selecting or enhancing the objects of virtual anatomy corresponding to the focus of the wearer.

In some example implementations, the alternative-reality headset further includes one or more patient-facing cameras coupled to the frame configured to capture gestures of the wearer. The processor of the alternative-reality headset is configured to process the gestures with a gesture-command algorithm to identify gesture-based commands issued by the wearer for execution thereof by the alternative-reality headset.

In some example implementations, the alternative-reality headset further includes one or more microphones coupled to the frame configured to capture audio of the wearer. The processor of the alternative-reality headset is configured to process the audio with an audio-command algorithm to identify audio-based commands issued by the wearer for execution thereof by the alternative-reality headset.

Also disclosed herein is a medical device-locating system including, in some example implementations, an array of magnetic sensors embedded within a housing and a console having electronic circuitry including memory and a processor. The housing is configured for placement about a limb of a patient. The console is configured to transform magnetic-sensor signals from the array of magnetic sensors into location information for a magnetized medical device within the limb of the patient when the housing is placed about the limb of the patient.

In some example implementations, the housing is a rigid frame. Each magnetic sensor of the array of magnetic sensors is embedded within the frame and has a fixed spatial relationship to another magnetic sensor. The fixed spatial relationship is available to the console for transforming the magnetic-sensor signals from the array of magnetic sensors into the location information for the medical device.

In some example implementations, the housing is a drape. Each magnetic sensor of the array of magnetic sensors embedded is embedded within the drape and has a variable spatial relationship to another magnetic sensor depending upon how the drape is placed about the limb of the patient.

In some example implementations, the medical device-locating system further includes a magnetic-field generator configured to generate a magnetic field. The console is configured to determine the spatial relationship of each magnetic sensor of the array of magnetic sensors to another magnetic sensor from the magnetic-sensor signals produced by the array of magnetic sensors while in the presence of the generated magnetic field. The determined spatial relationship is available to the console for subsequently transforming the magnetic-sensor signals from the array of magnetic sensors into the location information for the medical device.

In some example implementations, the medical device-locating system further includes a display screen configured to depict the medical device within the limb of the patient in accordance with the location information for the medical device.

In some example implementations, the display screen is a see-through display screen coupled to a frame of an alternative-reality headset. The display screen is configured to receive from the console a virtual object corresponding to the medical device for depicting the medical device within the limb of the patient in accordance with the location information for the medical device.

Also disclosed herein is a method of a medical device-placing system including emitting ultrasound signals into a limb of a patient and receiving echoed ultrasound signals from the patient's limb by way of a piezoelectric sensor array of an ultrasound probe; transforming the echoed ultrasound signals with a console having electronic circuitry including memory and a processor to produce ultrasound-image segments corresponding to anatomical structures of the patient's limb; transforming magnetic-sensor signals from an array of magnetic sensors embedded within a housing placed about the patient's limb with the console into location information for a magnetized medical device within the patient's limb; displaying over the patient's limb on a see-through display screen of an alternative-reality headset having electronic circuitry including memory and a processor in a frame coupled to the display screen a virtual medical device in accordance with the location information for the medical device within objects of virtual anatomy corresponding to the ultrasound-image segments.

In some embodiments, the method further includes capturing in the memory of the console ultrasound-imaging frames in accordance with a pulsed-wave Doppler imaging mode of the ultrasound probe while emitting and receiving the ultrasound signals; stitching the ultrasound-imaging frames together with a stitching algorithm; and segmenting the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with an image segmentation algorithm.

The method further includes transforming the ultrasound-image segments into the objects of virtual anatomy with a virtualization algorithm; and sending both the virtual medical device and the objects of virtual anatomy to the alternative-reality headset for display over the patient's limb.

In some embodiments, the method further includes anchoring the virtual medical device and the objects of virtual anatomy to the patient's limb over which the virtual medical device and the objects of virtual anatomy are displayed.

In some embodiments, the method further includes capturing in the memory of the console eye movements of the wearer using one or more eye-tracking cameras coupled to the frame of the alternative-reality headset; and processing the eye movements with an eye-movement algorithm to identify a focus of the wearer for selecting or enhancing the objects of virtual anatomy corresponding to the focus of the wearer.

In some embodiments, the method further includes capturing in the memory of the console gestures of the wearer using one or more patient-facing cameras coupled to the frame of the alternative-reality headset; and processing the gestures with a gesture-command algorithm to identify gesture-based commands issued by the wearer for execution thereof by the alternative-reality headset.

In some embodiments, the method further includes capturing in the memory of the console audio of the wearer using one or more microphones coupled to the frame of the alternative-reality headset; and processing the audio with an audio-command algorithm to identify audio-based commands issued by the wearer for execution thereof by the alternative-reality headset.

In some embodiments, each magnetic sensor of the array of magnetic sensors is embedded within a rigid frame for the housing, the magnetic sensors having a fixed spatial relationship to each other.

In some embodiments, each magnetic sensor of the array of magnetic sensors is embedded within a drape for the housing, the magnetic sensors having a variable spatial relationship to each other depending upon how the drape is placed about the limb of the patient.

In some embodiments, the method further includes generating a magnetic field with a magnetic-field generator; and determining the spatial relationship of each magnetic sensor of the array of magnetic sensors to another magnetic sensor from the magnetic-sensor signals produced by the array of magnetic sensors while in the presence of the generated magnetic field.

These and other features of the concepts provided herein will become more apparent to those of skill in the art in view of the accompanying drawings and following description, which disclose particular embodiments and example implementations of such concepts in greater detail.

Labels such as "left," "right," "front," "back," "top," "bottom," and the like are used for convenience and are not intended to imply, for example, any particular fixed location, orientation, or direction.

With respect to "proximal," a "proximal portion" or a "proximal end portion" of, for example, a medical device such as a catheter includes a portion of the catheter intended to be near a clinician when the catheter is used on a patient. Likewise, a "proximal length" of, for example, the catheter includes a length of the catheter intended to be near the clinician when the catheter is used on the patient. A "proximal end" of, for example, the catheter includes an end of the catheter intended to be near the clinician when the catheter is used on the patient. The proximal portion, the proximal end portion, or the proximal length of the catheter can include the proximal end of the catheter; however, the proximal portion, the proximal end portion, or the proximal length of the catheter need not include the proximal end of the catheter. That is, unless context suggests otherwise, the proximal portion, the proximal end portion, or the proximal length of the catheter is not a terminal portion or terminal length of the catheter.

With respect to "distal," a "distal portion" or a "distal end portion" of, for example, a medical device such as a catheter disclosed herein includes a portion of the catheter intended to be near or in a patient when the catheter is used on the patient. Likewise, a "distal length" of, for example, the catheter includes a length of the catheter intended to be near or in the patient when the catheter is used on the patient. A "distal end" of, for example, the catheter includes an end of the catheter intended to be near or in the patient when the catheter is used on the patient. The distal portion, the distal end portion, or the distal length of the catheter can include the distal end of the catheter; however, the distal portion, the distal end portion, or the distal length of the catheter need not include the distal end of the catheter. That is, unless context suggests otherwise, the distal portion, the distal end portion, or the distal length of the catheter is not a terminal portion or terminal length of the catheter.

With respect to "alternative reality," alternative reality includes virtual reality, augmented reality, and mixed reality unless context suggests otherwise. "Virtual reality" includes virtual content in a virtual setting, which setting can be a fantasy or a real-world simulation. "Augmented reality" and "mixed reality" include virtual content in a real-world setting. Augmented reality includes the virtual content in the real-world setting, but the virtual content is not necessarily anchored in the real-world setting. For example, the virtual content can be information overlying the real-world setting. The information can change as the real-world setting changes due to time or environmental conditions in the real-world setting, or the information can change as a result of an experiencer of the augmented reality moving through the real-world setting - but the information remains overlying the real-world setting. Mixed reality includes the virtual content anchored in every dimension of the real-world setting. For example, the virtual content can be a virtual object anchored in the real-world setting. The virtual object can change as the real-world setting changes due to time or environmental conditions in the real-world setting, or the virtual object can change to accommodate the perspective of an experiencer of the mixed reality as the experiencer moves through the real-world setting. The virtual object can also change in accordance with any interactions with the experiencer or another real-world or virtual agent. Unless the virtual object is moved to another location in the real-world setting by the experiencer of the mixed reality, or some other real-world or virtual agent, the virtual object remains anchored in the real-world setting. Mixed reality does not exclude the foregoing information overlying the real-world setting described in reference to augmented reality.

As set forth above, an ability to visualize anatomy such as the peripheral vasculature is needed. In addition, an ability to visualize such anatomy in conjunction with medical devices such as guidewires and catheters is needed to finally make it possible to determine exactly where such medical devices are during placement thereof. Lastly, such abilities should not adversely affect patients or clinicians. Disclosed herein are systems and methods for visualizing anatomy, locating medical devices, or placing medical devices that address one or more needs such as the foregoing.

<FIG> provides a block diagram for an anatomy-visualizing system <NUM> in accordance with some embodiments. <FIG> provides a block diagram for a medical device-locating system <NUM> in accordance with some embodiments. <FIG> provides a block diagram for a medical device-placing system <NUM> in accordance with some embodiments.

As shown, the anatomy-visualizing system <NUM> includes an ultrasound-imaging system <NUM> and an alternative-reality headset <NUM>, wherein the ultrasound-imaging system <NUM> includes a console <NUM> and an ultrasound probe <NUM>; the medical device-locating system <NUM> includes a console <NUM>, a medical-device detector <NUM>, and, optionally, the alternative-reality headset <NUM>; and the medical device-placing system <NUM> includes a console <NUM>, the ultrasound probe <NUM>, the alternative-reality headset <NUM>, and the medical-device detector <NUM>. Thus, the medical device-placing system <NUM> is a combination of at least some elements of the anatomy-visualizing system <NUM> and the medical device-locating system <NUM>.

While each console of the consoles <NUM>, <NUM>, and <NUM> is indicated herein by a different reference numeral, the consoles <NUM>, <NUM>, and <NUM> need not be different consoles. That is, the consoles <NUM>, <NUM>, and <NUM> can be the same console. For example, that same console can be the console <NUM> of the medical device-placing system <NUM>, wherein the console <NUM> is a combination of the console <NUM> of the anatomy-visualizing system <NUM> and the console <NUM> of the medical device-locating system <NUM>. In view of the foregoing, components and functions of the console <NUM> described in reference to the anatomy-visualizing system <NUM> should be understood to apply to the anatomy-visualizing system <NUM> or the medical device-placing system <NUM>. Likewise, components and functions of the console <NUM> described in reference to the medical device-locating system <NUM> should be understood to apply to the medical device-locating system <NUM> or the medical device-placing system <NUM>.

Notwithstanding the foregoing, in some embodiments of the anatomy-visualizing system <NUM>, the medical device-locating system <NUM>, and the medical device-placing system <NUM> the respective consoles <NUM>, <NUM>, and <NUM> are absent. In such embodiments, the alternative reality headset <NUM> or another system component serves as the console or performs the functions (e.g., processing) thereof.

Again, <FIG> provides the block diagram for the anatomy-visualizing system <NUM> in accordance with some embodiments.

As shown, the anatomy-visualizing system <NUM> includes the ultrasound-imaging system <NUM> and the alternative-reality headset <NUM>, wherein the ultrasound-imaging system <NUM> includes the console <NUM> and the ultrasound probe <NUM>.

<FIG> provides a block diagram for the ultrasound probe <NUM> connected to the console of the anatomy-visualizing system <NUM> in accordance with some embodiments.

As shown, the console <NUM> has electronic circuitry including memory <NUM> and one or more processors <NUM> configured to transform echoed ultrasound signals from a patient with one or more algorithms <NUM> to produce ultrasound images and ultrasound-image segments therefrom corresponding to anatomical structures of the patient. The console <NUM> is configured to capture in the memory <NUM> ultrasound-imaging frames (i.e., frame-by-frame ultrasound images) in accordance with a pulsed-wave Doppler imaging mode of the ultrasound probe <NUM>, stitch the ultrasound-imaging frames together with a stitching algorithm of the one or more algorithms <NUM>, and segment the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with an image segmentation algorithm of the one or more algorithms <NUM>. The console <NUM> is configured to transform the ultrasound-image segments into objects of virtual anatomy with a virtualization algorithm of the one or more algorithms <NUM>. The console <NUM> is configured to send the objects of virtual anatomy to the alternative-reality headset <NUM> for display over the patient by way of a wireless communications interface <NUM>.

The console <NUM> includes a number of components of the anatomy-visualizing system <NUM>, and the console <NUM> can take any form of a variety of forms to house the number of components. The one or more processors <NUM> and the memory <NUM> (e.g., non-volatile memory such as electrically erasable, programmable, read-only memory ["EEPROM"]) of the console <NUM> are configured for controlling various functions of the anatomy-visualizing system <NUM> such as executing the one or more algorithms <NUM> during operation of the anatomy-visualizing system <NUM>. A digital controller or analog interface <NUM> is also included with the console <NUM>, and the digital controller or analog interface <NUM> is in communication with the one or more processors <NUM> and other system components to govern interfacing between the probe <NUM>, the alternative-reality headset <NUM>, as well as other system components.

The console <NUM> further includes ports <NUM> for connection with additional components such as optional components <NUM> including a printer, storage media, keyboard, etc. The ports <NUM> can be universal serial bus ("USB") ports, though other ports or a combination of ports can be used, as well as other interfaces or connections described herein. A power connection <NUM> is included with the console <NUM> to enable operable connection to an external power supply <NUM>. An internal power supply <NUM> (e.g., disposable or rechargeable battery) can also be employed, either with the external power supply <NUM> or exclusive of the external power supply <NUM>. Power management circuitry <NUM> is included with the digital controller or analog interface <NUM> of the console <NUM> to regulate power use and distribution.

A display <NUM> can be, for example, a liquid crystal display ("LCD") integrated into the console <NUM> and used to display information to the clinician during a procedure. For example, the display <NUM> can be used to display an ultrasound image of a targeted internal body portion of the patient attained by the probe <NUM>. Alternatively, the display <NUM> can be separate from the console <NUM> instead of integrated into the console <NUM>; however, such a display is different than that of the alternative-reality headset <NUM>. The console <NUM> can further include a console button interface <NUM>. In combination with control buttons on the probe <NUM>, the console button interface <NUM> can be used by a clinician to immediately call up a desired mode on the display <NUM> for use by the clinician in the procedure.

The ultrasound probe <NUM> is configured to emit ultrasound signals into the patient and receive the echoed ultrasound signals from the patient by way of a piezoelectric sensor array <NUM>. The ultrasound probe <NUM> can be configured with a continuous wave or a pulsed-wave imaging mode. For example, the ultrasound probe <NUM> can configured with the foregoing pulsed-wave Doppler imaging mode for emitting and receiving the ultrasound signals.

The probe <NUM> further includes a button-and-memory controller <NUM> for governing operation of the probe <NUM> and buttons thereof. The button-and-memory controller <NUM> can include non-volatile memory such as EEPROM. The button-and-memory controller <NUM> is in operable communication with a probe interface <NUM> of the console <NUM>, which probe interface includes a piezoelectric input-output component <NUM> for interfacing with the piezoelectric sensor array <NUM> of the probe <NUM> and a button-and-memory input-output component <NUM> for interfacing with the button-and-memory controller <NUM> of the probe <NUM>.

<FIG> provides a block diagram for the alternative-reality headset <NUM> of the anatomy-visualizing system <NUM> in accordance with some embodiments.

As shown, the alternative-reality headset <NUM>, which can have a goggle-type or face shield-type form factor, includes a suitably configured display screen <NUM> and a window <NUM> thereover coupled to a frame <NUM> having electronic circuitry including memory <NUM> and one or more processors <NUM>. The display screen <NUM> is configured such that a wearer of the alternative-reality headset <NUM> can see the patient through the display screen <NUM> in accordance with an opacity of the window <NUM>, which opacity is adjustable is adjustable with an opacity control <NUM>. The display screen <NUM> is configured to display objects of virtual anatomy over the patient corresponding to the ultrasound-image segments produced by the console <NUM> with the image segmentation algorithm. (See, for example, <FIG>, wherein the objects of virtual anatomy correspond to vasculature in a limb of the patient. ) In displaying the objects of virtual anatomy over the patient, the alternative-reality headset <NUM> can be configured to three-dimensionally anchor the objects of virtual anatomy to the patient over which the objects of virtual anatomy are displayed, which allows the wearer of the alternative-reality headset <NUM> to see a true representation of the patient's anatomy for one or more subsequent medical procedures (e.g., accessing a vessel and placing a medical device such as a guidewire of catheter in the vessel). Anchoring the objects of virtual anatomy to the patient over which the objects of virtual anatomy are displayed is characteristic of mixed reality.

The alternative-reality headset <NUM> can further include a perceptual user interface ("PUI") configured to enable the wearer of the alternative-reality headset <NUM> to interact with the alternative-reality headset <NUM> without a physical input device such as keyboard or mouse. Instead of a physical input device, the PUI can have input devices including, but not limited to, one or more wearer-facing eye-tracking cameras <NUM>, one or more patient-facing cameras <NUM>, one or more microphones <NUM>, or a combination thereof. At least one advantage of the PUI the input devices thereof is the clinician does not have to reach outside a sterile field to execute a command of the alternative-reality headset <NUM>.

With respect to the one or more eye-tracking cameras <NUM>, the one or more eye-tracking cameras <NUM> can be coupled to the frame <NUM> and configured to capture eye movements of the wearer in a camera buffer <NUM> or the memory <NUM>. The processor <NUM> of the alternative-reality headset <NUM> can be configured to process the eye movements with an eye-movement algorithm of one or more algorithms <NUM> to identify a focus of the wearer for selecting the objects of virtual anatomy or other virtual objects (e.g., a virtual medical device) corresponding to the focus of the wearer. For example, the focus of the wearer can be used by the PUI to select an object of virtual anatomy for enhancing the object of virtual anatomy by way of highlighting the object of virtual anatomy or increasing the contrast between the object of virtual anatomy and its environment. In another example, the focus of the wearer can be used by the PUI to select an object of virtual anatomy for performing one or more other operations of the PUI such as zooming in on the object of virtual anatomy, providing a crosssection of the one or more objects of virtual anatomy, or the like. (See, for example, <FIG>, wherein the objects of virtual anatomy correspond to vasculature in a limb of the patient, and wherein the objects of virtual anatomy are in cross section.

With respect to the one or more patient-facing cameras <NUM>, the one or more patient-facing cameras <NUM> can be coupled to the frame <NUM> and configured to capture gestures of the wearer in a camera buffer <NUM> or the memory <NUM>. The processor <NUM> of the alternative-reality headset <NUM> can be configured to process the gestures with a gesture-command algorithm of the one or more algorithms <NUM> to identify gesture-based commands issued by the wearer for execution thereof by the alternative-reality headset <NUM>.

With respect to the one or more microphones <NUM>, the one or more microphones <NUM> can be coupled to the frame <NUM> configured to capture audio of the wearer in the memory <NUM>. The processor <NUM> of the alternative-reality headset <NUM> can be configured to process the audio with an audio-command algorithm of the one or more algorithms <NUM> to identify audio-based commands issued by the wearer for execution thereof by the alternative-reality headset <NUM>.

The electronic circuitry includes the processor <NUM>, a memory controller <NUM> in communication with the memory <NUM> (e.g., dynamic random-access memory ["DRAM"]), a camera interface <NUM>, the camera buffer <NUM>, a display driver <NUM>, a display formatter <NUM>, a timing generator <NUM>, a display-out interface <NUM>, and a display-in interface <NUM>. Such components can be in communication with each other through the processor <NUM>, dedicated lines of one or more buses, or a combination thereof.

The camera interface <NUM> is configured to provide an interface to the one or more eye-tracking cameras <NUM> and the one or more patient-facing cameras <NUM>, as well as store respective images received from the cameras <NUM>, <NUM> in the camera buffer <NUM> or the memory <NUM>. Each camera of the one or more eye-tracking cameras <NUM> can be an infrared ("IR") camera or a position-sensitive detector ("PSD") configured to track eye-glint positions by way of IR reflections or eye glint-position data, respectively.

The display driver <NUM> is configured to drive the display <NUM>. The display formatter <NUM> is configured to provide display-formatting information for the objects of virtual anatomy to the one or more processors <NUM> of the console <NUM> for formatting the objects of virtual anatomy for display on display <NUM> the over the patient. The timing generator <NUM> is configured to provide timing data for the alternative-reality headset <NUM>. The display-out interface <NUM> includes a buffer for providing images from the one or more eye-tracking cameras <NUM> or the one or more patient-facing cameras <NUM> to the one or more processors <NUM> of the console <NUM>. The display-in interface <NUM> includes a buffer for receiving images such as the objects of virtual anatomy to be displayed on the display <NUM>. The display-out and display-in interfaces <NUM>,<NUM> are configured to communicate with the console <NUM> by way of wireless communications interface <NUM>. The opacity control <NUM> is configured to change a degree of opacity of the window <NUM>.

Additional electronic circuitry includes a voltage regulator <NUM>, an eye-tracking illumination driver <NUM>, an audio digital-to-analog converter ("DAC") and amplifier <NUM>, a microphone preamplifier and audio analog-to-digital converter ("ADC") <NUM>, a temperaturesensor interface <NUM>, and a clock generator <NUM>. The voltage regulator <NUM> is configured to receive power from an internal power supply <NUM> (e.g., a battery) or an external power supply <NUM> through power connection <NUM>. The voltage regulator <NUM> is configured to provide the received power to the electronic circuitry of the alternative-reality headset <NUM>. The eye-tracking illumination driver <NUM> is configured to control an eye-tracking illumination unit <NUM> by way of a drive current or voltage to operate about a predetermined wavelength or within a predetermined wavelength range. The audio DAC and amplifier <NUM> is configured to provide audio data to earphones or speakers <NUM>. The microphone preamplifier and audio ADC <NUM> is configured to provide an interface for the one or more microphones <NUM>. The temperature sensor interface <NUM> is configured as an interface for a temperature sensor <NUM>. In addition, the alternative-reality headset <NUM> can include orientation sensors including a three-axis magnetometer <NUM>, a three-axis gyroscope <NUM>, and a three-axis accelerometer <NUM> configured to provide orientation-sensor data for determining an orientation of the alternative-reality headset <NUM> at any given time. Furthermore, the alternative-reality headset <NUM> can include a global-positioning system ("GPS") receiver <NUM> configured to receive GPS data (e.g., time and position information for one or more GPS satellites) for determining a location of the alternative-reality headset <NUM> at any given time.

Again, <FIG> provides the block diagram for the medical device-locating system <NUM> in accordance with some embodiments.

As shown, the medical device-locating system <NUM> includes the console <NUM>, the medical-device detector <NUM> including an array of magnetic sensors <NUM>, and, optionally, the alternative-reality headset <NUM>.

<FIG> provides a block diagram for the medical-device detector <NUM> connected to the console <NUM> of the medical device-locating system <NUM> in accordance with some embodiments.

As shown, the console <NUM> has electronic circuitry including memory <NUM> and one or more processors <NUM> configured to transform magnetic-sensor signals from the array of magnetic sensors <NUM> with one or more algorithms <NUM> (e.g., a location-finding algorithm including, for example, triangulation) into location information for a magnetized medical device (e.g., a catheter including a magnetic element) within a limb of a patient when the medical-device detector <NUM> is placed about the limb of the patient.

The console <NUM> includes a number of components of the medical device-locating system <NUM>, and the console <NUM> can take any form of a variety of forms to house the number of components. The one or more processors <NUM> and the memory <NUM> (e.g., non-volatile memory such as EEPROM) of the console <NUM> are configured for controlling various functions of the medical device-locating system <NUM> such as executing the one or more algorithms <NUM> during operation of the medical device-locating system <NUM>. A digital controller or analog interface <NUM> is also included with the console <NUM>, and the digital controller or analog interface <NUM> is in communication with the one or more processors <NUM> and other system components to govern interfacing between the medical-device detector <NUM>, the alternative-reality headset <NUM>, as well as other system components. The console <NUM> can also be configured with a wireless communications interface <NUM> to send to the alternative-reality headset <NUM> location information, or a representation of the medical device (e.g., a virtual medical device) in accordance with the location information, for a magnetized medical device within a limb of a patient for display on the display <NUM> of the alternative-reality headset <NUM>. (See, for example, <FIG>, wherein the objects of virtual anatomy correspond to vasculature in the limb of the patient, and wherein a virtual medical device such as a guidewire or catheter is being advanced therethrough.

The console <NUM> further includes ports <NUM> for connection with the medical-device detector <NUM> as well as additional, optional components such as a magnetic-field generator <NUM>, a printer, storage media, keyboard, etc. The ports <NUM> can be USB ports, though other ports or a combination of ports can be used, as well as other interfaces or connections described herein. A power connection <NUM> is included with the console <NUM> to enable operable connection to an external power supply <NUM>. An internal power supply <NUM> (e.g., disposable or rechargeable battery) can also be employed, either with the external power supply <NUM> or exclusive of the external power supply <NUM>. Power management circuitry <NUM> is included with the digital controller or analog interface <NUM> of the console <NUM> to regulate power use and distribution.

A display <NUM> can be, for example, an LCD integrated into the console <NUM> and used to display information to the clinician during a procedure. For example, the display <NUM> can be used to display location information, or depict a representation of the medical device (e.g., virtual medical device) in accordance with the location information, for a magnetized medical device within a limb of a patient. Alternatively, the display <NUM> can be separate from the console <NUM> instead of integrated into the console <NUM>; however, such a display is different than that of the alternative-reality headset <NUM>, which can also be configured to display location information (e.g., as a location-information overlay), or depict a representation of the medical device (e.g., virtual medical device) in accordance with the location information, for a magnetized medical device within a limb of a patient. The console <NUM> can further include a console button interface <NUM>. The console button interface <NUM> can be used by a clinician to immediately call up a desired mode (e.g., a mode with the magnetic-field generator <NUM>, a mode without the magnetic-field generator <NUM>, etc.) on the display <NUM> for use by the clinician in the procedure.

<FIG> provides a first medical-device detector <NUM> in accordance with some embodiments. <FIG> provides the first medical-device detector <NUM> about a limb of a patient in accordance with some embodiments. <FIG> provides a second medical-device detector <NUM> about a limb of a patient in accordance with some embodiments.

As shown, each medical-device detector of the first medical-device detector <NUM> and the second medical-device detector <NUM> includes the array of magnetic sensors <NUM> embedded within a housing <NUM>, <NUM> configured for placement about a limb (e.g., an arm or a leg) of a patient. The console <NUM> is configured to transform magnetic-sensor signals from the array of magnetic sensors <NUM> with the one or more algorithms <NUM> (e.g., a location-finding algorithm) into location information, or the representation of the medical device (e.g., virtual medical device) in accordance with the location information, for a magnetized medical device within the limb of the patient when the medical-device detector <NUM>, <NUM> is placed about the limb of the patient.

The housing <NUM> of the first medical-device detector <NUM> is a rigid frame. Each magnetic sensor of the array of magnetic sensors <NUM> embedded within the frame has a fixed spatial relationship to another magnetic sensor. The fixed spatial relationship is communicated to the console <NUM> upon connecting the first medical-device detector <NUM> to a port of the ports <NUM> of the console <NUM> or calling up one or more modes with the console button interface <NUM> of the console <NUM> for using the first medical-device detector <NUM> without the magnetic-field generator <NUM>. Using the fixed spatial relationship of the array of magnetic sensors <NUM> in the first medical-device detector <NUM>, the console <NUM> is able to transform the magnetic-sensor signals from the array of magnetic sensors <NUM> into the location information, or the representation of the medical device (e.g., virtual medical device) in accordance with the location information, for the magnetized medical device within the limb of the patient.

The housing <NUM> of the first medical-device detector <NUM> can further include one or more light-emitting diodes ("LEDs") or lasers embedded within the frame such as within a strut <NUM> of the frame. The one or more LEDs or lasers can be configured to illuminate the limb of the patient about which the first medical-device detector <NUM> is placed, or the one or more LEDs or lasers can be configured to illuminate just a portion of the limb of the patient. The portion of the limb of the patient can be the portion under which a tip of the medical device is located within the limb of the patient. (See, for example, <FIG>, wherein the 'X' indicates illumination of just a portion of the limb of the patient under which the tip of the medical device is located. ) As such, the one or more LEDs or lasers can function as a real-world light-based pointing system for identifying a medical device's location. The light-based pointing system can be used in conjunction with the alternative-reality headset <NUM> for confirmation of a medical device's location as the illumination provided by the light-based pointing system is visible through the see-through display <NUM> of the alternative-reality headset <NUM>.

The housing <NUM> of the second medical-device detector <NUM> is a drape. Each magnetic sensor of the array of magnetic sensors <NUM> embedded within the drape has a variable spatial relationship to another magnetic sensor depending upon how the drape is placed about the limb of the patient. For this reason, the medical device-locating system <NUM> can further include the magnetic-field generator <NUM>, which is configured to generate a magnetic field about the second medical-device detector <NUM> for determining the spatial relationship of one magnetic sensor of the array of magnetic sensors <NUM> to another magnetic sensor. Each magnetic sensor present in the array of magnetic sensors <NUM> is communicated to the console <NUM> upon connecting the second medical-device detector <NUM> to a port of the ports <NUM> of the console <NUM> or calling up one or more modes with the console button interface <NUM> of the console <NUM> for using the second medical-device detector <NUM> with the magnetic-field generator <NUM>. With each magnetic sensor of the array of magnetic sensors <NUM> known, the console <NUM> is configured to determine the spatial relationship of each magnetic sensor to another magnetic sensor from the magnetic-sensor signals produced by the array of magnetic sensors <NUM> while in the presence of the generated magnetic field. This is made possible, in part due to each magnetic sensor of the array of magnetic sensors <NUM> being in a unique magnetic environment with respect to at least the strength and orientation of the generated magnetic field. Using the determined spatial relationship of the array of magnetic sensors <NUM> in the second medical-device detector <NUM>, the console <NUM> is able to transform the magnetic-sensor signals from the array of magnetic sensors <NUM> into the location information, or the representation of the medical device (e.g., virtual medical device) in accordance with the location information, for the magnetized medical device within the limb of the patient. To ensure accuracy, the determined spatial relationship of the array of magnetic sensors <NUM> can be periodically confirmed in the presence of a newly generated magnetic field considering the medical device within the limb of the patient.

Again, <FIG> provides the block diagram for the medical device-placing system <NUM> in accordance with some embodiments.

As shown, the medical device-placing system <NUM> can include the ultrasound probe <NUM> of the anatomy-visualizing system <NUM>, the medical-device detector <NUM> including the array of magnetic sensors <NUM> of the medical device-locating system <NUM>, the alternative-reality headset <NUM>, and the console <NUM>, which includes electronic circuitry like that of both console <NUM> and console <NUM>. However, in some embodiments, other medical device-detecting technologies can be used instead of the medical device-locating system <NUM> such as the tip-location system ("TLS") of <CIT>.

<FIG> provides a block diagram for the ultrasound probe <NUM> and the medical-device detector <NUM> connected to the console <NUM> of the medical device-placing system <NUM> in accordance with some embodiments.

As shown, the console <NUM> has electronic circuitry including memory <NUM> and one or more processors <NUM>. Like the console <NUM>, the console <NUM> is configured to transform echoed ultrasound signals from a patient with one or more algorithms <NUM> to produce ultrasound images and ultrasound-image segments therefrom corresponding to anatomical structures of the patient. The console <NUM> is configured to capture in the memory <NUM> ultrasound-imaging frames in accordance with a pulsed-wave Doppler imaging mode of the ultrasound probe <NUM>, stitch the ultrasound-imaging frames together with a stitching algorithm of the one or more algorithms <NUM>, and segment the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with an image segmentation algorithm of the one or more algorithms <NUM>. The console <NUM> is configured to transform the ultrasound-image segments into objects of virtual anatomy with a virtualization algorithm of the one or more algorithms <NUM>. And like the console <NUM>, the console <NUM> is configured to transform magnetic-sensor signals from the array of magnetic sensors <NUM> with one or more algorithms <NUM> (e.g., a location-finding algorithm) into location information for a magnetized medical device within a limb of the patient when the medical-device detector <NUM> is placed about the limb of the patient. The console <NUM> is configured to send to the alternative-reality headset <NUM> by way of a wireless communications interface <NUM> both the objects of virtual anatomy and a representation of the medical device (e.g., virtual medical device) within the limb of the patient, in accordance with the location information, for display over the patient on the display <NUM> of the alternative-reality headset <NUM>. In displaying the objects of virtual anatomy and the representation of the medical device over the patient, the alternative-reality headset <NUM> can be configured to anchor the objects of virtual anatomy and the representation of the medical device to the patient, which is characteristic of mixed reality.

The console <NUM> includes a number of components of the medical device-placing system <NUM>, and the console <NUM> can take any form of a variety of forms to house the number of components. The one or more processors <NUM> and the memory <NUM> (e.g., non-volatile memory such as EEPROM) of the console <NUM> are configured for controlling various functions of the medical device-placing system <NUM> such as executing the one or more algorithms <NUM> during operation of the medical device-placing system <NUM>. A digital controller or analog interface <NUM> is also included with the console <NUM>, and the digital controller or analog interface <NUM> is in communication with the one or more processors <NUM> and other system components to govern interfacing between the probe <NUM>, the medical-device detector <NUM>, the alternative-reality headset <NUM>, as well as other system components.

The console <NUM> further includes ports <NUM> for connection with the medical-device detector <NUM> as well as additional, optional components such as the magnetic-field generator <NUM> or the optional components <NUM> (e.g., a printer, storage media, keyboard, etc.). The ports <NUM> can be USB ports, though other ports or a combination of ports can be used, as well as other interfaces or connections described herein. A power connection <NUM> is included with the console <NUM> to enable operable connection to an external power supply <NUM>. An internal power supply <NUM> (e.g., disposable or rechargeable battery) can also be employed, either with the external power supply <NUM> or exclusive of the external power supply <NUM>. Power management circuitry <NUM> is included with the digital controller or analog interface <NUM> of the console <NUM> to regulate power use and distribution.

A display <NUM> can be, for example, an LCD integrated into the console <NUM> and used to display information to the clinician during a procedure. For example, the console <NUM> can be used to display an ultrasound image of a targeted internal body portion of the patient attained by the probe <NUM>, the location information for a medical device within a limb of the patient, or a representation of the medical device (e.g., virtual medical device) in accordance with the location information for the medical device within the limb of the patient. Alternatively, the display <NUM> can be separate from the console <NUM> instead of integrated into the console <NUM>; however, such a display is different than that of the alternative-reality headset <NUM>, which can also be configured to display the objects of virtual anatomy and the representation of the medical device (e.g., virtual medical device) within the limb of the patient.

The console <NUM> can further include a console button interface <NUM>. In combination with control buttons on the probe <NUM>, the console button interface <NUM> can be used by a clinician to immediately call up a desired ultrasound-imaging mode (e.g., a continuous wave imaging mode or a pulsed-wave imaging mode) on the display <NUM> for use by the clinician in the procedure. The console button interface <NUM> can be used by the clinician to immediately call up a desired medical device-locating mode (e.g., a mode with the magnetic-field generator <NUM>, a mode without the magnetic-field generator <NUM>, etc.) on the display <NUM> for use by the clinician in the procedure.

With respect to the ultrasound probe <NUM> and the alternative-reality headset <NUM> of the medical device-placing system <NUM>, reference should be made to the description of the ultrasound probe <NUM> and the alternative-reality headset <NUM> provided for the anatomy-visualizing system <NUM>. With respect to the medical-device detector <NUM> and the magnetic-field generator <NUM> of the medical device-placing system <NUM>, reference should be made to the description of the medical-device detector <NUM> and the magnetic-field generator <NUM> provided for the medical device-locating system <NUM>.

Methods of the medical device-placing system <NUM> incorporate methods of both the anatomy-visualizing system <NUM> and the medical device-locating system <NUM>, which methods are discernable by references to the anatomy-visualizing system <NUM>, the medical device-locating system <NUM>, or the components thereof (e.g., the ultrasound probe <NUM>, the medical-device detector <NUM>, etc.) below.

Methods of the medical device-placing system <NUM> include emitting ultrasound signals into a limb of a patient and receiving echoed ultrasound signals from the patient's limb by way of the piezoelectric sensor array <NUM> of the ultrasound probe <NUM>; transforming the echoed ultrasound signals with the console <NUM> having the electronic circuitry including the memory <NUM>, the one or more algorithms <NUM>, and the one or more processors <NUM> to produce ultrasound-image segments corresponding to anatomical structures of the patient's limb; inserting a magnetized medical device into the patient's limb and transforming magnetic-sensor signals from the array of magnetic sensors <NUM> embedded within the housing <NUM>, <NUM> placed about the patient's limb with the one or more algorithms <NUM> (e.g., a location-finding algorithm) of the console <NUM> into location information for the medical device within the patient's limb; displaying over the patient's limb on the see-through display screen <NUM> of the alternative-reality headset <NUM> having the electronic circuitry including the memory <NUM> and the one or more processors <NUM> in the frame <NUM> coupled to the display screen <NUM> a virtual medical device in accordance with the location information for the medical device within objects of virtual anatomy corresponding to the ultrasound-image segments. Ultrasound imaging to produce the objects of virtual anatomy can be done at any time before inserting the medical device into the patient's limb, and the objects of virtual anatomy can be stored for later use in the memory <NUM> of the console <NUM> or a storage medium connected to a port of the console <NUM>.

The method can further includes capturing in the memory <NUM> of the console <NUM> ultrasound-imaging frames in accordance with the pulsed-wave Doppler imaging mode of the ultrasound probe <NUM> while emitting and receiving the ultrasound signals; stitching the ultrasound-imaging frames together with the stitching algorithm of the one or more algorithms <NUM>; and segmenting the ultrasound-imaging frames or the stitched ultrasound-imaging frames into the ultrasound-image segments with the image segmentation algorithm of the one or more algorithms <NUM>.

The method can further includes transforming the ultrasound-image segments into the objects of virtual anatomy with the virtualization algorithm of the one or more algorithms <NUM>; and sending both the virtual medical device and the objects of virtual anatomy to the alternative-reality headset <NUM> for display over the patient's limb.

The method can further includes anchoring the virtual medical device and the objects of virtual anatomy to the patient's limb over which the virtual medical device and the objects of virtual anatomy are displayed.

The method can further includes capturing in the memory <NUM> of the console <NUM> eye movements of the wearer using the one or more eye-tracking cameras <NUM> coupled to the frame <NUM> of the alternative-reality headset <NUM>; and processing the eye movements with the eye-movement algorithm of the one or more algorithms <NUM> to identify a focus of the wearer for selecting or enhancing the objects of virtual anatomy corresponding to the focus of the wearer.

The method can further includes capturing in the memory <NUM> of the console <NUM> gestures of the wearer using one or more patient-facing cameras <NUM> coupled to the frame <NUM> of the alternative-reality headset <NUM>; and processing the gestures with the gesture-command algorithm of the one or more algorithms <NUM> to identify gesture-based commands issued by the wearer for execution thereof by the alternative-reality headset <NUM>.

The method can further includes capturing in the memory <NUM> of the console <NUM> audio of the wearer using the one or more microphones <NUM> coupled to the frame <NUM> of the alternative-reality headset <NUM>; and processing the audio with the audio-command algorithm of the one or more algorithms <NUM> to identify audio-based commands issued by the wearer for execution thereof by the alternative-reality headset <NUM>.

The method can further include generating a magnetic field with the magnetic-field generator <NUM>; and determining a spatial relationship of each magnetic sensor of the array of magnetic sensors <NUM> to another magnetic sensor from the magnetic-sensor signals produced by the array of magnetic sensors <NUM> while in the presence of the generated magnetic field. Determining the spatial relationship of each magnetic sensor to another magnetic sensor in the array of magnetic sensors <NUM> is important when the array of magnetic sensors <NUM> is embedded within the housing <NUM> (e.g., a drape), for the magnetic sensors have a variable spatial relationship to each other depending upon how the housing <NUM> is placed about the limb of the patient.

Claim 1:
A medical device-placing system (<NUM>), comprising:
an ultrasound probe (<NUM>) configured to emit ultrasound signals into a limb of a patient and receive echoed ultrasound signals from the patient's limb by way of a piezoelectric sensor array (<NUM>);
an array of magnetic sensors (<NUM>) embedded within a housing configured for placement about the patient's limb;
a console (<NUM>) having electronic circuitry including memory and a processor configured to:
transform the echoed ultrasound signals to produce ultrasound-image segments corresponding to anatomical structures of the patient's limb, and
transform magnetic-sensor signals from the array of magnetic sensors (<NUM>) into location information for a magnetized medical device within the patient's limb when the housing is placed about the patient's limb; and
an alternative-reality headset (<NUM>), including:
a frame (<NUM>) having electronic circuitry including memory and a processor; and
a display screen (<NUM>) coupled to the frame through which a wearer of the alternative-reality headset (<NUM>) can see the patient's limb, the display screen (<NUM>) configured to display over the patient's limb a virtual medical device in accordance with the location information for the medical device within objects of virtual anatomy corresponding to the ultrasound-image segments;
wherein the console (<NUM>) is configured to transform the ultrasound-image segments into the objects of virtual anatomy with a virtualization algorithm and send both the virtual medical device and the objects of virtual anatomy to the alternative-reality headset (<NUM>) for display over the patient's limb.