Patent Description:
This disclosure generally relates to disposable stickers or patches, which may be used for registration of devices within augmented reality (AR) environments.

Interventional medical procedures can have complex work environments with multiple medical devices being introduced at various stages, each with their own particular setup mechanisms. Augmented reality (AR) devices are being developed for use in the interventional suite. It is desirable to provide user-friendly mechanisms for tracking use and reliable system setup. Devices operating within such an environment may have strict requirements on cybersecurity and patient confidentiality, which may require engineering trade-offs and sacrificing ease of use in conventional systems for registering the devices. Documents <CIT>, <CIT> and <CIT> form part of the technical background.

The disclosed embodiments have advantages and features which will be more readily apparent from the detailed description, the appended claims, and the accompanying figures (or drawings). A brief introduction of the figures is below.

Disclosed herein are systems and methods for using stickers, for example, in interventional medical procedure applications. Moreover, the stickers may be used for system security, device registration, or presenting graphics or other data in an augmented reality (AR) environment. To provide useful graphics in an AR environment, spatial locations should be marked and tracked in a non-ambiguous manner. Reference graphics should be placed precisely and dynamically with respect to objects they are referencing.

In one aspect, the sticker system receives an image of a sticker captured by an imaging sensor, where the image shows a fiducial marker of the sticker. The sticker system determines registration information by processing the fiducial marker of the sticker in the image. The sticker system determines location information of a head-mounted device (HMD) worn by a user. The sticker system determines a position of the sticker relative to the HMD using the registration information and the location information. The sticker system provides a graphic to the HMD for presentation to the user based on the position of the sticker.

In some embodiments, the HMD (or sticker system) generates an encryption key using the registration information, where the encryption key is unique to the sticker. The HMD communicates with another device using the encryption key.

In some embodiments, the HMD (or sticker system) generates an encryption key using the registration information for a medical procedure, where the encryption key is unique to the sticker. Responsive to determining that the encryption key was previously used by another device for a different medical procedure, the sticker system determines to not use the encryption key for the medical procedure.

In some embodiments, the sticker system determines additional registration information by processing an additional image showing the fiducial marker of the sticker, where the additional image is captured by an additional HMD. The sticker system determines that the HMD and the additional HMD are being used for a same medical procedure by comparing the registration information and the additional registration information.

In some embodiments, the sticker system determines position of the sticker relative to the HMD by determining a distance and angle in 3D between the sticker and the HMD. In some embodiments, size or orientation of the graphic is determined using the distance and angle.

In some embodiments, the sticker system generates augmented reality content as the graphic, and the HMD includes a display configured to present the augmented reality content. In some embodiments, the imaging sensor is coupled to the HMD. In some embodiments, the fiducial marker is a QR code or a barcode.

In some embodiments, a non-transitory computer-readable storage medium stores instructions that when executed by one or more processors cause the one or more processors to perform steps including any of the methods described herein.

Figure (<FIG> is an illustration of a system environment, according to various embodiments of the present invention. The system environment shown in <FIG> includes the sticker system <NUM>, a head-mounted device (HMD) <NUM> of a user <NUM>, and a computing device <NUM>, which can be communicatively connected to each other via a network <NUM>, e.g., using one or more network access points <NUM>. The network <NUM> may include wireless and/or wired connections. In other embodiments, the system environment may include any number of users and HMDs. The functions performed by the various entities of <FIG> may vary in different embodiments.

The HMD <NUM> may use an imaging sensor to capture an image of a scene <NUM> that includes a sticker <NUM>. Additionally, an HMD <NUM> may include one or more of: a sensor to capture information for determining position or orientation of the HMD <NUM> in physical space (e.g., in two or three dimensions), a microphone to capture audio input of users, a speaker for producing audio output, an imaging sensor for capture of images including digital information encoded in the sticker <NUM>, a wireless means to communicate information (e.g., audio or other sensor data such as position or orientation information) between HMDs <NUM> or other types of devices, a means for storing information, and one or more processors for computing or controlling the audio communicated between the HMDs <NUM>. In some embodiments, one or more HMDs <NUM> may be a MICROSOFT HOLOLENS, though various other types of HMDs <NUM> can also be used including HMDs <NUM> custom-designed for a medical environment. Example sensors include accelerometers, gyroscopes, inertial measurement units (IMU) (e.g., for determining location information of the HMD <NUM>), depth cameras, and global positioning system (GPS) sensors, ultrasonic sensors, infrared sensors, and proximity sensors, among others.

In some embodiments, a HMD <NUM> includes an electronic display that displays images to the user <NUM> in accordance with data received from the sticker system <NUM>, another HMD <NUM>, or another source. Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a liquid crystal on silicon display (LCOS), some other display, or some combination thereof. In some embodiments, the electronic display may include one or more optical lens, optical waveguides, or optical combiners. In some embodiments, a HMD <NUM> does not necessarily include an electronic display.

The computing device <NUM> may include any number of a display, keyboard, touchscreen, optical cameras, network communication device, or processor. The computing device <NUM> may use an imaging sensor to capture an image including digital information encoded in the sticker <NUM>. The computing device <NUM> may also use other means to capture the digital information encoded in the sticker <NUM> (e.g., touchscreen, keyboard, RFID, etc.). The computing device <NUM> may set a wireless encryption key on the network access point <NUM> using the captured digital information encoded in the sticker <NUM>. The computing device <NUM> may receive imaging information captured from an imaging sensor on the HMD <NUM>. The computing device <NUM> can process or analyze the received imaging information, for example, to provide graphics to one or more HMDs <NUM>.

A sticker <NUM> includes information that can be detected and used by the sticker system <NUM> for one or more applications. The sticker <NUM> may be a one-time use, disposable, and/or printed type of sticker. The sticker <NUM> may include an adhesive for securing the sticker <NUM> to a surface. In other embodiments, stickers may not necessarily have an adhesive, but rather resemble a patch, and may be secured to surfaces using other means such as magnets, Velcro, weights, pins, clips, or buttons, among other types of attachment mechanisms. The sticker <NUM> includes one or more means to establish orientation of the sticker <NUM> relative to an imaging sensor, and a means for optically encoding (e.g., digital) information, for example, a fiducial marker such as a QR code or barcode.

The sticker system <NUM> may identify one or more stickers <NUM> present in an environment. The sticker system <NUM> may receive information associated with a sticker <NUM> from a HMD <NUM> or computing device <NUM>, which can detect stickers <NUM> using a sensor. The information may include digital information encoded in the sticker <NUM> and/or other information about a captured scene <NUM>, e.g., presence, size, and position of other objects in vicinity of the sticker <NUM>. As an example, the sticker system <NUM> can use the received information to calculate a positional offset <NUM> from the position of the HMD <NUM> to position of the sticker <NUM>. The sticker system <NUM> may use the offset <NUM> to generate, modify, or display digital content <NUM> relative to position of the HMD <NUM> and/or sticker <NUM>.

The sticker system <NUM> may operate in an AR system, virtual reality (VR) system, mixed reality system, or other types of system environments not necessarily including AR or VR devices. In an AR environment, the (e.g., dynamic) detected position of the sticker <NUM> may be used as a reference against which AR graphics are positioned for presentation to the user <NUM> of the HMD <NUM>. The sticker system <NUM> may run on a HMD <NUM> or a different computing device <NUM> such as a server, which may be outside of a room in which a medical procedure is performed.

In some embodiments, the sticker system <NUM> uses existing mapping data available, collected, or established during system installation or calibration. The sticker system <NUM> may use information from a HMD <NUM> to establish a correction between different mapping data sets. For example, responsive to determining that the position of a sticker <NUM> as determined using mapping data from a HMD <NUM> deviates (e.g., by at least a threshold discrepancy in position or orientation) than the observed position (or orientation) as sensed by an imaging sensor (e.g., of another computing device <NUM>), the sticker system <NUM> may apply an offset to one or more data sets to correct the disparity, e.g., modify X, Y, or Z coordinates in a Cartesian coordinate system.

In some embodiments, responsive to the sensed position and/or orientation of the sticker <NUM>, the sticker system <NUM> (or HMD <NUM>) may establish an independent coordinate system to use for displaying AR information. In an aspect, responsive to sensing data from a sticker <NUM>, the sticker system <NUM> may combine other sources of data (e.g., private data) with the sensed data to generate a one-time use procedure dataset, e.g., for a surgical or another type of medical or interventional procedure. In another aspect, the HMD <NUM> or other computing device <NUM> may use human input to acquire digital information combined with private data for generating a one-time use procedure dataset. In some embodiments, the sticker system <NUM> uses human input for positioning of reference graphics relative to a detected position of the sticker <NUM> to determine a reference offset that is applied to the graphics to compensate for subsequent movement of the sticker <NUM>.

In some embodiments, the sticker system <NUM> (e.g., by a HMD <NUM>) may use information associated with a sticker <NUM> to generate an encryption key to communicate with the network access point <NUM>. In some embodiments, the sticker system <NUM> uses elliptic curve cryptography (ECC), which is an example class of public key cryptography with a symmetric key exchange. The sticker system <NUM> uses ECC to generate encryption keys, which may be used by the network access point <NUM> or other security devices or systems to verify the authenticity of HMDs <NUM> or computing devices <NUM>. In some embodiments, the sticker <NUM> can provide an extra token used by the sticker system <NUM> to verify the authenticity, e.g., after the ECC verification. In various embodiments, a sticker <NUM> provides enhanced security because the sticker <NUM> needs to be physically present nearby a HMD <NUM> such that the HMD <NUM> can capture an image of the sticker <NUM>, e.g., a QR code of the sticker <NUM> in particular.

In various embodiments, the sticker system <NUM> implements a one-time use procedure dataset to prevent re-use of stickers <NUM>. Once a one-time use sticker <NUM> is used, the sticker <NUM> may be disposed. The sticker system <NUM> may determine unauthorized use of stickers <NUM> by tracking previously used one-time use stickers <NUM>. The sticker system <NUM> may use the procedure dataset to establish a one-time use encryption key for wired or wireless network communication. The sticker system <NUM> can also detect and use encoded information of a sticker <NUM> as a component of a symmetric key being exchanged as a transport layer encryption. In some embodiments, the sticker system <NUM> uses information associated with stickers as an encryption key for the hardware layer in wireless key exchange, or as a seed for the negotiation of the wireless encryption key. In an embodiment, the sticker system <NUM> may use information associated with stickers <NUM> as a component for an application layer encryption scheme. In an embodiment, the sticker system <NUM> uses information associated with stickers <NUM> as an encryption key or as a seed for an encryption key for transmitting application data over the network <NUM>.

In various embodiments, the sticker system <NUM> may use information associated with stickers <NUM> in use cases including multiple HMDs <NUM> and/or users <NUM>, e.g., physicians and/or other personnel performing a medical procedure. For example, a HMD <NUM> may communicate the calculated offset <NUM> over the network <NUM> in a shared coordinate system between multiple HMDs <NUM> to establish a correction for one or more (or all) of the other HMDs <NUM>. In this aspect, one or more HMDs <NUM> may display or modify digital content <NUM> responsive to the sticker system <NUM> determining that at least one HMD <NUM> detected a sticker <NUM> and captured the associated encoded information. In another aspect, a particular HMD <NUM> may use position or orientation sensors to display digital content <NUM> responsive to determining that at least a portion of the sticker <NUM> is outside of the field view of an imaging sensor of the HMD <NUM>.

In some embodiments, the sticker system <NUM> may use information from one or more of HMDs <NUM> to map the location of a given one of the HMDs <NUM> relative to a particular sticker <NUM>. In some embodiments, each of multiple HMDs <NUM> may independently calculate the offset <NUM> to position of the sticker <NUM> for displaying digital content <NUM>. The sticker system <NUM> may customize graphics for a certain HMD <NUM> based on the corresponding offset <NUM>.

<FIG> is a data flow diagram of the sticker system <NUM>, according to various embodiments of the present invention. The sticker system <NUM> may use digital information captured from the sticker <NUM> or input by a user to generate one or more of position and/or orientation data, encryption keys, and one-time use procedure codes. In various embodiments, an imaging sensor <NUM> (e.g., of HMD <NUM> or computing device <NUM>) captures an image including information encoded by the sticker <NUM>. The sticker system <NUM> can use calibration parameters <NUM> for the imaging sensor <NUM> and a coordinate system for a code recognition algorithm <NUM> to extract location information <NUM>, e.g., one or more of scale, distance, and angle information in up to three dimensions. The distance and angle may indicate a relative position between the imaging sensor <NUM> and the sticker <NUM>. The scale may indicate a size of the sticker <NUM> relative to an image captured by the imaging sensor <NUM> or another reference. The sticker system <NUM> may use the location information <NUM> to determine an origin for a (e.g., 3D) coordinate system <NUM>, for example, for displaying 3D AR content on HMDs <NUM> or other displays.

The sticker system <NUM> can use the code recognition algorithm <NUM> to generate a one-time use procedure code <NUM>. Alternatively, the one-time use procedure code <NUM> may be provided to the sticker system <NUM> by user input <NUM>. The sticker system <NUM> may use the one-time use procedure code <NUM> with installation specific data <NUM> in procedure specific coding <NUM> to generate one or more unique encryption keys <NUM> and unique procedure code <NUM>. The sticker system <NUM> may use the unique procedure code <NUM> in procedure validation <NUM> to verify authenticity and single use of the one-time use procedure code <NUM>.

In an embodiment, each networked device independently completes process shown in <FIG>, e.g., such that all devices in the sticker system <NUM> have their respective key to establish network communications. However, for security purposes, neither the key itself, nor any information used to generate the key, is transmitted wirelessly between devices.

<FIG> is an illustration of example stickers, according to various embodiments of the present invention. Sticker <NUM> includes a QR code with anatomical words adjacent to the QR code to serve as a hint or indication to a user to position the sticker <NUM> in relation to a patient on a (e.g., surgical operating) table or another object. Sticker <NUM> includes a QR code with a humanoid or cartoon icon (representing a patient) showing the correct orientation for sticker placement.

In some embodiments, a sticker includes a fiducial marker or an optical pattern that is unique in rotation and of known scale. The fiducial marker may encode bits of data, which may be used for location calculation or security applications. A QR code is scale invariant and may be produced (e.g., printed) to known size. Based on known sizes of QR codes or other attributes of QR codes, the sticker system <NUM> may encode one or more of scale, rotation, and additional data in the same instance of a QR code or another type of unique fiducial marker. A sticker may also independently encode scale and rotation from additional data bits in a separate marker, for example, a barcode or a sequence of alphanumeric and/or hexadecimal characters. The sequence of characters may be adjacent to a rotationally unique optical marker of known scale.

<FIG> is a flowchart of a process <NUM> of device registration using stickers, according to various embodiments of the present invention. In various embodiments, the sticker system <NUM> may register a device such as an HMD <NUM> (or computing device <NUM>) to authorize the HMD <NUM> for receiving or providing information associated with a medical or interventional procedure. Additionally or alternatively, registration can be used to determine (or recalibrate) position of a registered device relative to boundaries of a room or another reference. The sticker system <NUM> may perform the method <NUM> in the system environment shown in <FIG>.

The sticker system <NUM> receives <NUM> an image of a sticker <NUM> captured by an imaging sensor (e.g., of a HMD <NUM> or computing device <NUM>), where the image shows a fiducial marker of the sticker <NUM>, e.g., resembling the example QR codes shown in <FIG>. The sticker system <NUM> determines <NUM> registration information by processing the fiducial marker of the sticker <NUM> in the image. Methods for detection of sticker fiducial markers are well known in the literature. In an embodiment, the sticker fiducial markers include high contrast black and white graphics such as a QR code. The registration information may include any number of attributes of the fiducial marker such as a size or orientation of a QR code. The registration information may include other types of information related to a medical procedure, e.g., a room in which the procedure will be conducted, locations or dimensions of objects in the room (which may be used as references for positioning), patient, user, or device information (e.g., HMDs) involved in the medical procedure.

The sticker system <NUM> determines <NUM> location information of a HMD <NUM> worn by a user <NUM>. The imaging sensor may be coupled to the HMD <NUM>. The sticker system <NUM> determines <NUM> a position of the sticker <NUM> relative to the HMD <NUM> using the registration information and the location information. The sticker system <NUM> provides <NUM> a graphic and appropriate 3D positioning information relative to the HMD <NUM> for presentation to the user <NUM> based on the position of the sticker <NUM>. Referring to the example illustrated in <FIG>, the graphic may include AR content such as a 3D rendering of physiology of a patient. For instance, a graphic of a heart (or a model of another physiological body part) is displayed by the HMD <NUM> for a cardiology procedure. The graphic may indicate position of a medical instrument relative to a body part. For example, the graphic may show a catheter being navigated into a patient's body by a physician.

In various embodiments, the sticker <NUM> may be attached to an operating table or another (e.g., stationary) object in vicinity of the patient during the procedure. In other embodiments, the sticker <NUM> is affixed to a non-stationary object (e.g., a mobile table, chair, or equipment), and the sticker system <NUM> can update the registration information as new images of the sticker <NUM> are received. Thus, in an AR environment, the sticker system <NUM> may provide graphics that are perceived by the user <NUM> as attached to a moving object.

In some embodiments, the sticker system <NUM> determines a size or orientation of the graphic using the position of the sticker <NUM> relative to the HMD <NUM>. As an example, as the user <NUM> of the HMD <NUM> walks around the patient or sticker <NUM>, the sticker system <NUM> may update the graphic to show a different perspective or angle, e.g., of a 3D model of the patient's heart. Additionally, the sticker system <NUM> can update the size of the graphic as the user <NUM> walks closer to or further away from the patient or sticker <NUM>.

In some embodiments, the sticker system <NUM> updates graphics responsive to detecting that a user <NUM> is gesturing or otherwise interacting with a sticker <NUM>, e.g., pointing toward or at the location of the sticker <NUM>. In some embodiments, the sticker system <NUM> generates an encryption key using the registration information (e.g., for a medical procedure), where the encryption key is unique to the sticker <NUM>.

In one use case, the sticker system <NUM>, executing on the HMD <NUM>, may communicate with another device using the encryption key. In a different example, responsive to determining that the encryption key was previously used by another device for a different medical procedure, the sticker system <NUM> determines to not use the encryption key for a current medical procedure. Thus, the sticker system <NUM> maintains the stickers for one-time use, e.g., for a particular procedure.

In some embodiments, the sticker system may use stickers in environments with multiple HMDs. For instance, the sticker system <NUM> processes an additional image to determine that the HMD <NUM> and another HMD are being used for a same medical procedure by comparing their associated registration information. Responsive to this determination, the sticker system <NUM> may coordinate information provided for presentation on the HMDs. For example, the sticker system <NUM> may synchronize annotations on graphics displayed on the HMDs, communicate messages between the HMDs, or share other data about the medical procedure between the HMDs.

The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

In one embodiment, a software module is implemented with a computer program product including a computer-readable non-transitory medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Embodiments of the invention may also relate to a product that is produced by a computing process described herein. Such a product may include information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Claim 1:
A method comprising:
receiving an image of a sticker captured by an imaging sensor, the image showing a fiducial marker of the sticker;
determining registration information by processing the fiducial marker of the sticker in the image;
determining location information of a head-mounted device, "HMD", worn by a user;
determining a position of the sticker relative to the HMD using the registration information and the location information;
providing a graphic to the HMD for presentation to the user based on the position of the sticker;
characterized by determining that the HMD and a computing device are being used for a medical procedure; and
responsive to determining that the HMD and the computing device are being used for the medical procedure, synchronizing annotations during the medical procedure on the graphic presented by the HMD and one or more graphics presented by the computing device.