Patent ID: 12197637

FIG.1shows the basic flow of the method according to the first aspect, which starts with a step S11of generating the augmented reality device registration, followed by a step S12of acquiring the object position information. In subsequent step S13, the object presence information is acquired, followed by a final step S14of determining and displaying the augmentation information.

FIG.2shows a first way of specifying the steps ofFIG.1. In step S21, the augmentation system (augmented reality device4) uses its distance measurement unit (depth sensor) to identify a body volume corresponding to an anatomical body part or a tracking markers (marker device) or other physical object6by measuring a distance between the distance measurement unit and the body volume or the tracking markers or other physical object6, respectively. The result is obtained in step S22as the position of the body volume or the tracking markers (a marker device) or other physical object6, respectively, in a virtual coordinate space (coordinate system) associated with the augmented reality device5. Steps S23and S24may be executed subsequently, before or in parallel to (simultaneously with) steps S21and S22. Step S21encompasses position tracking (identification) of the tracking markers by a tracking device embodying the position detector4of the position tracking system2. The result of the tracking is obtained in step S24as a position of the tracking markers in a virtual coordinate space associated with the position tracking system2, namely the tracking coordinate system. In step S25, the two virtual coordinate spaces are registered with one another (i.e. the spatial relationship, in particular transformation of bases between the two coordinate spaces is determined, for example by point-based rigid registration).

In subsequent step S26, the position of the tracking device is determined in the coordinate system associated with the augmented reality device5. In step S27, the field of view of the tracking system2is matched to the three-dimensional depth image (generated by the distance measurement using the distance measurement unit of the augmented reality device5) to identify a physical area visible to the tracking system (i.e. to identify the field of view of the position tracking system2in the real image acquired by the augmented reality device5). In step S28, this match is used to display information (the augmentation information) in the augmented reality device (also called augmentation device or augmentation system)5, for example to augment areas in the real image covered or non-covered, respectively, by the field of view of the tracking system2, with augmentation information.

FIG.3shows a variation of the example ofFIG.2in which step S31encompasses the augmented reality device5using a video camera to identify a body volume, a tracking device, a marker device (tracking marker) or another physical object6. Subsequently, the position of the body volume and the physical object is determined in a virtual coordinate space associated with the augmented reality device5. Steps S33and S34may be executed before, after or in parallel to (i.e. simultaneously with) steps S31and S32. Steps S33and S34correspond to steps S23and S24, respectively, ofFIG.2. Subsequent step S35corresponds to step S25ofFIG.2, and is followed by steps S26to S28ofFIG.2.

FIG.4shows a further variation of the example ofFIG.2in which step S41encompasses the augmented reality device5using point-based registration to register a body volume by depth scanning (i.e. surface scanning using the distance measurement unit of the augmented reality device5). Step S42may be executed before, after or in parallel to (i.e. simultaneously with) step S41. Subsequent step S43corresponds to step S25ofFIG.2, and is followed by steps S26to S28ofFIG.2.

FIG.5shows a combination of an even further variation of the example ofFIG.2with a variation of the example ofFIG.3in which steps S51and S52correspond to steps S31and S32, respectively. S51encompasses the augmented reality device5using point-based registration to register a body volume by depth scanning (i.e. surface scanning using the distance measurement unit of the augmented reality device). Steps S53and S54may be executed before, after or in parallel to (i.e. simultaneously with) steps S51and S52. In step S53, the position tracking system2identifies both the marker device and the augmentation system (by applying the respective tracking technology). The result of step S53is determined in step S54by determining the position of the marker device and the position of the augmented reality device5in step S54in a virtual coordinate space associated with the position tracking system2(the tracking coordinate system). Subsequent step S55corresponds to step S25ofFIG.2, and is followed by steps S26to S28ofFIG.2.

FIG.6gives a schematic overview of the medical system1according to the sixth aspect. The medical system1comprises a position tracking system2which comprises a computer3which is operably coupled to a multi-dimensional position detector4. The augmented reality device5and optionally a physical object6are also part of the medical system1. Within the framework of this disclosure, examples are given for determining a registration (relative position) R1between the position tracking system2and the physical object6, a registration (relative position) R2between the physical object6and the augmented reality device5, and a registration (relative position) R3between the augmented reality device5and the position tracking system2.

FIG.7shows an example of augmentation information embodied by a graphical representation of the field of view7of the position tracking system2which is overlaid onto a real image depicting a marker device8and an anatomical body part9comprising a part of a lower limb. As can be seen fromFIG.7, the marker device8is at least partly within the field of view of the field of view7.

The method according to the first aspect may be summarized as follows:First a correspondence of the coordinate systems of the tracking device and the augmentation system needs to be determined.

One of the following methods (or a combination thereof) may be applied:

The augmentation system identifies a tracking marker (assembly) through the use of video cameras

The augmentation system identifies a body volume via depth sensors that is registered to tracking markers

The augmentation system identifies a volume via depth sensors that is also picked by depth sensors contained in the tracking system

The augmentation system possesses tracking markers itself

The augmentation system identifies the tracking device itself through video cameras or depth sensors.

Based on the established correspondence, otherwise invisible aspects related to the tracking device can be augmented to the user's view by converting virtual objects from the tracking device into the augmentation coordinate system.

This can further comprise mapping this information on to a surface picked up by the depth sensors or determining the shadow area of objects depicted by the depth sensors of the augmentation system which might be invisible to the tracking device.

The tracking device may be an infra-red tracking device (usable in surgery or radiotherapy), an E/M (electromagnetic) tracking device, articulated arm or any other tracking device. The augmentation device could be any device with the ability to use another tracking method from a different direction. The second device incorporates a display, e.g. a head-mounted display in glasses worn by the user.

To improve surgical navigation or patient positioning in radiotherapy the following could be calculated and displayed on the augmented reality device, specifically in augmented reality glasses:The volume currently captured by the tracking system (i.e. its field of view) is displayed. Vice-versa areas currently not seen by the surgical navigation system can be marked.

Labels noting the estimated tracking accuracy can be displayed, e.g. on the edges of an EM tracking area

Markers or devices currently not seen by the tracking system but essential for correct utilization are labeled so that the user understands that they are not seen.

Objects that currently obstruct the tracking systems' line of sight (e.g. surgeons' hands) can be marked and the shadow that they are casting with respect to the tracking device can be displayed.

Devices seen by the surgical navigation system can be annotated with their labels.

Warnings can be displayed during the procedure, e.g. warnings to avoid contact with reference arrays.

If multiple augmented devices are in use, the specific content displayed in each device can be individually determined, e.g. by displaying accuracy warnings in the assistant's display and no warnings in the surgeon's display.