Patent Description:
Augmented-Reality (AR) and Mixed-reality (MR) solutions are known which detect real-world objects in the real-world scene, i.e., the physical environment, surrounding a user wearing a Head-Mounted Display (HMD) headset using sensors, such as cameras, Lidar, etc, and displaying such objects as an overlay onto the real-world scene viewed by the user. Such an HMD typically comprises a see-through display through which the user can view the real-world scene, and on which virtual content is displayed such that it is overlaid onto the real-world scene.

As an example, the C-THRU helmet by Qwake Technologies (https://www. tech/, https://www. net/gallery/<NUM>/C-Thru-Smoke-Diving-Helmet) has been developed as a tool for firefighters operating in environments of low or limited visibility, e.g., due to dense smoke. The C-THRU helmet uses a thermal optical (infrared) camera for capturing images of the real-world scene surrounding the user in the infrared range, which is less impacted by smoke as compared to the visible range. Therefore, in comparison with human vision which relies on visible light, the infrared camera is able to detect objects at an increased or extended range, which objects would not be visible to the user relying on his/her human vision only, e.g., due to smoke.

The C-THRU helmet relies on image processing of a video or image sequence captured by the infrared camera, and enhancing the edges or contours of objects which are captured in the video (a contour is the edge enclosing an object; if an edge defines an object, it becomes a contour), resulting in a wireframe image of edges. The edges are then overlaid onto the real-world scene which the user views through the see-through display of the C-THRU helmet.

The human vision is generally superior in terms of resolution, wide-range vision, etc, and in most circumstances outperforms digitally captured and displayed content. In situations of limited visibility, like dense smoke, but also dense vapor or darkness, solutions like the C-THRU helmet may advantageously be used for enhancing the human vision, thereby extending the range in which objects become visible to a human user of an AR headset.

Other examples of solutions for enhancing the human vision are described in:.

However, since visibility may change dynamically, e.g., while the user is moving through a smoky environment, or because smoke itself is not static and its density may change over time, displaying real-world objects as overlaid virtual content irrespective of the actual, current visibility within the Field-of-View (FoV) of the user may lead to situations where the natural (human-vision) view of the real-world by the user is disturbed or hampered. For example, this may be the case if real-world objects which the user in fact can see are displayed as virtual content and overlaid onto the user's (human-vision) view of the real-world scene.

It is an object of the invention to provide an improved alternative to the above techniques and prior art.

More specifically, it is an object of the invention to provide improved solutions for augmenting human vision in limited-visibility environments.

These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.

According to a first aspect of the invention, a Mixed-Reality (MR) device for augmenting human vision in limited-visibility environments is provided. The MR device comprises a visible-range sensor for capturing a visible-range representation of a real-world scene which surrounds the MR device. The visible-range sensor is configured to capture the visible-range representation of the real-world scene with a range which is commensurate with that of human vision. The MR device further comprises an extended-range sensor for capturing an extended-range representation of the real-world scene. The extended-range sensor is configured to capture the extended-range representation of the real-world scene with an extended range which is wider than the range which is commensurate with human vision. The MR device further comprises a display and processing circuitry. The processing circuitry causes the MR device to be operative to generate a visible-range representation of edges of one or more physical objects which are present in the real-world scene. The visible-range representation of edges of the one or more physical objects is generated by extracting edges from the visible-range representation captured by the visible-range sensor. The MR device is further operative to generate an extended-range representation of edges of the one or more physical objects. The extended-range representation of edges of the one or more physical objects is generated by extracting edges from the extended-range representation captured by the extended-range sensor. The MR device is further operative to generate a delta representation of edges of the one or more physical objects. The delta representation of edges comprises edges which are present in the extended-range representation of edges, but are absent in the visible-range representation of edges. The MR device is further operative to display the delta representation of edges on the display.

According to a second aspect of the invention, a method of augmenting human vision in limited-visibility environments is provided. The method is performed by an MR device and comprises capturing a visible-range representation of a real-world scene surrounding the MR device. The visible-range representation of the real-world scene is captured with a range which is commensurate with human vision. The visible-range representation of the real-world scene is captured using a visible-range sensor which is comprised in the MR device. The method further comprises capturing an extended-range representation of the real-world scene. The extended-range representation of the real-world scene is captured with an extended range which is wider than the range which is commensurate with human vision. The extended-range representation of the real-world scene is captured using a an extended-range sensor which is comprised in the MR device. The method further comprises generating a visible-range representation of edges of one or more physical objects which are present in the real-world scene. The visible-range representation of edges of the one or more physical objects is generated by extracting edges from the visible-range representation captured by the visible-range sensor. The method further comprises generating an extended-range representation of edges of the one or more physical objects. The extended-range representation of edges of the one or more physical objects is generated by extracting edges from the extended-range representation captured by the extended-range sensor. The method further comprises generating a delta representation of edges of the one or more physical objects. The delta representation of edges of the one or more physical objects comprises edges which are present in the extended-range representation of edges, but are absent in the visible-range representation of edges. The method further comprises displaying the delta representation of edges on a display which is comprised in the MR device.

According to a third aspect of the invention, a computer program is provided. The computer program comprises instructions which, when the computer program is executed by an MR device causes the MR device to carry out the method according to an embodiment of the second aspect of the invention.

According to a fourth aspect of the invention, a computer-readable data carrier is provided. The computer-readable data carrier has stored thereon the computer program according to the third aspect of the invention.

According to a fifth aspect of the invention, a data carrier signal is provided. The data carrier signal carries the computer program according to the third aspect of the invention.

Even though advantages of the invention have in some cases been described with reference to embodiments of the first aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention.

Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.

The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:.

The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

In the following, embodiments of the Mixed-Reality (MR) device <NUM> for augmenting human vision in limited-visibility environments are illustrated with reference to <FIG>.

The MR device <NUM> may, e.g., be a hand-held device, such as a mobile phone, a smartphone, a digital camera, or a tablet. Alternatively, the MR device <NUM> may be a Head-Mounted Device (HMD), similar to what is illustrated in <FIG>. As yet a further alterative, the MR device <NUM> may be a helmet-mounted device, i.e., integrated into a helmet, such as a firefighter helmet or rescue-worker helmet, similar to the C-THRU helmet by Qwake Technologies.

In the present context, limited-visibility environments are to be understood as environments, both indoor environments and outdoor environments, where human vision is hampered, e.g., by smoke, vapor (water of gases), or low-lighting conditions, resulting in low or limited visibility for a human operating in such environments, such as firefighters or rescue workers.

With reference to <FIG>, the MR device <NUM> comprises a visible-range sensor <NUM> for capturing a visible-range representation of a real-world scene <NUM> surrounding the MR device <NUM>. The real-world scene <NUM> is an indoor or outdoor location in which a user <NUM> of the MR device <NUM> is operating. The visible-range sensor <NUM> is operative to capture the visible-range representation of the real-world scene <NUM> with a range which is commensurate with human vision. In other words, the visible-range representation of the real-world scene <NUM> captured by the visible-range sensor <NUM> is representative for what the user <NUM> of the MR device <NUM> would be able to see relying on his/her human vision. This may, e.g., be achieved by employing a visible-range sensor <NUM> which is operative to capture the visible-range representation of the real-world scene <NUM> using wavelengths within the visible range. Typically, the human eye can detect wavelengths ranging from about <NUM> to about <NUM> nanometers, aka the visible spectrum. The Field-of-View (FoV) of the visible-range sensor <NUM> is illustrated as <NUM> in <FIG>.

The MR device <NUM> illustrated in <FIG> further comprises an extended-range sensor <NUM> for capturing an extended-range representation of the real-world scene <NUM>. The extended-range sensor <NUM> is operative to capture the extended-range representation of the real-world scene <NUM> with an extended range which is wider, i.e., extending to larger distance from the sensor <NUM>, than the range commensurate with human vision. This may, e.g., be achieved by employing an extended-range sensor <NUM> which is operative to capture the extended-range representation of the real-world scene <NUM> using wavelengths which are at least partly outside the visible spectrum. The wavelengths range of the extended-range sensor <NUM> may be exclusively outside the visible spectrum, i.e., below about <NUM> or above about <NUM>, but may alternatively include at least a portion of the visible range. The wavelength range of the extended-range sensor <NUM> may extend to wavelengths lower than the visible range, higher than the visible range, or both. For instance, the extended-range sensor <NUM> may be an infrared-light sensor, such as an infrared-light camera (aka infrared camera, thermal-imaging camera, or simply thermal camera), which typically operate at wavelengths beyond <NUM>. Using an extended-range sensor <NUM> operating with infrared light is advantageous in that infrared light is to lesser extent impacted by smoke, as compared to sensors operating in the visible spectrum. For the sake of simplicity, the extended-range sensor <NUM> is in <FIG> illustrated as having the same FoV <NUM> as the visible-range sensor <NUM>. In practice, different types of sensors typically have different FoVs, and an alignment of the sensors <NUM> and <NUM>, and or an alignment or transformation of the respective representations captured by the sensors <NUM> and <NUM>, may be required. For the purpose of describing embodiments of the invention described herein, it is assumed here that the FoV <NUM> illustrated in <FIG> is the FoV which is common to the visible-range sensor <NUM> and the extended-range sensor <NUM>.

Throughout this disclosure, a representation of the real-world scene <NUM> is understood to be data captured by a sensor, such as the visible-range sensor <NUM> or the extended-range sensor <NUM>, or data derived from captured sensor data, and which can be displayed to the user <NUM>, optionally after processing, for visualizing the real-world scene <NUM> similar to what is exemplified in <FIG>. For instance, a representation of the real-world scene <NUM> may be in the form of a digital image, or a sequence of images, i.e., a video. The video may either be based on a two-dimensional (2D) format or three-dimensional (3D) format, e.g., including depth information. As another example, a 3D representation of the real-world scene <NUM> may be in the form of a point cloud.

The visible-range sensor <NUM> may, e.g., be a digital camera, operating in the visible spectrum. Correspondingly, the extended-range sensor <NUM> may, e.g., be a digital camera, operating at wavelengths which are at least partly outside the visible spectrum, such as an infrared camera. In case the visible-/extended-range sensor <NUM>/<NUM> is a digital camera, the corresponding representation of the real-world scene <NUM> is captured as a video, i.e., a sequence of images. The digital camera <NUM>/<NUM> may either be a monocular camera, operative to capture a 2D representation of the real-world scene <NUM>, or a stereo-camera, operative to capture a 3D representation of the real-world scene <NUM>.

The extended-range sensor <NUM> may alternatively be a Lidar or a radar sensor. In this case, the extended-range representation is a 3D representation of the real-world scene <NUM>, e.g., in form of a point cloud. A Lidar sensor is operating at infrared wavelengths, relying on infrared laser beams which are reflected by objects. A radar sensor on the other hand is operating at wavelengths shorter than the visible spectrum, relying on radio waves which are reflected by objects. Both Lidar and radar rely on time-of-flight measurements to determine an object's distance to the sensor.

Further with reference to <FIG>, the MR device <NUM> further comprises a display <NUM>. The display <NUM> is arranged to be viewable by the user <NUM>, and can be used for displaying visual content, including visualizations of representations of the real-world scene <NUM> such as exemplified in <FIG>, to the user <NUM>. The display <NUM> may, e.g., be a see-through display, as is illustrated in <FIG>. Typically, this is the case if the MR device <NUM> is embodied as an HMD or a helmet-mounted device (either mono- or binocular, i.e., the display <NUM> is arranged to be viewed by one eye or two eyes, respectively). The see-through display <NUM> enables the user <NUM> to view the real-world scene <NUM> directly, relying on his/her human vision. Alternatively, the display <NUM> may be an opaque, non-see-trough display. Typically, this is the case if the MR device <NUM> is embodied as a hand-held device, such as a mobile phone, a smartphone, a digital camera, or a tablet.

The MR device <NUM> further comprises a processing circuitry <NUM>, which is described in further detail below with reference to <FIG>. The processing circuitry <NUM> causes the MR device <NUM> to be operative to generate a visible-range representation of edges of one or more physical objects <NUM> which are present in the real-world scene <NUM>. The visible-range representation of edges of the one or more physical objects <NUM> may be obtained by extracting edges from the visible-range representation captured by the visible-range sensor <NUM>.

The one or more physical objects <NUM> may, e.g., be furniture (such as tables, chairs, cupboards, shelfs, etc), walls, windows, doors, stairs, human or animal bodies, debris, vehicles, trees, pipes, etc..

The processing circuitry <NUM> further causes the MR device <NUM> to be operative to generate an extended-range representation of edges of the one or more physical objects <NUM>. This may be achieved by extracting edges from the extended-range representation captured by the extended-range sensor <NUM>.

In the present context, an "edge" is a boundary of a real-world object, such as object(s) <NUM>. An edge which is enclosing an object, i.e., defines the object, is commonly referred to as "contour". Throughout this disclosure, "edge" and "contour" are used synonymously. Depending on visibility and capabilities of the visible-range sensor <NUM> or the extended-range sensor <NUM>, respectively, only a part or parts of an edge may be captured by the sensor. A representation of edges is to be understood as a representation containing edges, or parts thereof, as seen by the sensor <NUM>/<NUM> which has captured the representation of the real-world scene <NUM> from which of the respective representation of edges is derived.

In <FIG>, an example visualization <NUM> of the visible-range representation of edges, which is derived from the visible-range representation of the real-world scene <NUM> captured by the visible-range sensor <NUM>, is illustrated. As can be seen in the visualization <NUM>, the visible-range sensor <NUM> captures edges, or parts thereof, of the one or more physical objects <NUM> which are not obscured by smoke <NUM>. For the sake of simplicity, it is assumed here that the smoke <NUM> has a well-defined boundary, rather than having a smooth transitional boundary in which visibility gradually diminishes. Also illustrated in <FIG> is an example visualization <NUM> of the extended-range representation of edges, which is derived from the extended-range representation of the real-world scene <NUM> captured by the extended-range sensor <NUM>. As can be seen in the visualization <NUM>, the extended-range sensor <NUM> captures edges, or parts thereof, of the one or more physical objects <NUM> to larger extent than what is visible in the visualization <NUM> of the visible-range representation of edges, and includes edges, or parts thereof, which are not visible in the visualization <NUM> of the visible-range representation of edges. This is due to the extended range of the sensor <NUM>, as compared to the visible-range sensor <NUM>, owing to the fact that the extended-range sensor <NUM> is to lesser extent affected by the smoke <NUM>.

Edges may be detected by finding the boundaries of objects within images or other types or captured representations of real-world objects, as is known in the art. For images captured by digital cameras, edges can, e.g., be detected as discontinuities in brightness. A digital image which has been image-processed to the extent that it only represents edges of captured objects is also known as a wireframe image, and has the appearance of a decluttered line drawing. As used herein, the terms "extracting edges" of "detecting edges" are to be understood to include enhancing edges in the captured visible-/extended range representations to the extent that they only contain edges of captured objects.

As an example for generating wireframe images, reference is made to <CIT> which discloses enhancing edges of objects in a thermal image by generating a gradient magnitude image comprising a plurality of pixels having associated gradient magnitude values, partitioning the gradient magnitude image into subregions, calculating gradient magnitude statistics for each subregion, and calculating mapping parameters for each of the subregions that equalize and smooth a dynamic range of the corresponding gradient magnitude statistics across the subregions. The calculated mapping parameters are then applied to pixels in the subregions to generate enhanced gradient magnitude values having equalized luminosity and contrast, resulting in a wireframe image.

Edge detection in 3D point clouds is also known in the art. As an example, "<NPL>) proposes applying supervised learning techniques to shape descriptors for local point-cloud features, and introduces a shape descriptor for capturing local surface properties near edges.

It is also possible to perform edge detection by merging 2D digital images and 3D point clouds, as is, e.g., described in "<NPL>. The authors describe a solution based on combining edge data from a point cloud of an object and its corresponding digital images. First, an edge extraction is applied on the 2D image by using the Canny edge detection algorithm. A pixel-data mapping mechanism is proposed for establishing a correspondence between 2D image pixels and 3D point-cloud pixels. By using the established correspondence map, 2D edge data can be merged into the 3D point cloud, where edge extraction is performed.

The processing circuitry <NUM> further causes the MR device <NUM> to be operative to generate a delta representation of edges of the one or more physical objects <NUM>. The delta representation of edges comprises edges which are present in the extended-range representation of edges (captured by the extended-range sensor <NUM>, as illustrated in visualization <NUM>) but are absent, i.e., not present, in the visible-range representation of edges (captured by the visible-range sensor <NUM>, as illustrated in visualization <NUM>). In other words, since the visible-range representation of edges contains edges which are visible to the user <NUM> relying on his/her human vision, and the extended-range representation of edges contains edges which can be captured by the extended-range sensor <NUM> having a range exceeding the range commensurate with human vision, the delta representation of edges contains edges, or parts thereof, which are not visible to the user <NUM> when solely relying on his/her human vision. In <FIG>, an example visualization <NUM> of the delta representation of edges, corresponding to the difference between the extended-range representation of edges (visualization <NUM>) and the visible-range representation of edges (visualization <NUM>).

The MR device <NUM> may, e.g., be operative to generate the delta representation of edges of the one or more physical objects <NUM> by subtracting the visible-range representation of edges from the extended-range representation of edges. If the visible-range representation of edges and the extended-range representation of edges are digital images, the respective images may be subtracted, as is known in the art of image processing. Depending on the alignment and/or calibration of the visible-range sensor <NUM> and the extended-range sensor <NUM>, the two images may need to be aligned prior to performing image subtraction, so as to minimize any mismatch in sensor alignment. This may, e.g., be performed based on edges contained in both images, which typically the edges which are contained in the visible-range representation. If the visible-range representation of edges and the extended-range representation of edges are point clouds, the delta representation of edges of the one or more physical objects <NUM> is generated by subtracting of the respective point clouds.

The processing circuitry <NUM> further causes the MR device <NUM> to be operative to display the delta representation of edges on the display <NUM>. If the display <NUM> is a see-through display, as is illustrated in <FIG>, the delta representation of edges is displayed as an overlay onto the real-world scene <NUM> as seen by the user <NUM> viewing the display <NUM>. Alternatively, if the display is an opaque, non-see-through display, the MR device <NUM> is further operative to display the visible-range representation of the real-world scene <NUM>, which is captured by the visible-range sensor <NUM>, on the display <NUM>. In other words, the delta representation of edges is displayed as an overlay onto the visible-range representation of the real-world scene <NUM>. This means that both the visible-range representation of the real-world scene <NUM>, which represents what the user <NUM> can see relying on his/her human vision, and the delta representation of edges, which contains the edges of the objects <NUM> which the user <NUM> cannot see, are displayed. Preferably, the visible-range representation of the real-world scene <NUM> is captured by a conventional, visible-light digital camera <NUM>, to provide a substantially undistorted view of the real-world scene <NUM> to the user <NUM>, similar to what the user <NUM> would experience while viewing the real-world scene <NUM> through a see-through display <NUM>.

An example visualization <NUM> of what the user <NUM> can see when viewing the display <NUM> is shown in <FIG>. The visualization <NUM> shows the delta representation of edges (visualization <NUM>) overlaid onto the view of the real-world scene <NUM> as seen by the user <NUM>, if the display <NUM> is a see-through display. Alternatively, if the display <NUM> is an opaque, non-see-through display, the visualization <NUM> shows the delta representation of edges (visualization <NUM>) overlaid onto the visible-range representation of the real-world scene <NUM>.

(visualization <NUM>) which represents "what the user can see". As is illustrated in <FIG>, the delta representation of edges (visualization <NUM>), which contains the edges of the objects <NUM> which the user <NUM> cannot see, may be displayed so as to emphasize and/or highlight the edges of the objects <NUM> which the user <NUM> cannot see, e.g., by displaying the edges as lines with increased thickness. As an alternative, the delta representation of edges (visualization <NUM>) may be displayed as lines of a different color.

Advantageously, by augmenting the human vision of the user <NUM> using the MR device <NUM> to view the real-world scene <NUM> only with edges of objects which are not visible to the user <NUM>, owing to limitation of human vision, e.g., because some of the objects <NUM> or parts thereof are obscured by smoke <NUM>, a more natural view of the real-world scene <NUM> as seen by the user <NUM> is achieved. In particular, this is the case if the display <NUM> is a see-through display through which the user <NUM> can view the real-world scene <NUM> relying on his/her human vision, which oftentimes is superior to digitally generated content. By only overlaying edges of the objects <NUM> which are not (human-)visible, the amount of digitally generated, or virtual, content is kept at minimum.

In the following, embodiments of the processing circuitry <NUM> which is comprised in the MR device <NUM> are described with reference to <FIG>.

The processing circuitry <NUM> may comprise one or more processors <NUM>, such as Central Processing Units (CPUs), microprocessors, application processors, application-specific processors, Graphics Processing Units (GPUs), and Digital Signal Processors (DSPs) including image processors, or a combination thereof, and a memory <NUM> comprising a computer program <NUM> comprising instructions. When executed by the processor(s) <NUM>, the instructions cause the MR device <NUM> to become operative in accordance with embodiments of the invention described herein. The memory <NUM> may, e.g., be a Random-Access Memory (RAM), a Read-Only Memory (ROM), a Flash memory, or the like. The computer program <NUM> may be downloaded to the memory <NUM> by means of a network interface circuitry <NUM> which may be comprised in the MR device <NUM> (not shown in <FIG>), as a data carrier signal carrying the computer program <NUM>. The processing circuitry <NUM> may alternatively or additionally comprise one or more Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or the like, which are operative to cause the MR device <NUM> to become operative in accordance with embodiments of the invention described herein.

The network interface circuitry <NUM> may comprise one or more of a cellular modem (e.g., GSM, UMTS, LTE, <NUM>, or higher generation, including communications solutions dedicated for first responders, emergency personnel, military, law enforcement, etc), a WLAN/Wi-Fi modem, a Bluetooth modem, an Ethernet interface, an optical interface, or the like, for exchanging data between the MR device <NUM> and other MR devices, an application server, the Internet, etc..

In the following, embodiments of the method <NUM> of augmenting human vision in limited-visibility environments are described with reference to <FIG>.

The method <NUM> is performed by an MR device <NUM> and comprises capturing <NUM> a visible-range representation of a real-world scene <NUM> surrounding the MR device <NUM> with a range commensurate with human vision. The visible-range representation of a real-world scene <NUM> is captured using a visible-range sensor <NUM> comprised in the MR device <NUM>. The visible-range representation of the real-world scene <NUM> may, e.g., be captured using wavelengths within the visible range. The visible-range sensor <NUM> may, e.g., be a digital camera.

The method <NUM> further comprises capturing <NUM> an extended-range representation of the real-world scene <NUM> with an extended range wider than the range commensurate with human vision. The extended-range representation of the real-world scene <NUM> is captured using a an extended-range sensor <NUM> comprised in the MR device <NUM>. The extended-range representation of the real-world scene <NUM> may, e.g., be captured using wavelengths at least partly outside the visible range. For example, the extended-range representation of the real-world scene <NUM> may be captured using infrared light. The extended-range sensor <NUM> may, e.g., be a digital camera. Alternatively, the extended-range sensor <NUM> may, e.g., be a Lidar or a radar sensor.

The method <NUM> further comprises generating <NUM> a visible-range representation of edges of one or more physical objects <NUM> present in the real-world scene <NUM>. The visible-range representation of edges of the one or more physical objects <NUM> is generated by extracting edges from the visible-range representation captured by the visible-range sensor <NUM>.

The method <NUM> further comprises generating <NUM> an extended-range representation of edges of the one or more physical objects <NUM>. The extended-range representation of edges of the one or more physical objects <NUM> may be generated by extracting edges from the extended-range representation captured by the extended-range sensor <NUM>.

The method <NUM> further comprises generating <NUM> a delta representation of edges of the one or more physical objects <NUM>. The delta representation of edges comprises edges which are present in the extended-range representation of edges but absent in the visible-range representation of edges. The delta representation of edges of the one or more physical objects <NUM> may, e.g., be generated <NUM> by subtracting the visible-range representation of edges from the extended-range representation of edges.

The method <NUM> further comprises displaying <NUM> the delta representation of edges on a display <NUM> comprised in the MR device <NUM>. The display <NUM> may be a see-through display, and the delta representation of edges may be displayed <NUM> as an overlay onto the real-world scene <NUM> as seen by a user <NUM> viewing the display <NUM>. Alternatively, the display <NUM> may be an opaque display (i.e., a non-see-through display), and the method <NUM> may further comprise displaying <NUM> the visible-range representation of the real-world scene <NUM> on the display <NUM>.

It will be appreciated that the method <NUM> comprise additional, alternative, or modified, steps in accordance with what is described throughout this disclosure.

An embodiment of the method <NUM> may be implemented as the computer program <NUM> comprising instructions which, when the computer program <NUM> is executed by a computing device, such as the MR device <NUM>, cause the MR device <NUM> to carry out the method <NUM> and become operative in accordance with embodiments of the invention described herein. The computer program <NUM> may be stored in a computer-readable data carrier, such as the memory <NUM>. Alternatively, the computer program <NUM> may be carried by a data carrier signal, e.g., downloaded to the memory <NUM> via the network interface circuitry <NUM>.

Claim 1:
A Mixed-Reality, MR, device (<NUM>), for augmenting human vision in limited-visibility environments, the MR device comprising:
a visible-range sensor (<NUM>) for capturing a visible-range representation of a real-world scene (<NUM>) surrounding the MR device with a range commensurate with human vision,
an extended-range sensor (<NUM>) for capturing an extended-range representation of the real-world scene (<NUM>) with an extended range wider than the range commensurate with human vision,
a display (<NUM>), and
processing circuitry (<NUM>) causing the MR device to be operative to:
generate a visible-range representation of edges of one or more physical objects (<NUM>) present in the real-world scene (<NUM>) by extracting edges from the visible-range representation captured by the visible-range sensor (<NUM>), and
generate an extended-range representation of edges of the one or more physical objects (<NUM>) by extracting edges from the extended-range representation captured by the extended-range sensor (<NUM>),
characterized by the processing circuitry (<NUM>) further causing the MR device to be operative to:
generate a delta representation of edges of the one or more physical objects (<NUM>), comprising edges present in the extended-range representation of edges but absent in the visible-range representation of edges, and
display the delta representation of edges on the display (<NUM>).