Patent Description:
A (ground-based) human operator flies a drone (or an unmanned aerial vehicle, UAV) using a remote controller (sometimes at least partly assisted by an autopilot).

The human operator has to simultaneously look towards the drone in the air, operate the hand-held remote controller, and occasionally look towards a display of the remote controller. This leads to poor situational awareness, causing potentially hazardous situations.

A legal requirement is that the human operator must maintain a visual contact (by a line of sight) to the drone in the air. This is quite challenging as the drone may not be visible due to a long distance, low ambient light, or a physical obstacle, for example.

These problems may be mitigated by another person, a so-called spotter, retaining the visual contact to the drone, even using binoculars, whereas the human operator may concentrate on operating the remote controller (but may still need to check occasionally the display of the remote controller). Naturally, such a setup requires good communication skills for the human operator and the spotter. Additionally, the manual labour is doubled, leading to higher operation costs for the drone.

<CIT> discloses a head-mounted display device including an image display, a control section configured to control the image display section, a position detecting section, a visual-field detecting section configured to detect a visual field of the user, and a wireless communication section configured to perform wireless communication with an external apparatus. The control section includes a state-information acquiring section configured to acquire, via the wireless communication section, state information including a position of an own aircraft, which is a mobile body set as an operation target object of the user, and a support-information creating section configured to create, on the basis of a relation between the position and the visual field of the user and the position of the own aircraft, a support image including support information for operation of the own aircraft and cause the image display section to display the support image as the virtual image.

<CIT> discloses an apparatus for augmented reality travel route planning. An apparatus such as a head-mounted display (HMD) may have a camera for capturing a visual scene for presentation via the HMD. A user of the apparatus may specify a pre-planned travel route for a vehicle within the visual scene via an augmented reality (AR) experience generated by the HMD. The pre-planned travel route may be overlaid on the visual scene in the AR experience so that the user can account for real-time environmental conditions determined through the AR experience. The pre-planned travel route may be transferred to the vehicle and used as autonomous travel instructions.

<CIT> discloses a modified-reality device including: a head-mounted device including one or more displays, wherein the one or more displays are configured to receive image data representing at least an image element and to display a modified-reality image including at least the image element; one or more sensors configured to provide head tracking data associated with a location and an orientation of the head-mounted device; and a processing arrangement configured to receive flight data associated with a flight of an unmanned aerial vehicle, generate the image data representing at least the image element based on the head tracking data and the flight data, and provide the image data to the one or more displays.

According to an aspect, subject matter of independent claims is provided. Dependent claims define some embodiments.

One or more examples of implementations are set forth in more detail in the accompanying drawings and the description of embodiments.

Some embodiments will now be described with reference to the accompanying drawings, in which.

Reference numbers, both in the description of the embodiments and in the claims, serve to illustrate the embodiments with reference to the drawings, without limiting it to these examples only.

The embodiments and features, if any, disclosed in the following description that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

Let us study <FIG>, which illustrates a simplified block diagram of an apparatus <NUM> for assisting a human operator (or pilot) <NUM> in flying a drone <NUM> using a remote controller <NUM>. The drone <NUM> is also known as a UAV (unmanned aerial vehicle). A UAS (unmanned aircraft system) may be defined as including the drone (or UAV) <NUM>, the (ground-based) remote controller <NUM>, and a wireless communications system <NUM> between the remote controller <NUM> and the drone <NUM>.

Simultaneously, <FIG>, which is a flow chart illustrating embodiments of a method for assisting the human operator <NUM> in flying the drone <NUM> using the remote controller <NUM>, is referred to.

The method starts in <NUM> and ends in <NUM>. Note that the method may run as long as required (after the start-up of the apparatus <NUM> until switching off) by looping back to an operation <NUM>.

The operations are not strictly in chronological order in <FIG>, and some of the operations may be performed simultaneously or in an order differing from the given ones. Other functions may also be executed between the operations or within the operations and other data exchanged between the operations. Some of the operations or part of the operations may also be left out or replaced by a corresponding operation or part of the operation. It should be noted that no special order of operations is required, except where necessary due to the logical requirements for the processing order.

The apparatus <NUM> comprises an internal data communication interface <NUM> configured to receive <NUM> data related to the flying from the remote controller <NUM>. The data related to the flying may include telemetry data of the drone <NUM>. The data related to the flying may include, but is not limited to: sensor readings such as gyroscope and magnetometer, angular rate, velocity, fusion data such as altitude and global position, aircraft information such as battery, gimbal, and flight status, etc. Note that depending on the drone environment, some data may also be received by the apparatus <NUM> directly from the drone <NUM>.

The internal data communication interface <NUM> may be implemented using a wireless radio transceiver configured to communicate with a wireless transceiver of the remote controller <NUM>. The technologies for the internal data communication interface <NUM> include, but are not limited to one or more of the following: a wireless local area network (WLAN) implemented using an IEEE <NUM>. 11ac standard or a Wi-Fi protocol suite, a short-range radio network such as Bluetooth or Bluetooth LE (Low Energy), a cellular radio network employing a subscriber identity module (SIM) or an eSIM (embedded SIM), or another standard or proprietary wireless connectivity means. Note that in some use cases, the internal data communication interface <NUM> may additionally or alternatively utilize a standard or proprietary wired connection such as an applicable bus, for example. An embodiment utilizes a wired connection according to the USB (Universal Serial Bus) standard.

The apparatus <NUM> also comprises an augmented reality (AR) display <NUM> configured to display <NUM> the data related to the flying to the human operator <NUM>. Note that the drawings from <FIG> show specific embodiments, but besides these, also various notifications and statuses related to the flying may be shown on the augmented reality display <NUM>.

In the drawings, the augmented reality display <NUM> is implemented as a head-mounted display attached with a headband (or being a helmet-mounted) and worn as a visor in front of the eyes by the human operator <NUM>. In the drawings, the augmented reality display <NUM> is implemented as a see through display on which holographic images are displayed. In an alternative embodiment, the augmented reality display <NUM> may employ cameras to intercept the real world view and display an augmented view of the real world as a projection.

In an embodiment, the apparatus <NUM> is implemented using Microsoft® HoloLens® <NUM> (or a later version) mixed reality smartglasses employing see-through holographic lenses as the augmented reality display <NUM>, offering a complete development environment. The head-mounted apparatus <NUM> then includes the necessary processors (including a system on a chip, a custom-made holographic processing unit, and a coprocessor) <NUM>, memories <NUM> and software <NUM>, a depth camera, a video camera, projection lenses, an inertial measurement unit (including an accelerometer, a gyroscope, and a magnetometer), a wireless connectivity unit <NUM>, <NUM>, and a rechargeable battery. Note that some of these parts are not illustrated in <FIG>. Such a ready-made environment offers an augmented reality engine <NUM> configured to provide the basic operations related to fusing the real world and the augmented reality together and tracking head and eye movements of the human operator <NUM>, for example.

However, also other applicable implementations of the augmented reality display <NUM> may be used, including, but not limited to: eyeglasses, a head-up display, contact lenses with an augmented reality imaging, etc. For the purposes of the present embodiments, the augmented reality display <NUM> is configured to provide an interactive real-time experience of a real-world flying environment <NUM> and the drone <NUM> enhanced by computer-generated perceptual information. The data related to the flying is superimposed (or overlaid) in addition to the natural environment <NUM> and the drone <NUM>.

The apparatus <NUM> also comprises one or more memories <NUM> including computer program code <NUM>, and one or more processors <NUM> configured to execute the computer program code <NUM> to cause the apparatus <NUM> to perform required data processing. The data processing performed by the apparatus <NUM> may be construed as a method or an algorithm <NUM>.

The term 'processor' <NUM> refers to a device that is capable of processing data. In an embodiment, the processor <NUM> is implemented as a microprocessor implementing functions of a central processing unit (CPU) on an integrated circuit. The CPU is a logic machine executing the computer program code <NUM>. The CPU may comprise a set of registers, an arithmetic logic unit (ALU), and a control unit (CU). The control unit is controlled by a sequence of the computer program code <NUM> transferred to the CPU from the (working) memory <NUM>. The control unit may contain a number of microinstructions for basic operations. The implementation of the microinstructions may vary, depending on the CPU design. The one or more processors <NUM> may be implemented as cores of a single processors and/or as separate processors.

The term 'memory' <NUM> refers to a device that is capable of storing data run-time (= working memory) or permanently (= non-volatile memory). The working memory and the non-volatile memory may be implemented by a random-access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), a flash memory, a solid state disk (SSD), PROM (programmable read-only memory), a suitable semiconductor, or any other means of implementing an electrical computer memory.

The computer program code <NUM> is implemented by software. In an embodiment, the software may be written by a suitable programming language, and the resulting executable code may be stored in the memory <NUM> and executed by the one or more processors <NUM>.

The computer program code <NUM> implements the method/algorithm <NUM>. The computer program code <NUM> may be coded as a computer program (or software) using a programming language, which may be a high-level programming language, such as C, C++, or Rust, for example. The computer program code <NUM> may be in source code form, object code form, executable file, or in some intermediate form, but for use in the one or more processors <NUM> it is in an executable form as an application <NUM>. There are many ways to structure the computer program code <NUM>: the operations may be divided into modules, subroutines, methods, classes, objects, applets, macros, etc., depending on the software design methodology and the programming language used. In modern programming environments, there are software libraries, i.e. compilations of ready-made functions, which may be utilized by the computer program code <NUM> for performing a wide variety of standard operations. In addition, an operating system (such as a general-purpose operating system) may provide the computer program code <NUM> with system services.

An embodiment provides a computer-readable medium <NUM> storing the computer program code <NUM>, which, when loaded into the one or more processors <NUM> and executed by one or more processors <NUM>, causes the one or more processors <NUM> to perform the method/algorithm <NUM> described in <FIG>. The computer-readable medium <NUM> may comprise at least the following: any entity or device capable of carrying the computer program code <NUM> to the one or more processors <NUM>, a record medium, a computer memory, a read-only memory, an electrical carrier signal, a telecommunications signal, and a software distribution medium. In some jurisdictions, depending on the legislation and the patent practice, the computer-readable medium <NUM> may not be the telecommunications signal. In an embodiment, the computer-readable medium <NUM> may be a computer-readable storage medium. In an embodiment, the computer-readable medium <NUM> may be a non-transitory computer-readable storage medium.

As shown in <FIG> and <FIG>, the computer-readable medium <NUM> may carry the computer program code <NUM> as the executable application <NUM> for the apparatus <NUM>, and as an executable application <NUM> for the remote controller <NUM> to transmit the data related to the flying to the apparatus <NUM>. In a typical drone environment, such as DJI®, a software development kit may be used for the application <NUM> to interface with the remote controller <NUM>.

<FIG> illustrates the apparatus <NUM> as an integrated unit comprising the augmented reality display <NUM>, the one or more memories <NUM> including the computer program code <NUM>, and the one or more processors <NUM>.

However, as illustrated in <FIG>, the apparatus <NUM> may also be implemented as a distributed apparatus <NUM> so that the human operator <NUM> is provided with the augmented reality display <NUM>, but with a separate processing part <NUM>, which is communicatively coupled with the augmented reality display <NUM> and the remote controller <NUM>, and which comprises the one or more memories <NUM> including the computer program code <NUM>, and the one or more processors <NUM>. This may be implemented so that processing part <NUM> is a user apparatus such as a smartphone, tablet computer or a portable computer carried by the human operator <NUM>, and the communication coupling may be wired or wireless. Another implementation is such that the processing part <NUM> is a networked computer server, which interoperates with the augmented reality display <NUM> according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or another applicable distributed computing architecture.

<FIG> illustrate embodiments of views offered by the augmented reality display <NUM> of the apparatus <NUM>. Note that all drawings from <FIG> illustrate each use case as a combination of two different visual angles.

Let us examine <FIG> in more detail. As shown, a first visual angle illustrates the flying: the human operator <NUM> operates the remote controller <NUM> and observes (or looks towards) <NUM> the drone <NUM> in the air <NUM> through the apparatus <NUM>, or, expressed more precise, through the augmented reality display <NUM> of the apparatus <NUM>. As, shown, a second visual angle illustrates elements <NUM>, <NUM> shown on the augmented reality display <NUM>.

This convention is used in all drawings from <FIG>: a dotted arrow line <NUM> shows the direction the human operator <NUM> is looking towards, normally towards the drone <NUM> in the air, but in some use cases, the human operator <NUM> is looking towards another direction such as towards the ground, the direction of gaze marked with dotted arrow lines referred to by reference signs <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. However, in the previously mentioned alternative embodiment using the cameras to intercept the real world view and display the augmented view of the real world as the projection, the direction of the gaze towards the augmented reality display <NUM> may differ from the capture direction of the cameras. For example, to ease the flying position, the human operator <NUM> need not tilt the head to gaze into the sky, but the cameras are tilted upwards.

Note that in all described embodiments, the human operator <NUM> is standing on the ground <NUM>, and the drone <NUM> is flying in the air <NUM>. However, the embodiments are also applicable to other kind of environments, such as flying the drone <NUM> in an underground cave, inside a man-made structure (such as a building or a tunnel), or even in such use cases where the drone <NUM> is flying below the human operator <NUM>, i. e, the human operator <NUM>, while looking <NUM> towards the drone <NUM>, is looking down and not up. In such a use case, the human operator <NUM> may be standing on a high platform (such as a skyscraper or a mountain), and the drone <NUM> is flying below (such as above the streets or in a valley). The embodiments may also be applied to flying the drone <NUM> submersed, i.e., the drone <NUM> is then an unmanned underwater vehicle (UUV), and the human operator <NUM> may operate the drone <NUM> from the land or from a vessel, for example, while the drone is underwater in a river, lake, sea, water-filled mine or tunnel, etc..

In a way, all drawings from <FIG> are hybrids illustrating an augmented reality on top of the real world. The real world is illustrated from an external view (like a view of another person observing the use case from outside of the real world), whereas the augmented reality display <NUM> is illustrated from a first person view of the human operator <NUM>.

Let us now return to <FIG>. In an embodiment, the apparatus <NUM> is caused to superimpose <NUM>, on the augmented reality display <NUM>, a target symbol <NUM> indicating a position of the drone <NUM> (in the air <NUM> for the UAV) while the human operator <NUM> is looking <NUM> towards the drone <NUM> (in the air <NUM> for the UAV). In an embodiment, the apparatus is also caused to superimpose <NUM>, on the augmented reality display <NUM>, an orientation symbol <NUM> indicating an orientation of the drone <NUM> (in the air <NUM> for the UAV) while the human operator <NUM> is looking <NUM> towards the drone <NUM> (in the air <NUM> for the UAV).

The use of the augmented reality display <NUM> enables the human operator <NUM> to look <NUM> towards the drone <NUM> in the sky <NUM> during the flying. This improves the situational awareness of the human operator <NUM> regarding the flying, without needing the spotter. The human operator maintains a visual contact (by a line of sight) to the drone <NUM> in the air <NUM>, but is also simultaneously shown aviation data in actual correct world positions as will be explained.

The target symbol <NUM> indicates the position of the drone <NUM> in the air <NUM>, which makes it easier for the human operator <NUM> to track the drone <NUM> during the flying. In an embodiment, the target symbol <NUM> is a reticle as illustrated. The reticle <NUM> is commonly used in a telescopic sight of a firearm. The reticle <NUM> may include a combination of a circle <NUM> and a partial crosshair <NUM> as shown in <FIG>, but also other patterns may be used such as dots, posts, chevrons, etc..

The orientation symbol <NUM> indicates the orientation of the drone <NUM> in the air <NUM>, which makes it easier for the human operator <NUM> to understand an effect of the steering commands given with the remote controller <NUM> to the drone <NUM> during the flying. In an embodiment, the orientation symbol <NUM> is an arrow as illustrated. As shown in <FIG>, the arrow <NUM> may be augmented by an arc <NUM>, which illustrates a part of a <NUM> degrees circle around the human operator <NUM>. The arrow <NUM> may point a heading of the drone <NUM> as will later be explained.

In the augmented reality display <NUM>, the target symbol <NUM> and the orientation symbol <NUM> from the digital world blend into the human operator's <NUM> perception of the real world, through the integration of immersive sensations, which are perceived as natural parts of the flying environment <NUM>.

Let us next study <FIG>, <FIG>, <FIG>, which illustrate embodiments of an orientation of the drone <NUM>.

In an embodiment, the orientation symbol <NUM> is configured to point out a predetermined direction fixed in relation to the orientation of the drone <NUM> in the air <NUM>. As the human operator <NUM> is aware of the predetermined direction, it is easy to for the human operator <NUM> to understand the way the steering commands given with the remote controller <NUM> influence the flying. As shown in <FIG>, the remote controller <NUM> may include two joysticks <NUM>, <NUM>, for example, to give the steering commands. Naturally, also other kinds of steering arrangements are compatible with the described embodiments. However, the remote controller <NUM> may control the drone <NUM> in various degrees of freedom: a roll, which tilts the drone <NUM> left or right, a pitch, which tilts the drone <NUM> forward or backward, and a yaw, which rotates the drone <NUM> clockwise or counterclockwise. Furthermore, an altitude control controls the drone <NUM> to fly higher or lower. Note that some user interface elements of the remote controller <NUM> may be programmed to interact with the apparatus <NUM> so that user interface operations of the apparatus <NUM>, besides being performed in the augmented reality environment, may also be performed with (physical) user interface elements of the remote controller <NUM>.

In an embodiment illustrated in <FIG>, the predetermined direction is fixed in relation to a heading <NUM> of the drone <NUM>. In navigation, the heading <NUM> of the drone <NUM> is a compass direction in which a nose of the drone <NUM> is pointed. Note that the drone <NUM>, being a quadcopter (= a helicopter with four rotors), for example, may not have a "natural" nose, in which case one direction of the drone <NUM> is just defined as the nose.

<FIG> illustrates the various coordinate systems <NUM>, <NUM>, <NUM> that need to be related to each other in order to enable the embodiments. A world coordinate system <NUM> defines a three-dimensional world model visualization, which is mapped to a coordinate system <NUM> of the apparatus <NUM> and to a coordinate system <NUM> of the drone <NUM>. The apparatus <NUM> then shows the augmented reality using its own coordinate system <NUM> but also illustrating the position of the drone <NUM> and the position of the human operator <NUM> in the world coordinate system <NUM>.

In an embodiment illustrated in <FIG> and <FIG>, the apparatus <NUM> is caused to perform:.

In this way, the augmented reality coordinate system <NUM> that constantly tracks any movement of the head of the human operator <NUM>, is now firmly based in the world coordinates <NUM>, and also follows the actual compass directions <NUM>. The coupling of world latitude and longitude (x and z of the world coordinate system <NUM>) and the compass heading information <NUM> into the augmented reality presentation is thus achieved.

In a more specific embodiment, the apparatus <NUM> is caused to perform:.

At first, the augmented reality system is shown the position of the drone <NUM> in the world coordinate system <NUM>, and the position of the drone <NUM> in relation to the augmented reality coordinate system <NUM>. By indicating that the drone <NUM> centre is situated in this exact spot within the augmented reality field of view <NUM>, with augmented reality indicators, that spot is now known both in the real world coordinate system <NUM> and in the augmented reality system coordinates <NUM>. With this combination, a fixed common position with the world latitude and longitude information is obtained. This latitude and longitude comes from the <NUM> drone, as it knows at this moment its exact coordinates (provided by GPS or another global navigation satellite system, or by another positioning technology such as a cellular radio -based positioning). An augmented reality pointer stick or another type of the calibration position symbol may indicate a position in the augmented reality display <NUM> for the human operator <NUM>. When showing the drone <NUM> location, this stick, which moves at a fixed distance in front of the human operator <NUM> and points down, is guided to be on top of the centre of the drone <NUM>. It is held steady to confirm the position, which then locks the coordinate systems <NUM>, <NUM> together. Alternatively, this may also be done using a machine vision, just seeing the drone <NUM> and deciphering its place in the augmented reality coordinate system <NUM>, then locking the drone <NUM> latitude, longitude and even heading into that shape. Showing the position of the drone <NUM> may be done in many ways, but it needs to be done with confidence to lock the world and augmented reality coordinate systems <NUM>, <NUM> reliably together.

Secondly, as the drone <NUM> knows where its nose is pointed at, i.e., the drone <NUM> tells its compass heading in degrees, this may be used to finalize the coupling of the coordinate systems <NUM>, <NUM>. The augmented reality system is used to align a displayed line or another type of the calibration orientation symbol with a tail-nose -line of the drone <NUM>, and when this is achieved, this compass orientation of the displayed line in the world coordinate system <NUM> is now known. Thus, the world compass heading of any direction, for example North may be calculated from it.

As an optional step, at the time when the world position (latitude, longitude) is obtained from the drone <NUM>, an exact altitude (y in the world coordinate system <NUM>) may also be queried from a map system based on the exact world coordinates <NUM>, or from the drone <NUM> itself, possibly via the remote controller <NUM>. So, we may also calibrate an altitude for this point in space (with a drone-specific offset of the top surface of the drone <NUM> from the ground <NUM>, if an exact precision is needed), and so use the map data to accurately determine any other world point terrain altitude from here on. To summarize, the latitude, the longitude, possibly the altitude, and the compass heading may be needed for the world locking to be achieved.

After this coupling, everything else in the whole system is built around the knowledge of where the drone <NUM> actually is in the world coordinates <NUM> and what is around it exactly there in the world. Note that the described embodiments related to the coupling may operate as stand-alone embodiments, irrespective of all other embodiments, also those described in relation to the independent claims and other dependent claims.

The data related to the flying is mapped to the world coordinates <NUM>, and is consequently displayed <NUM>, <NUM>, <NUM> so that its visualization takes advantage of knowing its three-dimensional position expressed in the world coordinate system <NUM>, which is locked to the augmented reality coordinate system <NUM>.

In an embodiment, illustrated in <FIG>, the situational awareness may further be enhanced with numerical information. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a cruising altitude <NUM> of the drone <NUM> using a numerical value and a scale <NUM> visually coupled with the target symbol <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. As shown in <FIG>, the scale <NUM> may include horizontal lines each indicating a specific altitude. The apparatus is also caused to superimpose, on the augmented reality display <NUM>, a heading <NUM> of the drone <NUM> in degrees <NUM> visually coupled with the orientation symbol <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. This may be useful for an expert human operator <NUM>.

In an embodiment illustrated in <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, an indirect line of sight guideline <NUM> extending horizontally to the geographic location of the drone <NUM> on the ground <NUM>, from which the indirect line of sight guideline <NUM> continues to extend vertically to the target symbol <NUM> in a cruising altitude of the drone <NUM> in the air <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. This may further enhance the situational awareness, as the human operator <NUM> may first observe the horizontal guideline <NUM> to see the geographic location of the drone <NUM> on the earth surface <NUM>, and then observe the vertical guideline <NUM> to grasp where the drone <NUM> is in the air <NUM>.

In an embodiment illustrated in <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a track symbol <NUM> indicating a track <NUM> and a speed of the drone <NUM> in the air <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. In navigation, the track <NUM> is a route that the drone <NUM> actually travels. A difference between the heading <NUM> and the track <NUM> is caused by a motion of the air <NUM> (such as by an air current). By showing the track <NUM> and the speed, the human operator <NUM> foresees an effect of the current control, which may then be adjusted as necessary.

Let us next study <FIG>, and <FIG>, which illustrate embodiments of visualizing an obstacle in relation to the drone <NUM>.

In an embodiment illustrated in <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, an obstruction indicator symbol <NUM> configured to depict a distance <NUM> of the drone <NUM> to a real object <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. The distance <NUM> may be the shortest distance between the drone <NUM> and the real object <NUM>. As shown in <FIG>, the obstruction indicator symbol <NUM> may mark the distance using an arrow, possibly augmented by a numerical value indicating the distance <NUM>. The real object <NUM> may be a man-made object such as a building, a bridge, etc., or a natural object such as a hill, a forest, etc..

<FIG> illustrates an additional embodiment, wherein the obstruction indicator symbol <NUM> comprises a visual indicator <NUM> superimposed at least partly over the real object <NUM>. As shown in <FIG>, the visual indicator <NUM> may be a shading or a similar visual effect overlaid on the real object <NUM>. In this way, the human operator <NUM> immediately recognizes a collision danger as the drone <NUM> approaches the object <NUM>.

<FIG> illustrates a further embodiment, applicable to either the embodiment of <FIG> or the embodiment of <FIG>. The obstruction indicator symbol <NUM> comprises elements depicting a shortest horizontal <NUM> and vertical <NUM> distance from the drone <NUM> to the real object <NUM>. In this way, an effect of both vertical and horizontal movement of the drone <NUM> may be recognized in order to avoid a collision with the real object <NUM>.

Let us next study <FIG>, which illustrates embodiments of visualizing a waypoint in relation to the drone <NUM>. The embodiments related to <FIG> are not part of the claimed invention. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a map <NUM> showing a geographic location <NUM> of the human operator <NUM>, a geographic location <NUM> of the drone <NUM>, and a waypoint <NUM>. In this way, the human operator <NUM> intuitively has a better understanding of the surroundings related to the flying. As shown in <FIG>, the map <NUM> and the drone <NUM> may be simultaneously within the field of vision of the human operator <NUM>, and the gaze may alternate as being directed <NUM> to the drone or being directed <NUM> to the map <NUM>. The apparatus <NUM> is also caused to superimpose, on the augmented reality display <NUM> a vertical waypoint symbol <NUM> starting from a geographic location of the waypoint <NUM> on the ground <NUM> and extending towards a predetermined altitude of the waypoint <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. A narrow part of the waypoint symbol <NUM> may accurately pinpoint the geographic location on the earth <NUM>, whereas a broader part of the waypoint symbol <NUM> may indicate the set altitude of the waypoint in the air <NUM>. In this way, the waypoint symbol <NUM> is shown in the correct location of the real world.

Next, <FIG> illustrate embodiments of visualizing data captured by the drone <NUM>. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, one or more visual elements <NUM> captured in real-time using one or more sensors <NUM> onboard the drone <NUM> in the vicinity of the target symbol <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>, and position, on the augmented reality display <NUM>, the one or more visual elements <NUM> so that a line of sight remains unobstructed while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. The visual element(s) <NUM> may be placed on either side of the target symbol <NUM> as shown, but also anywhere around the target symbol <NUM>. In any case, the human operator <NUM> may quickly glance <NUM> the visual element(s) <NUM>, but mainly look <NUM> towards the drone <NUM> and simultaneously steer the drone <NUM>. In the illustrated embodiment, an image sensor <NUM> captures images or a video feed as the data, which is then superimposed as the visual element <NUM> on the augmented reality display <NUM>. In this way, the human operator <NUM> may steer the drone <NUM> so that the image sensor <NUM> shoots the desired view. Note that the image sensor may operate as a (normal) visible light camera such as a photographic camera or a video camera. Besides this, the image sensor may operate as a thermal (or infrared) camera, a multispectral camera, a hyperspectral camera, or a corona discharge camera, for example. The one or more sensors <NUM> onboard the drone <NUM> may comprise, but are not limited to one or more of the following technologies: a lidar (light detection and ranging, or laser imaging, detection, and ranging, or <NUM>-D laser scanning) sensor, a sonar (sound navigation and ranging) sensor, a radar (radio detection and ranging) sensor, a chemical sensor, a biological sensor, a radiation sensor, a particle sensor, a magnetic sensor, a network signal strength sensor, etc. The drone <NUM> may carry any combination of these sensors <NUM> as the payload, whose data is then visualized as explained with the dynamically positioned one or more visual elements <NUM>.

<FIG> illustrate embodiments of visualizing maps related to the flying of the drone <NUM>. The human operator <NUM> may choose the layout of the map <NUM>, <NUM>, or the apparatus <NUM> may automatically decide which layout to use depending on the flying situation. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a map <NUM> in a vertical layout showing a geographic location <NUM> of the human operator <NUM> and a geographic location <NUM> of the drone <NUM> in the vicinity of the target symbol <NUM> on the augmented reality display <NUM> while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. Alternatively, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a map <NUM> in a horizontal layout showing a geographic location <NUM> of the human operator <NUM> and a geographic location <NUM> of the drone <NUM> while the human operator <NUM> is looking <NUM> towards the ground <NUM>. By using the vertical layout map <NUM>, the situational awareness may be retained at all times as the human operator gazes <NUM> towards the drone <NUM> and sees the map <NUM> at the side. By using the horizontal layout map <NUM>, the human operator <NUM> needs to look <NUM> towards the ground <NUM>, but as shown the map <NUM> may be shown larger and more intuitively as the map <NUM> surface is parallel with the earth surface <NUM>.

<FIG> also illustrates that the used maps <NUM> may be three-dimensional topographic maps illustrating also the altitude data as depicted by the three-dimensional buildings <NUM>.

<FIG> illustrate embodiments of visualizing menu structures of the apparatus <NUM>. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, a menu structure <NUM> around the human operator <NUM> while the human operator <NUM> is looking <NUM> towards the ground <NUM>. The apparatus <NUM> is caused to detect a gesture <NUM> from the human operator <NUM> as a command related to the menu structure <NUM>, and control, on the augmented reality display <NUM>, the display <NUM> of the data related to the flying based on the command. In this way, the human operator <NUM> may quickly manipulate the apparatus <NUM>. As shown, in <FIG>, the basic display of the target symbol <NUM> and the orientation symbol <NUM> are shown, whereas in <FIG>, the human operator <NUM> has chosen from the menu structure <NUM> to display the cruising altitude of the drone <NUM> using the numerical value and the scale <NUM> visually coupled with the target symbol <NUM> as explained earlier with reference to <FIG>.

<FIG>, and <FIG> illustrate embodiments of visualizing external data related to a physical environment of the drone <NUM>.

As shown in <FIG> and <FIG>, the apparatus <NUM> comprises an external data communication interface <NUM> configured to receive external data <NUM> related to a physical environment of the drone <NUM>. Note that the external data communication interface <NUM> may in an embodiment be implemented using the internal data communication interface <NUM>. The apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, one or more visualizations <NUM> of the external data <NUM>. In this way, the apparatus <NUM> may increase the situational awareness of the human operator <NUM> by incorporating external data sources to the single user interface implemented by the augmented reality display <NUM>. As explained earlier, the external data <NUM> is mapped to the world coordinates <NUM>, and is consequently displayed so that its visualization takes advantage of knowing its three-dimensional position expressed in the world coordinate system <NUM>, which is locked to the augmented reality coordinate system <NUM>. Besides obtaining external data from various sources, the external data communication interface <NUM> may also be used to communicate data related to the flying to outside receivers <NUM>, the data being transmitted including, but not being limited to: the position of the drone <NUM>, speech from the human operator <NUM>, one or more video feeds from the drone <NUM>, etc..

As shown in <FIG>, the external data <NUM> may comprise weather data, and the one or more visualizations <NUM> depict the weather data. In an embodiment, the weather data includes information on a speed and a direction of the wind. The direction may be indicated by arrows, and the speed may be indicated by a scale of the arrows as shown or alternatively by a numerical value. Additionally, or alternatively, the weather data may include one or more of the following: turbulences (predicted or known), humidity, cloud visualizations, rain warnings, hail warnings, snow warnings, storm warnings, warnings about lightning, lighting conditions (time of day, position of sun and/or moon), fog, air temperature and pressure, visibility, dew point (important for aviation pilots), "feels like" temperature. And all this may also be tied to time, i.e., the weather predictions may be visualized, for example incoming cloud fronts and wind changes.

As shown in <FIG>, the external data may comprise <NUM> air traffic control data including classifications of airspaces, and the one or more visualizations <NUM>, <NUM> depict the classification of the airspace matching the position of the drone <NUM> in the air <NUM>. As shown, a free airspace <NUM> may be marked with "I", and a restricted airspace <NUM> may be marked with "II" and a shaded rectangle as shown or with another three-dimensional shape (such as a polygon mesh) or even with a two-dimensional shape (such as a polygon). In general, the classifications of airspace may include, but are not limited to: drone no fly zones (areas, volumes), reservations and notifications of airspaces for drone and/or other aviation operations, airfield control zones, airspace control zones, power lines and other obstacles, country border zones, all of the aforementioned in different altitudes, warning/danger/restricted zones, UAV reserved areas, UAS reserved areas, model airplane reserved areas. An aviation map may be visualized using a three-dimensional polygon mesh with various walls, roofs, flight levels, etc, all of which are in their correct places as seen in the augmented reality display <NUM>.

As shown in <FIG>, the external data may comprise <NUM> air traffic control data including positions of aircraft <NUM> in the air <NUM>, and the one or more visualizations <NUM>, <NUM> depict the positions of the aircraft <NUM> in the air <NUM>. In an embodiment, the one or more visualizations <NUM>, <NUM> are shown for the aircraft <NUM> flying within a predetermined distance (such as within a radius of <NUM>, <NUM> or <NUM> kilometres, for example) from the position of the drone <NUM> in the air <NUM>. The visualizations may be implemented with arrows <NUM>, <NUM> indicating the location of the aircraft <NUM>, and additionally or alternatively a simulation of the aircraft <NUM> may be shown.

<FIG>, <FIG>, <FIG>, <FIG> illustrate embodiments of visualizing a line of sight to the drone <NUM> during different visibilities.

In an embodiment of <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, the data related to the flying while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM> with a visual line of sight to the drone <NUM> during a good visibility. This is the ideal flying situation.

In an embodiment of <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, the data related to the flying while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM> with an augmented line of sight to the drone <NUM> during an impaired visibility. The augmented line of sight may be achieved by guiding the human operator <NUM> to look at the right direction with the target symbol <NUM>. Optionally, a simulated drone <NUM> may be shown in the correct position. The impaired visibility may be caused by a low-light condition, cloud, fog, smog, rain, snowfall, or some other physical phenomenon.

In an embodiment of <FIG>, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, the data related to the flying while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM> with an augmented and simulated line of sight to the drone <NUM> during an obstructed visibility. The obstructed visibility may be caused by an obstacle <NUM>, i.e., the drone <NUM> is behind the obstacle <NUM>. The obstacle <NUM> may be the real object <NUM> of <FIG>, i. e, the obstacle <NUM> may be a man-made object such as a building, a bridge, etc., or a natural object such as a hill, a forest, etc. The augmentation is achieved by guiding the human operator <NUM> to look at the right direction with the target symbol <NUM>, and the simulation by showing a simulated drone <NUM> in the correct position.

In an embodiment, the apparatus <NUM> is caused to superimpose, on the augmented reality display <NUM>, the data related to the flying while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM> with an augmented line of sight to the drone <NUM> during a long-distance visibility. This is not shown in any drawing, but basically the drone <NUM> is then high up in the sky, or near the horizon, for example, and the human operator <NUM> is guided to look at the right direction with the target symbol <NUM>, whereby the human operator <NUM> may only see the drone <NUM> as a tiny object in the distance.

In an embodiment illustrated in <FIG>, the apparatus <NUM> is caused to adjust <NUM>, on the augmented reality display <NUM>, the display <NUM> of the data related to the flying so that a line of sight <NUM> remains unobstructed while the human operator <NUM> is looking <NUM> towards the drone <NUM> in the air <NUM>. In <FIG>, the human operator <NUM> keeps on looking <NUM> towards the drone <NUM> with a free line of sight <NUM>. However, as the drone is descending <NUM>, the map <NUM> would eventually obstruct the line of sight <NUM>. As shown in <FIG>, the drone <NUM> is now flying relatively low, but the line of sight <NUM> remains free due to the moving <NUM> of the map <NUM> to the left.

Let us finally study <FIG>, which illustrate embodiments of a system comprising two apparatuses <NUM>, <NUM>.

A first apparatus <NUM> is used for assisting a first human operator <NUM> in flying the drone <NUM> in the air using <NUM> the remote controller <NUM>.

A first geographic location <NUM> of the first human operator <NUM> in relation to the position of the drone <NUM> in the air <NUM> is used to adjust a first viewpoint for rendering the data related to the flying including a first target symbol <NUM> and a first orientation symbol <NUM> to be superimposed on a first augmented reality display <NUM> of the first apparatus <NUM>.

As illustrated in <FIG>, a second apparatus <NUM> is used for informing a second human operator <NUM> in relation to flying the drone <NUM> in the air <NUM>.

A second geographic location <NUM> of the second human operator <NUM> in relation to the position of the drone <NUM> in the air <NUM> is used to adjust a second viewpoint for rendering the data related to the flying including a second target symbol <NUM> and a second orientation symbol <NUM> to be superimposed on a second augmented reality display <NUM> of the second apparatus <NUM>.

In this way, the second human operator <NUM> may at least observe <NUM> the flying of the drone <NUM> in the air <NUM>. This may be useful just for fun, for educational purposes, for passing a test for a flying license, for surveillance, for tracking a missing person, or even for assisting the first human operator <NUM>, for example. One or both operators <NUM>, <NUM> may also be provided with the one or more visual elements based on the data captured in real-time using the one or more sensors <NUM> onboard the drone <NUM> as explained earlier.

In an embodiment illustrated in <FIG>, the second apparatus <NUM> is used for assisting the second human operator <NUM> in relation to controlling <NUM> one or more sensors <NUM> onboard the drone <NUM>, while the first human operator <NUM> controls the flying direction <NUM> and speed of the drone <NUM>.

For example, if the sensor <NUM> is an image sensor as described earlier, the second geographic location <NUM> of the second human operator <NUM> is used to adjust the second viewpoint for rendering the data related to the flying including also the one or more video feeds captured in real-time from the one or more video cameras <NUM> onboard the drone <NUM> to be superimposed on the second augmented reality display <NUM> of the second apparatus <NUM>. As shown in <FIG>, the one or more video feeds <NUM> are superimposed on the second augmented reality display <NUM>.

Note that the use case of <FIG> may also be such that both operators <NUM>, <NUM> may be shown the same information on the augmented reality displays <NUM>, <NUM>, and as they both have remote controllers <NUM>, <NUM>, the responsibility for the flying may be seamlessly transferred on the fly between the operators <NUM>, <NUM>. This may be especially useful during a training session or during a long mission. It is also envisaged, that in an especially hazardous or restricted airspace, an authorized pilot <NUM> may pilot the drone <NUM> safely across, and thereafter the (original) operator <NUM> regains the control of the drone <NUM>.

Note that the scenarios of <FIG> are not limited to the second human operator <NUM> being physically present near the drone <NUM> and the first human operator <NUM>. As was explained earlier, the external data communication interface <NUM> may communicate data related to the flying to the outside receiver <NUM>. The outside receiver <NUM> may be a networked computer server, which interoperates with the first apparatus <NUM> and the second apparatus <NUM> according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or another applicable distributed computing architecture. In this way, the second human operator <NUM> may be far away, even in a different city, country, or continent, and still able to observe or even assist as described. Naturally, data transmission delays need to be minimized and taken into account, especially if the remote second human operator <NUM> is controlling <NUM> the one or more sensors <NUM>, for example.

Claim 1:
An apparatus (<NUM>) for assisting a human operator (<NUM>) in flying a drone (<NUM>) using a remote controller (<NUM>), comprising:
an internal data communication interface (<NUM>) configured to receive data related to the flying from the remote controller (<NUM>);
an augmented reality display (<NUM>) configured to display the data related to the flying to the human operator (<NUM>);
one or more memories (<NUM>) including computer program code (<NUM>); and
one or more processors (<NUM>) configured to execute the computer program code (<NUM>) to cause the apparatus (<NUM>) to perform at least the following:
superimposing, on the augmented reality display (<NUM>), a target symbol (<NUM>) indicating a position of the drone (<NUM>) while the human operator (<NUM>) is looking (<NUM>) towards the drone (<NUM>);
superimposing, on the augmented reality display (<NUM>), an orientation symbol (<NUM>) indicating an orientation of the drone (<NUM>) while the human operator (<NUM>) is looking (<NUM>) towards the drone (<NUM>); and
superimposing, on the augmented reality display (<NUM>), a map (<NUM>) in a vertical layout showing a geographic location (<NUM>) of the human operator (<NUM>) and a geographic location (<NUM>) of the drone (<NUM>) in the vicinity of the target symbol (<NUM>) on the augmented reality display (<NUM>) while the human operator (<NUM>) is looking (<NUM>) towards the drone (<NUM>) in the air (<NUM>); or
superimposing, on the augmented reality display (<NUM>), a map (<NUM>) in a horizontal layout showing a geographic location (<NUM>) of the human operator (<NUM>) and a geographic location (<NUM>) of the drone (<NUM>) while the human operator (<NUM>) is looking (<NUM>) towards the ground (<NUM>).