Systems and methods for augmented reality video feed on an unmanned vehicle

A remote station and method for rendering a ground plane over video feed are provided. The remote station comprises a processor, a communication interface for communicating with an unmanned vehicle (UV), and a non-transitory memory device storing a communications module. The communications module comprises machine-readable instructions that, when executed by the processor, causes the processor to render a ground plane over video feed. The method comprises receiving a video feed from a camera on a UV, receiving from the UV telemetry information of the camera, receiving from the camera a zoom factor of the camera, calculating a horizon of the camera at a UV controller of the UV, and rendering the horizon as an overlay image on the video feed at a display of the UV controller.

FIELD

This disclosure generally relates to the field of unmanned vehicles, and in particular to augmented reality video feed on an unmanned vehicle.

BACKGROUND

An unmanned aerial vehicle (UAV) does not have a human operator located at the UAV. A UAV may include various components such as sensors and measurement and navigation instruments. A UAV may carry a payload which may be configured to perform specific duties such as taking aerial photographs and videos.

SUMMARY

In accordance with some embodiments, there is provided a remote station comprising a processor, a communication interface for communicating with an unmanned vehicle (UV), and a non-transitory memory device storing machine-readable instructions that, when executed by the processor, causes the processor to render a ground plane over images or video feed. The processor is configured to receive an image or video feed from a camera on the UV, receive from the UV telemetry information of the camera, receive from the camera a zoom factor of the camera, calculate a horizon of the camera, and render the horizon as an overlay image on the image or video feed.

In accordance with some embodiments, there is provided a method for rendering a ground plane over images or video feed. The method comprises receiving an image or video feed from a camera on an unmanned vehicle (UV), receiving from the UV telemetry information of the camera, receiving from the camera a zoom factor of the camera, calculating a horizon of the camera at a UV controller of the UV, and rendering the horizon as an overlay image on the image or video feed at a display of the UV controller.

In accordance with some embodiments, there is provided an observer overlay system. The observer overlay system comprises a processor, a communication interface for communicating with a camera, and a non-transitory memory device storing machine-readable instructions that, when executed by the processor, causes the processor to render a ground plane over images or video feed from the camera. The processor is configured to receive an image or video feed from the camera, receive telemetry information from the camera, receive a zoom factor of the camera from the camera, calculate a horizon of the camera, and render the horizon as an overlay image on the image or video feed.

In accordance with some embodiments, there is provided a method for rendering a ground plane over images or video feed. The method comprises receiving an image or video feed from the camera, receiving telemetry information from the camera, receiving a zoom factor of the camera from the camera, calculating a horizon of the camera, and rendering the horizon as an overlay image on the image or video feed.

In various further aspects, the disclosure provides corresponding systems and devices, and logic structures such as machine-executable coded instruction sets for implementing such systems, devices, and methods.

In this respect, before explaining at least one embodiment in detail, it is to be understood that the embodiments are not limited in application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

It is understood that throughout the description and figures, like features are identified by like reference numerals.

DETAILED DESCRIPTION

The term unmanned vehicle (UV) is used herein and may include an unmanned aerial vehicle (UAV), an unmanned aircraft (UA), an unmanned aquatic vessel, an unmanned ground vehicle (UGV), and any other vehicle or structure which may be unmanned, operate autonomously or semi-autonomously, and/or controlled remotely. The UGV may be a remotely controlled, autonomous or semi-autonomous vehicle system which is comprised of a main body and a drive system supported by the main body. In some examples, the drive system is comprised of a propulsion system, such as a motor or engine, and one or more tracks or wheels. Other arrangements, such as a rail or fixed-track ground vehicle, a tether or rope-pulled ground vehicle without a motor or engine, a ground vehicle using balls, sleds or rails, and a ground vehicle which hovers but navigates in proximity to terrain, are also contemplated herein.

Some of the features taught herein are described with reference to embodiments of a UAV by way of example only. However, the description and features may also apply generally to any UV.

FIG. 1illustrates an example of an unmanned system (US)100(such as an unmanned aircraft system) comprising an unmanned vehicle (UV)110(such as an unmanned aerial vehicle) and its associated system elements, in accordance with some embodiments. The UV110may be designed to operate with no operator (or pilot) onboard. In the embodiment shown inFIG. 1, the unmanned system100includes a remote operator (or pilot) station102and command and control links104between the UV110and the remote operator (or pilot) station102. The command and control links104may include any data link for the purposes of managing the movement (e.g., flight) of the UV110. The UV110may operate autonomously without operator (or pilot) intervention in the management of the movement (e.g., flight) during the entire movement (e.g., flight) operation or a portion thereof. The unmanned system100may also include other system elements as may be required at any point during movement (e.g., flight) operation.

In some embodiments, UV110may be an unmanned aircraft (UA) or UAV as shown inFIG. 1.

The example UV110shown inFIG. 1may include a body112, arms114extending away from the body112to support components such as propellers116, and legs118to support the body112when UV110is positioned on a surface. When not in use, a propeller may be in a folded position. It is understood that propellers116may be in the folded position during storage of the UV110, while the open position is used during flight operation of the UV110. Although four arms114and four legs118are illustrated in the embodiment shown inFIG. 1, it is understood that UV110may include any other number of arms114and legs118. As noted above, the example ofFIG. 1pertains to a UAV by way of example only. Other types of UVs may also employ the teachings described herein.

In some embodiments, remote pilot (or operator) station102may comprise a ground station. In other embodiments, remote pilot (or operator) station102may comprise a client device acting as a control station. In still other embodiments, remote pilot (or operator) station102may comprise both a ground station and a client device.

FIG. 2illustrates, in a component diagram, an example of a US200, in accordance with some embodiments. The US200may include one or more loaded vehicles210, a ground station240, and one or more client devices250. The US200may include more than one ground station240. A loaded vehicle210may include a UV110and a payload220. The ground station240may communicate with one or more loaded vehicles210via air interface230which may include satellite communication or other types of radio frequency communication between station240and loaded vehicles210. The ground station240may communicate with one or more client devices250through a number of communication links and network interfaces, such as a wired or wireless local area network, a cellular network (such as global system for mobile (GSM) communication, long-term evolution (LTE), fifth generation (5G), or other cellular networks) or a proprietary or private radio link.

A loaded vehicle210may include a UV110and a payload220. The payload220may include one or more of: a freight package, a camera, a measuring device, one or more sensors, and a storage device (e.g., a universal serial bus (USB) drive). A payload220can also include, for example, flame retardant for use in a forest fire. Generally speaking, a payload220may be any cargo or equipment a UV110carries that is not necessarily required for flight, control, movement, transportation and/or navigation of the UV110itself. A payload220may be attached or coupled to the UV110in a number of ways. For example, a payload220may be connected to the UV110by one or more interfaces such as an Ethernet connection, a controller area network (CAN) bus connection, a serial connection, an inter-integrated circuit (I2C) connection, a printed circuit board (PCB) interface, a USB connection, a proprietary physical link, and so on.

The ground station240may be configured to communicate with one or more loaded vehicles210(or simply “vehicles210” hereinafter). The ground station240may also communicate with UVs110not carrying any payload. The ground station240may control one or more loaded vehicles210, one or more UVs110, one or more payloads220concurrently in real-time or near real-time. The ground station240may also receive commands and/or data from one or more client devices250, process the commands or data, and transmit the processed commands or data to one or more vehicles210, UVs110, or payloads220. In some embodiments, the ground station240may receive user input directly at a user console (not shown) without client devices250. In some embodiments, a client device250may be the user console for the ground station240.

A client device250may serve to control the operation of one or more vehicles210, UVs110, or payloads220remotely. In some embodiments, a client device250may also be referred to as a control station. The client device250may be implemented as a computing device.

A user, such as an owner or operator of a UV110, may use a client device250to communicate with, and to control, one or more vehicles210, UAVs110, or payloads220. A client device250may have an application implemented for communicating with or controlling vehicles210, UVs110, or payloads220. Such an application may be launched as a stand-alone process in an operation system, or within an Internet browser. The user may enter information through a user interface provided by the application. In addition, information relating to, or from, the vehicle210, UV110, or payload220may be displayed by the application on a display of client device250. Client device250may communicate with, or control, vehicle210, UV110, or payload220through the ground station240, or in some embodiments, client device250may communicate with, or control, vehicle210, UV110, or payload220directly without the ground station240.

In some embodiments, the client device250is operable to register and authenticate users (using a login, unique identifier, biometric information or password for example) prior to providing access to loaded vehicles, payloads, UVs, applications, a local network, network resources, other networks and network security devices. The client device250may serve one user or multiple users.

Either or both of the ground station240and the client device250may be configured to control vehicle210, UV110, or payload220. Flight control, navigation control, movement control, and other types of command signals may be transmitted to the UV110for controlling or navigating one or more of vehicle210, UV110, or payload220. Command signals may include command data (e.g., coordinate information) required to execute flight control, movement control or navigation control of one or more of vehicle210, UV110, or payload220.

Either or both of the ground station240and the client device250may be configured to receive data from one or more of vehicle210, UV110, or payload220. For example, payload220may transmit data, including but not limited to, sensor data, audio, video or photographs to the ground station240or to the client device250.

FIG. 3illustrates, in a component diagram, an example of a ground station240, in accordance with some embodiments. The ground station240may include a sensor subsystem302(which may include a global positioning system (GPS) subsystem), a communications module304configured to process received data packets, and to prepare data packets for transmission through an external radio frequency (RF) interface306, an external RF interface configured to communicate with an external RF interface on a UV110, a processor or controller308, a payload control module310, and a UV control module312. The sensor subsystem302may be used to acquire environmental data if the ground station240is proximate or near the UV110, where the environmental data may be used for controlling the UV110, the payload220, or the loaded vehicle210, such as location data, weather data, and so on. The payload control module310may generate command signals for controlling the payload220, and the UV control module312may general command signals for controlling the UV110. Both types of control commands may be processed by the communications module304and transmitted to the UV110and the payload220via external RF interface306. The ground station240may also include an operator console (not shown) that includes a display (not shown) providing video feed from a camera payload on the UV110. The embodiments described herein refer to a video feed from a camera. It should be understood that the same teachings apply to an image or video feed from the camera.

FIG. 4illustrates, in a component diagram, an example of a client device250, in accordance with some embodiments. The client device250may comprise a communications subsystem404, a processor or central computer system408and a display402. The communications subsystem404allows for seamless communications between the client device250and UV110, seamless communications between the client device250and payload220, and seamless communications between the client device250and each ground station240, when ground stations240are used. The user interface (UI)406is generated by processor408for display on the display402of a client device250, which remotely controls the UV110, the payload220, and/or the loaded vehicle210or as part of a control system for one or more vehicles210. Display402may be a touch-screen display, or a non-touch display. In some embodiments, client device250may be on a single-unit computer (e.g., one with a built-in display), or a multi-unit computer (e.g., with a separate display). The payload control module410may generate command signals for controlling the payload220, and the UV control module412may general command signals for controlling UV110. Both types of control commands may be processed by communications module404and transmitted to the UV110and the payload220via the ground station240.

The client device250is configured to display at least a subset of the received vehicle status data for each UV110or payload220in an interface (such as UI406, for example). A display402may provide a graphical representation of the respective vehicle location data of each of the vehicles110. Through the interface406, the client device250may receive control command input. The control command input is associated with one of the UV110having its vehicle status data displayed in the interface406. The client device250may then transmit the received control command, or a command derived therefrom, to the UV110. The interface406may enable a user to view status and control operation of each of one or more UVs110such that the location of each UV110is shown in the interface406, and each UV110may be independently controlled through the interface406by selecting a particular one of the UV110to control. In this way, multiple UV110may be monitored and controlled through an interface406at the client device250.

Further detail on the controlling UVs110using interface406is provided in PCT Application No. PCT/CA2013/000442 entitled “System and Method for Controlling Unmanned Aerial Vehicles”, the entire contents of which are hereby incorporated by reference. Client device or control station250may control interface panels to display a location of the UV110.

FIG. 5illustrates, in a component diagram, an example of a control station500, in accordance with some embodiments. The control station500may be a client device250, and/or a ground station240having a display, and/or a remote pilot station102. In some embodiments, the control station500may be implemented on a tablet, phone, computer, purpose-built control station or other capable device or system. A processor or controller408can execute instructions in memory512to configure the communications module404, the payload control module410and the UV control module412. A processor408can be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, or any combination thereof.

Each I/O unit502enables the control station500to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices, such as a display screen402and a speaker. The discussion below will focus on a camera (payload) as an input device and a display402as the output device. As will be further described below, UV110telemetry readings will also be used as input.

Each communication unit or interface404enables the control station500to communicate with other components, to exchange data with other components, to access and connect to network resources, to serve applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these. For example, a communication interface506may include an Ethernet connection to the ground station240, or a wireless communication interface operable to communicate with ground station240. In some embodiments, the communication interface404may include a RF interface operable to communicate with the UV110.

In some embodiments, the UI406in a control station500(and/or a UI implemented in a ground station240having a display; and/or a UI implemented in a remote pilot station102having a display) may compute and display an overlay on top of a video feed from a UV110. In some embodiments, the overlay may comprise a grid showing equivalent distances on the ground.FIG. 6illustrates, in a screenshot, an example of a grid overlay602on a video feed, in accordance with some embodiments. The grid overlay602is a milliradian (mRad) overlay comprising a static image with specific characteristics that an observer may then use to mathematically approximate sizes and distances. Other features may be implemented to allow the use of UI elements to calculate linear distances on the ground, drive, fly, move and/or navigate the UV110through telemetry information in the absence of a video feed, show navigational features on the grid, and calculate/calibrate errors in the GPS.

In some embodiments, it is desirable for a UV110with video capability to providing ranging information to the operator of the UV110from a video pane in the display402. Ranging is the ability to show distances (ranges)—either distances between objects on the screen, or distances from the UV110to a point or object. It is also desirable to allow operators in other mission scenarios to measure objects and terrain, determine if a vehicle can get through a gap in structures or obstacles, and measure the speed of an object.

FIG. 7illustrates, in a flowchart, an example of a method700of rendering a ground plane over video feed, in accordance with some embodiments. The method may be performed by the control station700, and comprises receiving702a video feed from a camera on a UV110, receiving704from the UV telemetry information of the camera, receiving706from the camera a zoom factor of the camera, calculating708a horizon of the camera, and rendering710the horizon as an overlay image on the video feed. In some embodiments, the telemetry information of the camera includes, but is not limited to, a location position, an orientation position, an altitude, and any other data used to calculate a horizon. In some embodiments, the telemetry information of the camera may be identical or similar to the telemetry information of the UV100. It should be understood that steps702,704and706may be performed in any order. A UV110may be equipped with a video camera as a payload220, which streams video back to a control station500. The control station500may display and/or store this video feed that it receives702from the camera. The UV110may consistently report its location and orientation and the direction/angle of view of the camera to the control station500. The control station500and/or the ground station240may also know the field of view of the camera. This field of view may be hard-coded in the control station storage510or transmitted by the UV110at some point, along with the current zoom factor of the camera. In some embodiments, the zoom factor of the camera and the location and orientation of the UV110may be acquired some time before step702and stored in a cache. The ground station240may then calculate a horizon for the camera based upon the telemetry information of the UV110and the field of view and zoom factor of the camera. One algorithm that may be used to calculate the horizon is called H-COV-LUM. It operates on the principle that if an image is divided into two regions with a line, and then the variance of luminance is calculated for each region (on either side of the line), then the horizon is the line that minimizes the sum of those two variances. There are other algorithms for calculating the horizon. The calculated horizon may then be rendered as an overlay image on top of the video image in the display402. Thus, the method700calculates and renders a ground plane based on telemetry information from the UV110.

FIG. 8illustrates, in a flow chart, another method800of rendering a ground plane, in accordance with some embodiments. The method800comprises the method700, calculating812a three dimensional (3D) model of the ground plane, and rendering814a grid as an overlay image on the video feed. The grid may shows equal distances at ground level. As shown inFIG. 6, the grid602may be an mRad grid.FIG. 9illustrates, in a screenshot, an example of a rectangular grid902, in accordance with some embodiments, stretching from the bottom of the field of view out to the horizon904. The user can then observe ground distances from this grid by viewing the image overlaid over the video feed on the display402. The grid902is rectangular and made of major grids906where each major grid906is sub-divided into a minor grid908. The grid902spacing details may be calculated dynamically (e.g., based on the distance to the ground of the camera) or may be predetermined and selected in a user setting. The grid may alternatively be a radial grid.FIG. 10illustrates, in a screenshot, an example of a radial grid overlay1002on a video feed, in accordance with some embodiments. The radial grid1002also allows for the communication of angular direction. Major angular grid lines1006may be rendered at 40 or 30 increments for ease of presentation and readability for the observer. The remainder of this description will focus on the rectangular grid902format. However, it should be understood that the teachings may be modified to apply to the radial grid1002format.

The user may adjust the location of the horizon in the UI406, to account for weather-induced telemetry issues or other inaccuracies in the telemetry information. Alternatively, the ground station240and/or the UV110may sense these inaccuracies in the video feed (for instance, by algorithmically examining the video feed for a horizon line) and account for them. Once the location of the horizon is changed, the horizon rendering may be modified accordingly. Moreover, the grid902,1002may be recalculated based on the new location of the horizon and re-rendered accordingly.

FIG. 11illustrates, in a flowchart, an example of a method1100of determining a ground plane point, in accordance with some embodiments. The user may use a UI element (such as a clicking with a tablet stylus or mouse) to select a point on the grid902,1002. The control station500receives1102this point (i.e., first location selection input associated with a first point on a video feed), calculates1104where on the ground plane this point would lie (i.e., a first ground plane point associated with the first video feed point), and correlate1106the first ground plane point with the telemetry information received from the UV110, resulting in GPS co-ordinates of a first point on the ground corresponding to the point that the user selected (i.e., associated with the first video feed point). The control station500may also calculate the ground distance from this point to the UV110, the distance from this point to a home base or other known point, an estimated time to drive to, navigate to, fly to above, or otherwise move to this point, or other navigational or cartographic information about this point. Any of this information may be displayed by the control station500.

FIG. 12illustrates, in a flowchart, an example of a method1200of determining a distance between two point, in accordance with some embodiments. A user may use a UI element (such as drawing a line with a stylus or clicking and dragging with a mouse) to draw a line on the grid902. The control station500can calculate the starting and ending points of this line, using the method ofFIG. 11described above. I.e., method1100may be performed1202on the starting point, and method1100may be performed1204on the ending point. Next, the distance between the ground level starting and ending points may be calculated1206, and rendered1208on the display402. Ancillary information, such as estimate navigation, transportation, movement or flight time between these points, may also be rendered.FIG. 13illustrates, in a screenshot, an example of a rendering of the line1302and calculated distance1304between the endpoints1306,1308on a rectangular grid902, in accordance with some embodiments.FIG. 14illustrates, in a screenshot, another example of a rendering of the line1402and calculated distance1404between the endpoints1406,1408on a radial grid1002, in accordance with some embodiments.

As the zoom factor on the UV110camera changes (whether autonomously or from user control), the UV110may send back updates on the current zoom factor. The control station500may recalculate the location of the horizon904and the layout of the grid902,1002. These recalculations may be dynamically updated on the UI406as the zoom factor changes so that the overlay matches the video.

In some embodiments, it is desirable to see how much of a footprint a UV110would occupy at a certain location in the video image, e.g., how large the UV110would be if it were 100 metres (or any other distance) ahead of its current location. This is useful for determining whether the UV110would fit into an opening which can be seen in the video image.

To facilitate this, the overlay may have a feature where the user may specify a location for the UV110(either an absolute location or a location relative to the UV's current position), and the UI406will show a representation of the UV110(or a box which approximates the size of the UV110) at that location in the video image.

In some embodiments, it is desirable to see what regions can likely hear the UV110at any given time. This may be used in stealth operations, but may have other applications, such as abiding by noise bylaws.FIG. 15illustrates, in a flowchart, an example of a method1500of rendering audibility model surface, in accordance with some embodiments. A control station500may perform the method1500by using an audibility model of a known level of loudness for the UV110. For example, the UV110may be known to emit an audial sound (e.g., due to the vibration of the motor, etc.) at a specific level that is audible at a corresponding radius from the UV110. In some embodiments, a plurality of audial sound levels to distance radius may be known. The control station500may generate1502a 3D model of the loudness at a given distances from the UV110(assuming a constant attenuation of the loudness with distance from the UV110), generate1504a 3D surface (e.g., a sphere) corresponding to a specific level of loudness, and then indicate where that surface intersects with the ground (or other) plane. For example, the control station500may render on the display402where the surface intersects the grid902,1002plane.

Enhancements to this process may include real-time loudness measurements (instead of a single hard-coded value for loudness), accounting for wind when calculating the attenuation of the loudness with distance, accounting for surface reflections, and accounting for the nature of the local ground cover (i.e., trees attenuate more than open air). An operator may use the audibility model calculations to pilot the UV110such that it is flying above a distance from the ground or another plane. Such distance being the minimum distance so that the UV110does not emit a sound above a loudness threshold.

An observer overlay902,1002allows for piloting the UV110much more accurately than with just video alone. For more accurate navigation, since the calculations of the grid902,1002use a generation of a 3D model of the navigational area and ground plane, navigational features such as waypoints, targets, no-fly zones, or other elements, may be plotted on the grid902,1002as part of the overlay. The UI406may have controls for turning these elements on and off on the display402. As with the point/line selection discussed above, the UI406may allow for the calculation of distances between selected points and navigational elements.

Sometimes, video is not available, but telemetry information (GPS, compass, altimeter, roll/pitch/yaw measurements, etc.) may still be available. This can happen when the UV110is a long distance from the control station500(or from an intermediate ground station240that may be relaying communications). For example, the high-reliability/low-speed control/telemetry link may still be reliable between the UV110and the control station500, but the less reliable, high-speed video channel may be too lossy for usable video. In this case, the control station500may allow the user to fly, navigate, drive or otherwise move the UV110using only the overlay (without a background, or with intermittent still images).

In some embodiments, as a result of weather or other issues, GPS, compass or other telemetry information may become increasingly inaccurate or may be unavailable. If the telemetry information is not available, the control station500may take action. In some embodiments, the control station500may use the last available telemetry values to calculate the grid902,1002(i.e., not update the grid). This method may be used when the telemetry outages are brief or when the UV110continues operating as it was doing before the telemetry information became unavailable. In some embodiments, if there is a video stream present, the control station500may examine frame-to-frame changes to calculate an updated grid902,1002. In some embodiments, the control station500may display an indication that the overlay may not be accurate. In some embodiments, the control station500may disable the overlay feature. The control station may take these actions based on a score calculated by the presence or absence of various pieces of telemetry information, or their perceived accuracy. One example of calculating telemetry inaccuracies is to track changes in the orientation reported by a compass—in the absence of controlling the UV110to change its orientation. The variance of the reported orientations taken over a certain period should not exceed a threshold. An unreliability score for the compass may be calculated by subtracting this variance from the threshold.

If topographical information for the area being displayed is available from a stored or online source of terrain data (e.g., digital terrain elevation data (DTED), Google Maps, etc.), the ground contours may be calculated from the camera's perspective and may be incorporated into the 3D model of the ground plane. This would allow for more accurate ground-level distance calculations and more accurate mapping when the terrain is uneven or when the ground is sloping.

If an object of a known size is present in the video image, the user may be able to draw a line on the overlay (as noted above), but rather than have the UI406report the distance between the endpoints of the line, the user specifies that distance through the UI406. This may allow the recalculation/recalibration of the overlay grid902,1002with this new information. The calculated values of the distances in the grid902,1002may be re-adjusted to align the calculations with the empirical value that is being provided. For example, the calculation of the grid902,1002may indicate a distance of 1030 metres (m) (or any other distance) between two points. However, if the user knows that this distance is precisely 1000 m, they may draw a line between the two points on the display and specify that the grid902,1002be recalculated to show this distance as 1000 m. The calculations could be worked backwards to provide new grid coefficients, or the values could be iteratively changed until the grid902,1002lines up with the newly-provided data.

The overlay902,1002as discussed above may be a rendering of the ground plane from the perspective of the UV110, meaning that distances and lines drawn on this overlay902,1002are all as though they were at ground level. While this is useful for many applications, there are situations where a navigational plane other than the ground plane may be desired. For example, if the UV110is flying in a heavily treed area or in an urban environment with many buildings, it may be desired to plot a plane at the altitude of the tree canopy or of the local rooftops. This would allow for the user to see distances at an altitude corresponding to the visual features they are seeing in the video.

Since a 3D model is created to be able to plot the ground plane, another plane at an arbitrary altitude may be calculated and plotted from the same information set. Multiple altitude planes may be specified and displayed, differentiated by color or another UI characteristic.

In some embodiments, the UV110itself may calculate the overlay902,1002and transmit it within the video stream. In some embodiments, the UV110or an intermediate ground station240may perform some of the calculations necessary for the overlay, with the final calculations and display performed by the control station500.

While this discussion has focused on video transmissions and viewing on the ground station240, the same features may be accommodated on static images, or periodically-updated still images, sent back from a video or still camera.

In some embodiments, the overlay902,1002may be used to determine the distance between objects or points in the video feed, including between the UV and another UV, or between other UVs.

In some embodiments, the UV110may be a UAV that calculates the overlay902,1002during flight or rest of the UAV. The overlay902,1002during flight may be used to determine the dimensions of a confined area and, with knowledge of the dimensions of the UAV (either stored or obtained), determine if the UAV may fit within the confined area. In some embodiments, the dimensions of the confined area may be used to determine if another vehicle or structure (with stored or obtained knowledge of the dimensions) may fit within the confined area. In some embodiments, the observer overlay of a UAV may be used to assist with the navigation of another UV (e.g., a UGV).

In some embodiments, the UV110may be a UGV that calculates the overlay902,1002during movement or rest of the UGV. During movement or rest, the UGV may determine if it may fit within an upcoming a confined space. Such determination may take place when the UGV is at the same plane as the confined space, or of the UGV is an a higher altitude looking down upon an area to determine if the UGV may navigate in that area.

In some embodiments, the UV110may be a unmanned aquatic vessel that calculates the overlay902,1002during navigation or rest of the vessel. The observer overlay system may determine if the vessel may fit within an upcoming confined space, or how large are waves in the distance approaching the vessel.

Other UVs or structures may employ an observer overlay system. In some embodiments, a fixed and/or robotic structure having a camera may employ an observer overlay system. In some embodiments, the camera may be affixed or coupled to the fixed and/or robotic structure or on an arm or other sub element of the fixed and/or robotic structure. The fixed and/or robotic structure, sub element or camera may be configured to rotate or otherwise move such that the camera may obtain different views of images and/or video. In some embodiments, a security zone may employ one or more cameras on one or more fixed and/or robotic structures to calculate the overlay902,1002to determine the speed, distance, and other measurements between objects within the camera's field of view of the zone.

FIG. 16illustrates, in component diagram, an example of an observer overlay system1600, in accordance with some embodiments. The observer overlay system1600comprises a processor1610for processing operating instructions, a communications unit1620for receiving images and/or video feed from a camera, a memory1630that stores operating instructions for the observer overlay system1600, and an overlay module1640for calculating an overlay902,1002of the view of the images and/or video feed. Other components may be added to the observer overlay system1600, including one or more cameras and one or more UVs.

The processor1610may be configured to perform many of the methods described above. For example, the processor1610may be configured to receive an image or video feed from the camera, receive from the camera telemetry information, receive from the camera a zoom factor of the camera, calculate a horizon of the camera; and render the horizon as an overlay image on the image or video feed. In some embodiments, the telemetry information of the camera includes, but is not limited to, a location position, an orientation position, an altitude, and any other data used to calculate a horizon. In some embodiments, the telemetry information of the camera may be identical or similar to the telemetry information of an object (such as, but not limited to, the UV100) to which the camera is coupled.

Throughout the foregoing discussion, numerous references may be made regarding control and computing devices. It should be appreciated that the use of such terms may represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a remote station102,240,250,500may have a server that includes one or more computers coupled to a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

The foregoing discussion provides many example embodiments. Although each embodiment represents a single combination of inventive elements, other examples may include all possible combinations of the disclosed elements. Thus, if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, other remaining combinations of A, B, C, or D, may also be used.

The technical solution of embodiments may be in the form of a software product instructing physical operations, such as controlling movement of the UV110, for example. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the processes provided by the embodiments.

The processor or controller308,408, ground station240, or client device250,500may be implemented as a computing device with at least one processor, a data storage device (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. The computing device components may be connected in various ways including directly coupled, indirectly coupled via a network, and distributed over a wide geographic area and connected via a network (which may be referred to as “cloud computing”).

For example, and without limitation, the computing device may be a server, network appliance, microelectromechanical systems (MEMS) or micro-size mechanical devices, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, cellular telephone, smartphone device, UMPC tablets, video display terminal, gaming console, electronic reading device, and wireless hypermedia device or any other computing device capable of being configured to carry out the processes described herein.

A processor may be, for example, a general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

Computing device may include an I/O interface to enable computing device to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker.

Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the scope as defined by the appended claims.

As can be understood, the examples described above and illustrated are intended to be exemplary only. The scope is indicated by the appended claims.