Inferring the presence of an occluded entity in a video captured via drone

In one example, the present disclosure describes a device, computer-readable medium, and method for inferring the presence of an occluded entity in a video captured via drone. For instance, in one example, a video is obtained. The video is captured by a drone monitoring a field of view of a scene. It is determined, based on a combination of statistical reasoning and contextual modeling of the video, that an occluded entity is likely to be present, but not entirely visible, in the field of view. A signal is sent to the drone to instruct the drone to adjust its orientation to make the occluded entity more visible.

The present disclosure relates generally to computer vision, and relates more particularly to devices, non-transitory computer-readable media, and methods for determining when an entity is occluded in a video.

BACKGROUND

Drones are unmanned aerial vehicles (UAVs) that may be remotely controlled or may be flown autonomously (e.g., using computer vision processes). Although previously used largely in a military context, in recent years, civilian use of drones has become more widespread. For instance, drones are often used to capture video for the purposes of surveillance, traffic and weather monitoring, personalization, biometrics, and the like.

SUMMARY

In one example, the present disclosure describes a device, computer-readable medium, and method for inferring the presence of an occluded entity in a video captured via drone. For instance, in one example, a video is obtained. The video is captured by a drone monitoring a field of view of a scene. It is determined, based on a combination of statistical reasoning and contextual modeling of the video, that an occluded entity is likely to be present, but not entirely visible, in the field of view. A signal is sent to the drone to instruct the drone to adjust its orientation to make the occluded entity more visible.

In another example, a device includes a processor and a computer-readable medium storing instructions which, when executed by the processor, cause the processor to perform operations. The operations include obtaining a video that is captured by a drone monitoring a field of view of a scene, determining that an occluded entity is likely to be present, but not entirely visible, in the field of view based on a combination of statistical reasoning and contextual modeling of the video, and sending a signal to the drone to instruct the drone to adjust its orientation to make the occluded entity more visible.

In another example, a non-transitory computer-readable storage medium stores instructions which, when executed by a processor, cause the processor to perform operations. The operations include obtaining a video that is captured by a drone monitoring a field of view of a scene, determining that an occluded entity is likely to be present, but not entirely visible, in the field of view based on a combination of statistical reasoning and contextual modeling of the video, and sending a signal to the drone to instruct the drone to adjust its orientation to make the occluded entity more visible.

DETAILED DESCRIPTION

In one example, the present disclosure infers the presence of an occluded entity in a video captured via drone. As discussed above, drones are often used to capture video for the purposes of surveillance, traffic and weather monitoring, personalization, biometrics, and the like. In many of these fields, an entity of interest (e.g., a person or an object) may be occluded in the captured video due to the field of view of the drone camera. That is, although the entity is present in a captured scene, it may not be entirely visible from the particular field of view of that scene that is captured by the drone camera (e.g., due to another object being positioned between the drone sensor and the occluded entity). Complicating the matter is the fact that the typical characteristics of videos captured via drone are different from the characteristics of video captured via handheld techniques or even via other aerial techniques.

Examples of the present disclosure provide a way of inferring when an entity is occluded from a particular field of view of a drone, and of responsively maneuvering the drone to a new field of view from which the target is no longer occluded. In one example, a combination of statistical reasoning and contextual modeling is used to infer when an occlusion is present in a video captured by a drone. The statistical reasoning helps to infer the presence of occlusions based on typical or expected characteristics of occlusions (as determined by analysis of historical or training videos). The contextual modeling helps to infer the presence of occluded entities from the presence of other contextually related entities in the field of view. If it is determined that an occlusion is likely to be present based on the statistical reasoning and contextual modeling, the orientation of the drone can be responsively maneuvered to confirm the presence of the occlusion and to adjust the field of view so that the occlusion is removed.

To better understand the present disclosure,FIG. 1illustrates an example network100, related to the present disclosure. The network100may be any type of communications network, such as for example, a traditional circuit switched network (CS) (e.g., a public switched telephone network (PSTN)) or an Internet Protocol (IP) network (e.g., an IP Multimedia Subsystem (IMS) network, an asynchronous transfer mode (ATM) network, a wireless network, a cellular network (e.g., 2G, 3G and the like), a long term evolution (LTE) network, and the like) related to the current disclosure. It should be noted that an IP network is broadly defined as a network that uses Internet Protocol to exchange data packets. Additional exemplary IP networks include Voice over IP (VoIP) networks, Service over IP (SoIP) networks, and the like.

In one embodiment, the network100may comprise a core network102. In one example, core network102may combine core network components of a cellular network with components of a triple play service network; where triple play services include telephone services, Internet services, and television services to subscribers. For example, core network102may functionally comprise a fixed mobile convergence (FMC) network, e.g., an IP Multimedia Subsystem (IMS) network. In addition, core network102may functionally comprise a telephony network, e.g., an Internet Protocol/Multi-Protocol Label Switching (IP/MPLS) backbone network utilizing Session Initiation Protocol (SIP) for circuit-switched and Voice over Internet Protocol (VoIP) telephony services. Core network102may also further comprise an Internet Service Provider (ISP) network. In one embodiment, the core network102may include an application server (AS)104and a database (DB)106. Although only a single AS104and a single DB106are illustrated, it should be noted that any number of application servers and databases may be deployed. Furthermore, for ease of illustration, various additional elements of core network102are omitted fromFIG. 1, including switches, routers, firewalls, web servers, and the like.

The core network102may be in communication with one or more wireless access networks120and122. Either or both of the access networks120and122may include a radio access network implementing such technologies as: global system for mobile communication (GSM), e.g., a base station subsystem (BSS), or IS-95, a universal mobile telecommunications system (UMTS) network employing wideband code division multiple access (WCDMA), or a CDMA3000 network, among others. In other words, either or both of the access networks120and122may comprise an access network in accordance with any “second generation” (2G), “third generation” (3G), “fourth generation” (4G), Long Term Evolution (LTE), or any other yet to be developed future wireless/cellular network technology including “fifth generation” (5G) and further generations. The operator of core network102may provide a data service to subscribers via access networks120and122. In one embodiment, the access networks120and122may all be different types of access networks, may all be the same type of access network, or some access networks may be the same type of access network and other may be different types of access networks. The core network102and the access networks120and122may be operated by different service providers, the same service provider or a combination thereof.

In one example, the access network120may be in communication with one or more user endpoint devices (also referred to as “endpoint devices” or “UE”)108and110, while the access network122may be in communication with one or more user endpoint devices112and114.

In one example, the user endpoint devices108,110,112, and114may be any type of subscriber/customer endpoint device configured for wireless communication such as a laptop computer, a Wi-Fi device, a Personal Digital Assistant (PDA), a mobile phone, a smartphone, an email device, a computing tablet, a messaging device, a wearable “smart” device (e.g., a smart watch or fitness tracker), a portable media device (e.g., an MP3 player), a gaming console, a portable gaming device, a set top box, a smart television, and the like. In one example, at least some of the UEs108,110,112, and114are drones equipped with video cameras. In one example, any one or more of the user endpoint devices108,110,112, and114may have both cellular and non-cellular access capabilities and may further have wired communication and networking capabilities (e.g., such as a desktop computer). It should be noted that although only four user endpoint devices are illustrated inFIG. 1, any number of user endpoint devices may be deployed.

The AS104may comprise a general purpose computer as illustrated inFIG. 4and discussed below. In one example, the AS104may perform the methods discussed below related to inferring the presence of an occluded entity in a video captured via drone. For instance, in one example, the AS104hosts an application that communicates with one or more of the UEs108,110,112, and114. As an example, the application may be a surveillance application, or a traffic or weather monitoring application, that subscribes to the output (e.g., video stream) of one or more of the UEs108,110,112, and114. In particular, the AS104may receive videos recorded by the UEs108,110,112, and114and may analyze the videos to infer the presence of occluded entities. The AS104may further send signals to the UEs108,110,112, and114instructing the UEs to adjust their orientations so that the videos they are recording are captured from a different field of view in which the entities are not occluded.

In one example, the DB106may store videos recorded by one or more of the UEs108,110,112, or114, e.g., by one or more drones. These videos may include videos that contain occlusions and videos that do not contain occlusions. These videos may be used to train the AS104to infer when an occluded entity is likely to be present (but not entirely visible) in a new video.

It should also be noted that as used herein, the terms “configure” and “reconfigure” may refer to programming or loading a computing device with computer-readable/computer-executable instructions, code, and/or programs, e.g., in a memory, which when executed by a processor of the computing device, may cause the computing device to perform various functions. Such terms may also encompass providing variables, data values, tables, objects, or other data structures or the like which may cause a computer device executing computer-readable instructions, code, and/or programs to function differently depending upon the values of the variables or other data structures that are provided.

Those skilled in the art will realize that the network100has been simplified. For example, the network100may include other network elements (not shown) such as border elements, routers, switches, policy servers, security devices, a content distribution network (CDN) and the like. The network100may also be expanded by including additional endpoint devices, access networks, network elements, application servers, etc. without altering the scope of the present disclosure.

To further aid in understanding the present disclosure,FIG. 2illustrates a flowchart of an example method200for training a system to infer the presence of an occluded entity in a video captured via drone. In one example, the method200may be performed by an application server that subscribes to the output of one or more drones, e.g., AS104illustrated inFIG. 1. However, in other examples, the method200may be performed by another device. For instance, a UE108,110,112, or114that is in communication with a drone, or even a UE that is a drone could also perform all or some steps of the method200. As such, any references in the discussion of the method200to the AS104ofFIG. 1are not intended to limit the means by which the method200may be performed.

The method200begins in step202. In step204, the AS104obtains a set of training data. In one example, the training data comprises a collection of videos recorded by drones. At least some of the videos depict fields of view of scenes in which entities (e.g., people or objects) are occluded. Fields of view of scenes depicted in other videos do not include occluded entities. In one example, each of the videos in the training data is labeled, e.g., to indicate whether or not an occluded entity is present in the video. The training data may be stored, for example, in the DB106.

In step206, the AS104applies a statistical reasoning technique to the training data in order to learn (e.g., via machine learning) a statistical occlusion model, e.g., a model that determines how likely it is that an occluded entity is present in a field of view based on feature descriptions of entities that are visible in the field of view. In this case, the labels associated with the videos may help the AS104to learn features (e.g., via a machine learning technique) that are statistically likely to indicate the presence of an occluded entity.

In step208, the AS104applies a contextual modeling technique to the training data in order to learn a contextual occlusion model, e.g., a model that determines how likely it is that an occluded entity is present in a field of view based on contextual clues extracted from entities that are easily visible in the field of view (e.g., entities that often accompany the potentially occluded entity, and from which a presence of the occluded entity can be inferred). In this example, a relevant context from which clues can be extracted may be an appearance context, a social context, a color/texture/shape co-occurrence context, or the like. For instance, if the side of a boat is visible in a field of view, it may be inferred, from this context, that water is likely to be present nearby (e.g., perhaps just outside the field of view).

In step210, the AS104analyzes a plurality of different views of the same scene to learn how maneuvering of a drone improves the confidence in an occlusion estimation. In one example, the AS104analyzes the different views for motion patterns (e.g., encompassing a full range of pitch, yaw, and roll) that improve the confidence.

In step212, the AS104learns weights for the statistical occlusion model, the contextual occlusion model, and the motion patterns. In one example, the weights indicate the relative influences of the statistical occlusion model, the contextual occlusion model, and the drone maneuvering in accurately inferring the presence of an occluded entity in a field of view. The results of the statistical occlusion model, the contextual occlusion model, and the drone maneuvering, as applied to a given video, may be weighted and aggregated in order to generate a final score that indicates the likelihood of an occluded entity being present.

The method200ends in step214.

FIG. 3illustrates a flowchart of an example method300for inferring the presence of an occluded entity in a video captured via drone. In one example, the method300may be performed by an application server that subscribes to the output of one or more drones, e.g., AS104illustrated inFIG. 1. However, in other examples, the method300may be performed by another device. For instance, a UE108,110,112, or114that is in communication with a drone, or even a UE that is a drone could also perform all or some steps of the method200. As such, any references in the discussion of the method300to the AS104ofFIG. 1are not intended to limit the means by which the method300may be performed.

The method300begins in step302. In step304, the AS104obtains a video captured by a drone which may be monitoring a scene. The video may depict the scene from a particular field of view. The video may be obtained directly from the drone in real time (e.g., streaming from the drone as the frames of the video are captured, subject to any network delay, buffering, or the like).

In step306, the AS104applies a statistical occlusion model to the video to determine a first likelihood that there is an occluded entity present in the video. The statistical occlusion model may be a learned model that is trained using a set of labeled videos, some of which include occluded entities and some of which do not, as discussed above. In one example, the first likelihood that the video obtained in step306includes an occluded entity may be based (at least in part) on a statistical measure of a distance between features of the video and features of the set of labeled videos. In one example, the statistical measure may include one or more of a Euclidean distance, an inverse cosine metric, a Procrestes distance, a Cremer-Rao metric, or the like.

In step308, the AS104applies a contextual occlusion model to the video to determine a second likelihood that there is an occluded entity present in the video. In one example, the second likelihood that the video obtained in step306includes an occluded entity may be based (at least in part) on the visibility of other entities in the video that are contextually associated (e.g., socially, color/texture/shape co-occurrence-wise, or the like) with entities that are not seen.

In step310, the AS104combines the first likelihood and the second likelihood in order to infer an overall likelihood that an occlusion is present. In one example, the first likelihood may be assigned a first weight and the second likelihood may be assigned a second weight, so that the overall likelihood comprises a weighted combination of the first likelihood and the second likelihood. In one example, one or both of the first eight and the second weight may be zero.

In step312, the AS104determines whether it is likely that an occluded entity is present in the video, based on the overall likelihood. For example, the AS104may determine whether the overall likelihood is less than a threshold (e.g., less than fifty percent indicates not likely) or more than the threshold (e.g., more than fifty percent indicates likely). In one example, this threshold is adjustable.

If the AS104concludes in step312that an occluded entity is not likely to be present, then the method300returns to step304and continues to analyze the received video for occluded entities as new frames of the video are received.

If, however, the AS104concludes in step312that an occluded entity is likely to be present, then the AS104proceeds to step314. In step314, the AS104sends a signal to the drone from which the video was received instructing the drone to change its orientation. Changing the orientation of the drone will change the field of view that is captured in the video, and may thus allow a previously occluded entity to be more visible. In one example, the orientation is adjustable in up to three dimensions (e.g., including translation and rotation). The method300then returns to step304and re-computes the likelihood that an occluded entity is present in the video as new frames of the video are received from the new orientation of the drone (e.g., from the new field of view). Thus, steps304-314may be repeated any number of times until the occluded entity is no longer occluded, or until it is determined that an occluded entity is unlikely to be present.

Although not expressly specified above, one or more steps of the method200or the method300may include a storing, displaying and/or outputting step as required for a particular application. In other words, any data, records, fields, and/or intermediate results discussed in the method can be stored, displayed and/or outputted to another device as required for a particular application. Furthermore, operations, steps, or blocks inFIG. 2orFIG. 3that recite a determining operation or involve a decision do not necessarily require that both branches of the determining operation be practiced. In other words, one of the branches of the determining operation can be deemed as an optional step. Furthermore, operations, steps or blocks of the above described method(s) can be combined, separated, and/or performed in a different order from that described above, without departing from the examples of the present disclosure.

FIG. 4depicts a high-level block diagram of a computing device specifically programmed to perform the functions described herein. For example, any one or more components or devices illustrated inFIG. 1or described in connection with the method200or the method300may be implemented as the system400. For instance, a mobile device or an application server could be implemented as illustrated inFIG. 4.

As depicted inFIG. 4, the system400comprises a hardware processor element402, a memory404, a module405for inferring the presence of an occluded entity in a video captured via drone, and various input/output (I/O) devices406.

The hardware processor402may comprise, for example, a microprocessor, a central processing unit (CPU), or the like. The memory404may comprise, for example, random access memory (RAM), read only memory (ROM), a disk drive, an optical drive, a magnetic drive, and/or a Universal Serial Bus (USB) drive. The module405for inferring the presence of an occluded entity in a video captured via drone may include circuitry and/or logic for performing special purpose functions relating to monitoring, analyzing, and providing feedback relating to a drone's current field of view and potentially occluded entities therein. The input/output devices406may include, for example, a camera, a video camera, storage devices (including but not limited to, a tape drive, a floppy drive, a hard disk drive or a compact disk drive), a receiver, a transmitter, a speaker, a microphone, a transducer, a display, a speech synthesizer, a haptic device, a neurotransmitter, an output port, or a user input device (such as a keyboard, a keypad, a mouse, and the like).

Although only one processor element is shown, it should be noted that the general-purpose computer may employ a plurality of processor elements. Furthermore, although only one general-purpose computer is shown in the Figure, if the method(s) as discussed above is implemented in a distributed or parallel manner for a particular illustrative example, i.e., the steps of the above method(s) or the entire method(s) are implemented across multiple or parallel general-purpose computers, then the general-purpose computer of this Figure is intended to represent each of those multiple general-purpose computers. Furthermore, one or more hardware processors can be utilized in supporting a virtualized or shared computing environment. The virtualized computing environment may support one or more virtual machines representing computers, servers, or other computing devices. In such virtualized virtual machines, hardware components such as hardware processors and computer-readable storage devices may be virtualized or logically represented.

The processor executing the computer readable or software instructions relating to the above described method(s) can be perceived as a programmed processor or a specialized processor. As such, the present module405for inferring the presence of an occluded entity in a video captured via drone (including associated data structures) of the present disclosure can be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, e.g., volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device or diskette and the like. More specifically, the computer-readable storage device may comprise any physical devices that provide the ability to store information such as data and/or instructions to be accessed by a processor or a computing device such as a computer or an application server.

While various examples have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of a preferred example should not be limited by any of the above-described example examples, but should be defined only in accordance with the following claims and their equivalents.