Unmanned aerial vehicle with neural network for enhanced mission performance

An unmanned aerial vehicle (UAV) or “drone” executes a neural network to assist with inspection, surveillance, reporting, and other missions. The drone inspection neural network may monitor, in real time, the data stream from a plurality of onboard sensors during navigation to an asset along a preprogrammed flight path and/or during its mission (e.g., as it scans and inspects an asset).

FIELD OF THE INVENTION

The present invention relates, generally, to navigation of, and mission execution by, unmanned aerial vehicles (UAVs).

BACKGROUND

A UAV, commonly known as a drone or unmanned aerial system (UAS), and also referred to as a remotely piloted aircraft, is a flight vehicle without a human pilot aboard. Its path is controlled either autonomously by onboard computers or by the remote control of a pilot on the ground or in another vehicle. Drones have proliferated in number as recognition of their widespread and diverse commercial potential has increased.

Drones may make use of global positioning system (GPS) navigation functions, e.g., using GPS waypoints for navigation, and tend to follow preprogrammed flight paths. These may lead the drone to or around assets it will scan and inspect using onboard sensors. Drone systems may utilize a variety of onboard sensors (including one or more cameras, radiofrequency (RF) sensors, etc.) to monitor the operating environment, pre-calculate and follow a path to an asset to be inspected, and perform the inspection.

Despite the presence of these sensors, the drone may not be equipped to use the data they provide to react to unplanned changes in flight path (e.g., to avoid unexpected obstacles or perform collision-avoidance maneuvers), or adapt to GPS drift that can affect waypoint accuracy. GPS drifts occur when a preprogrammed flight path fails to account for GPS drift vector calibrations and corrections. When the operator defines a flight plan with waypoints in the absence of such corrections, for example, the navigation system may take the drone off-course; even a small deviation may take all or part of the asset of interest outside the preprogrammed flight path and, consequently, the field of view of the onboard sensors. Conventional drones may also fail to react to anomalies or unexpected conditions in the operating environment. The drone may cease collecting information, for example, if sensor readings fall below preset trigger thresholds, or may overcollect data if it veers off-course and begins recording sensor data before the target is actually reached.

More generally, when inspecting a structure such as an antenna, drones are typically controlled by an operator within the drone's line of sight. Not only does this require the presence of personnel at the site of each inspection, but demands close, sustained attention to the drone's flight. The drone must closely approach each region of the structure requiring inspection, and these regions may not be fully known until data is initially gathered and the inspection actually begins; yet the drone must also maintain a safe distance from the structure and steer around obstacles in its approach, notwithstanding wind and weather conditions. Particularly when inspecting a large installation (such as a power station) that includes many assets, the task of operating the drone safely and efficiently is a challenging one.

SUMMARY

Embodiments of the present invention permit a drone to navigate autonomously to an inspection site, execute a preliminary flight plan, and compute an inspection path that will take it close to regions of an asset requiring inspection, activating sensors (including one or more cameras) that both gather the relevant inspection data and indicate specific regions requiring close approach and imaging. The drone recognizes obstacles in its path and may monitor relevant weather conditions, altering both its flight path and approach patterns to avoid collisions.

In various embodiments, the drone includes a neural network that analyzes image frames captured in real time by an onboard camera as the drone travels. Neural networks are computer algorithms modeled loosely after the human brain and excel at recognizing patterns, learning nonlinear rules, and defining complex relationships among data. They can help drones navigate and provide mission support to ensure proper asset inspection without data overcollection. A drone in accordance herewith may execute a neural network to assist with inspection, surveillance, reporting, and other missions. The invention may make use of unsupervised “deep learning” neural networks executed onboard low-altitude inspection drones. Such a drone inspection neural network (“DINN”) may monitor, in real time, the data stream from a plurality of onboard sensors during navigation to an asset along a preprogrammed flight path and/or during its mission (e.g., as it scans and inspects an asset). This neural network may communicate with unmanned traffic-management systems, as well as with manned air traffic, to allow for safe and efficient drone operation within an airspace. Using a bidirectional connection to terrestrial and/or satellite-based communication networks, the DINN may request or receive real-time airspace change authorizations so it can adapt the drone flight path to account for airspace conflicts with other air traffic, terrain or obstacle conflicts, or to optimize the drone's flight path for more efficient mission execution. Importantly, the DINN can enable the drone to compensate for GPS drift or other course deviations, or unexpected target anomalies, by enabling target acquisition and locating all assets to be inspected.

Drone operation can be enhanced using high-altitude pseudosatellite (“HAPS”) platforms, also called a high-altitude, long-duration (“HALE”) platforms. These are unmanned aerial vehicles that operate persistently at high altitudes (of, e.g., at least 70,000 feet) and can be recharged by solar radiation during the day so they can remain in flight for prolonged periods of time to provide broad, satellite-like coverage of airspace. A HAPS drone equipped with RF communications payloads can offer vast areas of RF coverage—alone or in concert with existing communication satellite constellations or ground-based telecommunications networks, national airspace surveillance infrastructures, national airspace navigational aids, or individual air-traffic communication and surveillance systems—to offer connectivity and real-time communications and surveillance services to air traffic including drones.

HAPS platforms can be operated with less expense and greater flexibility than satellite constellations, which are not easily recalled for upgrades to meet changing bandwidth demands or augmented in number on short notice. In addition, satellites do not readily integrate with existing terrestrial air-traffic surveillance systems, making them less well suited than HAPS platforms for monitoring drone operation and maintaining the safe separation of drones and manned air traffic operating in the same airspace. Terrestrial alternatives such as telecommunication sites generally have short range and small areas of coverage, and once again, expanding coverage or capabilities is expensive and may not even be feasible due to features of the terrain or manmade structures.

A HAPS platform may execute a neural network (a “HAPSNN”) as it monitors air traffic; the neural network enables it to classify, predict, and resolve events in its airspace of coverage in real time as well as learn from new events that have never before been seen or detected. The HAPSNN-equipped HAPS platform may provide surveillance of nearly 100% of air traffic in its airspace of coverage, and the HAPSNN may process data received from a drone to facilitate safe and efficient drone operation within an airspace. The HAPSNN also enables bidirectional connection and real-time monitoring so drones can better execute their intended missions.

In various embodiments, the DINN cooperates with a HAPSNN. One application benefiting from such cooperation is pinpointing of passive intermodulation (PIM) on active telecommunication structures. PIM comes from two or more strong RF signals originating with transmitters sharing an antenna run, transmitters using adjacent antennas, or nearby towers with conflicting antenna patterns. PIM shows up as a set of unwanted signals created by the mixing of two or more strong RF signals in a nonlinear device, such as loose or corroded connectors, cables, duplexers, circulators, damaged antennas or nearby rusted members such as fences, barn roofs or bolts. Other sources include poorly terminated or damaged cables with a seam in the shielding, and aging lightning arrestors. PIM can be time-consuming and difficult to detect using traditional probing methods. A combination of a DINN and HAPSNN that provides connectivity to unmanned traffic management can enable the drone to fly around the telecommunications asset autonomously and unsupervised to detect, classify, and pinpoint PIM sources. If the drone detects novel or unexpected readings, it may be able to resolve and classify the nature of the readings based on its training. When the drone inspection is complete, the drone, using the DINN, may fly to the next preprogrammed asset location and adapt its flight path in real time along the way to optimize its operation in the airspace.

Accordingly, in a first aspect, the invention relates to a UAV comprising, in various embodiments, a flight package; a navigation system; an image-acquisition device; a communication facility; a computer memory; and a computer including a processor and electronically stored instructions, executable by the processor, for using data received from the image-acquisition device as input to a predictor that has been computationally trained to identify and classify objects appearing in images acquired by the image-acquisition device during flight.

In various embodiments, the predictor is a neural network. The UAV may include a database of actions, with the computer configured to select and cause execution of an action from the database in response to a detected object classified by the predictor.

The communication facility may be configured to interact with terrestrial and airborne control systems. In some embodiments, the UAV also includes a weather-surveillance module for monitoring weather conditions during drone flight; the computer includes data from the weather-surveillance module in selecting an action. The computer may be configured to cause the UAV to execute a preliminary flight plan around an asset to be inspected and, based on object classifications made by the predictor during the preliminary flight plan, compute and execute a revised flight plan around the asset. The computer may be further configured to communicate with a HAPS vehicle and, for example, to execute flight commands received from the HAPS vehicle, to communicate an altered flight to the HAPS vehicle for obtaining authorization from air-traffic control infrastructure, and/or to communicate a detected but unclassified object to the HAPS vehicle and receive, from the HAPS vehicle, a classification and associated action to be taken.

In another aspect, the invention pertains to a method of inspecting an asset using a UAV. In various embodiments, the method comprises the steps of acquiring digital images in real time during a flight of the UAV; computationally analyzing the acquired digital images with a predictor that has been computationally trained to identify and classify objects appearing in the images; and taking an action based on at least one classified object. The predictor may be a neural network, and the action may be determined based on database lookup in response to a detected object classified by the predictor. For example, the action may be altering a flight path of the drone.

In various embodiments, the method further comprises monitoring weather conditions during drone flight, and the action may be further based on the monitored weather conditions. The method may comprises acquiring signals from an asset to be inspected, and the action may be based on the acquired signals. The method may further comprise acquiring images of an asset to be inspected, and the action may be based on the acquired images. The method may further comprises the steps of causing the UAV to execute a preliminary flight plan around an asset to be inspected and, based on object classifications made by the predictor during the preliminary flight plan, compute and execute a revised flight plan around the asset.

In some embodiments, the method includes communicating with a HAPS vehicle, e.g., communicating an altered flight to the HAPS vehicle for obtaining authorization from air-traffic control infrastructure and/or communicating a detected but unclassified object to the HAPS vehicle and receiving, from the HAPS vehicle, a classification and associated action to be taken.

DETAILED DESCRIPTION

Refer first toFIGS.1-7, which illustrate the functions performed by a conventional HAPS platform105and the manner in which these functions may be enhanced through operation of a HAPSNN. InFIG.1, the HAPS platform105communicates with a plurality of cell towers representatively indicated at108, a plurality of aviation control systems representatively indicated at110, and a series of aircraft representatively indicated at113; all of these systems and vehicles are within the defined airspace115that the HAPS platform105monitors. The HAPS platform105may also communicate with a plurality of drones118in the airspace115, directly and/or via a cell tower. Finally, the HAPS platform105may communicate with satellites representatively indicated at120, e.g., for geolocation to maintain a substantially fixed position. In this way, the HAPS platform105obtains and updates a complete snapshot of the airspace115and monitors air traffic therein, serving as a communication hub among the various intercommunicating entities in the airspace115.

As seen inFIG.2, as the HAPS105monitors the position, state, and velocity of the drone118and manned air traffic133that operate in the monitored airspace115. The onboard HAPSNN recognizes that, given the operating altitude of the drone118, terrain or other obstacles205will block line-of-sight communications between the drone118and the aviation control system110and even the aircraft113. The obstacles205may be recognized and mapped in real time by the HAPS105, but more likely they will be stored in a map accessible locally to the HAPS105or via a wireless link. More specifically, the HAPSNN may compute a likelihood of the obstacles205interfering with current or future communications between the drone118and the aviation control system110; and if the likelihood exceeds a threshold, registering the need to establish a communication link bridging the drone118and the aviation control system110. In that event, the HAPS105responsively obtains the state (position, including altitude, and trajectory) of the drone118by any suitable modality or combination thereof, e.g., observation, telemetry, signal monitoring and/or direct communication with the drone118and/or terrestrial air traffic control systems that monitor its flight.

As a result of this recognized need, HAPS105may enter the communication network as an intermediate node or relay messages (i.e., act as a transmission link) between the drone118and the aviation control system110(e.g., UTM and LAANC) or other ground-based air-traffic surveillance infrastructure. In the absence of the HAPSNN, the HAPS105would have operated reactively—e.g., if the drone118had previously been communicating with the HAPS105and the control system110, the HAPS105could serve as a backup communication channel when direct communication between the drone118and the control system110is lost as the drone approaches the obstacle205. The HAPSNN facilitates proactive, predictive intercession by the HAPS105even if no prior communication between the drone118and control system110has taken place. Based on stored or acquired knowledge of the terrain and the locations of fixed communication features within the airspace115, as well as the computed trajectory of the drone118(which may have only just entered the airspace115), the HAPSNN recognizes the need for communication between the drone118and the control system110and causes the HAPS105to establish a wireless link with itself as the hub. Similarly, based on knowledge of the terrain and the monitored altitudes of the drone118and the manned aircraft113, the HAPSNN may cause the HAPS105to establish a wireless link between the drone118and the aircraft113with itself as the hub.

FIG.3illustrates a situation in which a drone118, transiting through an altitude-limited authorized airspace305, will encounter obstructions collectively indicated at310that it must fly around in order to remain within the airspace305. This deviation from a preprogrammed flight path may be considerable, depending on the extent of the obstructions, and may require course corrections that the drone is not equipped to handle with great accuracy—potentially leading it to miss all or part of the target when it arrives there. With the HAPS105in communication with the drone118, the HAPSNN predicts the need for additional airspace authorization and requests its extension to the region315. Once the authorization is obtained, the HAPS105communicates the relaxation of the altitude restriction to the drone118. In some embodiments, the HAPS105may compute a revised flight path320at higher altitude for the drone118, enabling it to remain on course for the target without any lateral deviation that could affect navigation accuracy.

Similarly, inFIG.4, the planned and authorized flight path405for the drone118would take it through a dangerous localized weather pattern, which the HAPS105detects with onboard radar or from real-time weather updates. The HAPSNN, aware of the drone's flight path and the weather condition, recognizes the need for an alternative flight segment410, which it or another onboard computational module computes. The HAPS105obtains authorization for the new flight path410and communicates it to the navigation system of the drone118.

FIG.5shows a drone118traveling through obstructions that prevent line-of-sight communication with cell towers108and a source505of National Airspace System navigational aid signals (such as VOR, VOR/DME, TACAN, etc.). The HAPSNN infers this condition from knowledge of terrestrial features within the airspace115and the state of the drone118. As a consequence, the HAPSNN causes the HAPS105at least to relay the signals from the blocked sources108,505or to establish a wireless link among the communicating entities108,118,505with itself as the hub.

InFIG.6, a drone605is flying in its authorized airspace610. A second drone615enters the airspace610, and the HAPSNN detects that the drones605,615are not communicating or cannot communicate; for example, one or both of the drones605,615may be “uncooperative,” i.e., not equipped to communicate with other drones and de-conflict flight paths. The HAPSNN may infer this based on communication, or the absence thereof, between the drones and/or with ground-based air-traffic surveillance and monitoring systems. The HAPSNN determines that, to avoid collision, the drone605should follow an alternative flight path620, which may be temporary until the drone605has passed the drone615. The HAPSNN or another onboard computational module computes the new flight path620. The HAPS105thereupon obtains authorization for the new flight path620and communicates it to the navigation system of the drone605.

With reference toFIG.7, a drone118may follow a flight path705within an authorized airspace corridor707to inspect or take data readings from a series of assets (e.g., a sequence of transformers7101. . .7105along a power line712) using, for example, an RFID reader. The HAPSNN monitors the data stream received by the drone118and, if the drone detects an anomaly, the HAPSNN may infer that the drone118will require a change to its airspace authorization; in particular, if the anomaly is associated with the transformer7103, the predicted authorization need may encompass the region720. The HAPSNN may cause the HAPS105to take action, e.g., relaying the information to appropriate UTM/LAANC systems or requesting a change to airspace authorization on behalf of the drone118, which will then be free to inspect the transformer7103more closely and record images of it for real-time or later evaluation.

FIG.8Aillustrates a representative HAPSNN architecture800, which includes the HAPS flight vehicle and various hardware and software elements. In general, a plurality of software subsystems, implemented as instructions stored in a computer memory, are executed by a conventional central processing unit (CPU)802. The CPU802may control the flight and operation of the HAPS vehicle as well as the functions described below, or these functions may be allocated among separate processors802. In addition, for efficient execution of neural-network functionality, the system may include a dedicated graphics processing unit. An operating system (such as, e.g., MICROSOFT WINDOWS, UNIX, LINUX, iOS, or ANDROID) provides low-level system functions, such as file management, resource allocation, and routing of messages from and to hardware devices (including at least one nonvolatile storage element803) and the software subsystems, which execute within a computer memory804. More generally, the HAPSNN800may include modules implemented in hardware, software, or a combination of both. For functions provided in software, programs may be written in any of a number of high level languages such as PYTHON, FORTRAN, PASCAL, JAVA, C, C++, C#, BASIC, various scripting languages, and/or HTML. The software modules responsible for operation of the HAPS vehicle, as well as the mechanical and flight features, are conventional and not illustrated; see, e.g., U.S. Pat. No. 10,618,654, the entire contents of which are hereby incorporated by reference.

The HAPSNN800includes a neural network module805, a transceiver module808, and a field-programmable gate array (FPGA)810. The modules transceiver module810and the FPGA810may constitute, or be part of, a communication facility configured to support airborne communication among flight vehicles and with terrestrial and satellite-based control infrastructure. The HAPSNN800may (but need not) operate in conjunction with drones that are equipped with a DINN812. The cloud neural network module805may be local to the HAPS vehicle but more typically operates in the cloud, i.e., on a remote (e.g., terrestrial) server in wireless communication with the HAPS vehicle as described below. The modules808,810are typically located on the HAPS vehicle itself.

The cloud neural network module805includes a classification neural network815that processes images and data received, via agile transceivers808, from a drone in real time, and which may be passed to the cloud neural network805. The classification neural network815has been trained using a database817of training images relevant to the missions that monitored drones will undertake. The classification neural network815processes and classifies received images and data and detects—i.e., computes the probability of—anomalies associated therewith. That is, an anomaly may be detected based on something unexpected in a received image or when considered alongside other drone telemetry; for example, an otherwise unexceptional image may trigger an anomaly detection when taken in conjunction with weather conditions reported by the drone. When an anomaly is detected, the classification neural network815may consult a classification database819to determine the proper response; that is, the database819includes records each specifying an anomaly and one or more associated actions that may be taken in sequence. If anomalies are detected that do not have a database record, the images may be transmitted for human inspection and classification. New classifications are then added to the training database817and used to retrain the neural network815. The resulting adjusted weights may be propagated, by the cloud server associated with the neural network805, back to the DINN812(if there is one) transmitting drone and other drones in the field with similar mission profiles. This procedure is described further below.

The agile transceiver package808includes Automatic Dependent Surveillance Broadcast (ADS-B), Traffic Collision Avoidance System (TCAS), Secondary Surveillance Radar (SSR), and Automatic Dependent Surveillance Rebroadcast (ADS-R) subsystems that operate at 978 MHz, 1090 MHz, and 1030 MHz for interrogations, responses, and rebroadcasts. These enable the HAPSNN800to “listen” to the positions of manned air traffic so the neural network815can computationally represent nearby traffic in 3D or 2D space and resolve any conflicts between drones and manned air traffic. This can be achieved by broadcasting the position of a drone to manned air traffic or the positions of manned air traffic to the drone. Emergency alerts may be issued to manned and/or unmanned traffic with instructions on which way to move to deconflict the airspace.

The agile transceivers808may include a cellular network package including 3G, 4G, LTE, 5G or any future telecommunication protocol and bandwidth to support communication links between drones operating in the airspace of the HAPSNN800, with the terrestrial telecommunications network that some UTM systems utilize, or with backhaul communications channels to transmit data from the HAPS to the cloud-based neural network. VHF and UHF transceiver (TX/RX) modules may be used to monitor navigational aids such as VORs, VOR/DMEs or TACANs that enable the neural network805to resolve the position of drones as well as of the HAPS using signal time of flight in the event GPS signal is lost. This also enables leveraging satellite communication constellations to transmit or receive data should the need arise. The drone virtual radar (DVR) data link facilitates communication with drone platforms that implement this technology (described, for example, in U.S. Pat. No. 10,586,462, the entire disclosure of which is hereby incorporated by reference) to send and receive air-traffic position information to help resolve conflicts or track drones. The neural network (NN) data link is a dedicated high-bandwidth backhaul channel that enables the HAPSNN800to communicate with DINN neural network compute engines825, transmitting real-time data received from a plurality of drones operating in the monitored airspace and receiving predictions and action instructions obtained from the classification database819. The FPGA810is employed as hardware accelerators to run software that tunes the transceivers808and filters out noise.

A representative DINN812, implemented in a drone118, includes a neural network compute engine825, a classification database825, and “back-end” code to perform various data-handling and processing functions as described below. In addition, the drone118includes a communication facility comprising or consisting of a set of agile transceivers808and an FPGA810, as detailed above. Also, the drone118may include a CPU802, storage803, a computer memory804.

As noted, although the DINN812may interact with a HAPSNN800, either can exist and operate on its own; that is, a HAPSNN is unnecessary for successful deployment and use of a DINN, while a HAPSNN may perform its surveillance and safety roles for flight vehicles lacking DINNs. The role of the DINN812is to enable the drone118to classify objects of interest on an asset it is inspecting (e.g., recognizing a cell tower to be inspected and a cracked antenna on such a tower), as well as obstacles that it will need to avoid during flight. The neural network825is configured to process and classify images received from an image-acquisition device827, e.g., a videocamera on the drone118. Hence, the neural network825may be a convolutional neural network (CNN) programmed to detect and recognize objects in the incoming images. These may be classified based on the neural network's training and the DINN812(e.g., the back-end code) may consult a classification database830to determine the proper response to a detected image. In this case, the database819includes records each specifying an object associated with some semantic meaning or action. For example, if the neural network825detects a tree in an incoming image, the corresponding database entry may identify a tree as an obstacle and trigger an avoidance maneuver that the drone's navigation system832executes by controlling the drone's steering and propulsion system. These are part of the drone's flight package835, which is conventional and therefore not shown in detail, but includes a power source, communications platform, the propulsion and steering systems, an autopilot system, etc.

The DINN812may also receive data from one or more surveillance systems837,839, which may include one or more of DVR, UTM, LAANC, ADS-B and TCAS systems. Although these may be implemented as part of the drone's communication platform, they are illustrated as conceptually within the DINN812since the neural network825may use this data in classifying an image. Similarly, while a weather surveillance system842would conventionally be implemented within the drone's communication platform, it is shown as part of the DINN812because, once again, weather conditions may be relevant to image classification or database lookup; as shown inFIG.4, the same visual scene may prompt different actions depending on the weather, e.g., the drone118may give buildings a wider berth under windy conditions.

In embodiments where the drone118interacts cooperatively with a HAPSNN800, the latter may provide further support and more powerful classification capabilities; for example, images with detected objects unclassifiable by the neural network825may be uploaded to the HAPSNN800for examination, and real-time instructions issued in return by the HAPSNN may be executed by the drone's navigation system832. Moreover, the HAPSNN800may update or supply different weight files for the neural network825in real time to better fit the drone's mission based on the classifications that are being made by that drone (and which are communicated to the HAPSNN800in real time). The neural network825responsively loads these new weight files when received.

This process is illustrated inFIG.8B. With reference also toFIG.8A, the DINN812processes incoming image frames in real time (e.g., ˜30 frames per second (FPS)) to enable the drone118to react fast enough to avoid collisions and to fly around the tower850. Accordingly, the drone118may include a graphics-processing unit (GPU) to support CNN operations. Real-time frame analysis allows the GPU to process the images and classify items on interest on an asset being inspected, notify the back-end code of the classification, and enable the back-end code to execute logic to react to the classification—generally by performing a look-up in the classification database830to obtain action corresponding to the classification.

The drone118transmits image data to the HAPSNN, which includes a high-precision CNN (in the compute engine815or even within the HAPS itself, if desired) capable of processing, for example, a 60 Megapixel (MP) photographic image each second. The CNN architecture is designed for speed and accuracy of classification by leveraging back-end logic that runs on the compute engine825. This back-end logic can change the CNN weight and configuration files based on the asset that is being classified based on the first few images of the asset captured by the drone. These preliminary images are collected as part of a “preliminary” flight path around the asset at a safe distance, and may be 60 MP or greater in resolution. These preliminary images are downscaled to the CNN's input image size (e.g., 224×224, or larger depending on the asset to be inspected), and pass through a sequence (of, e.g.,20) convolutional layers, followed by an average pooling layer, and a fully connected layer pre-trained to classify different assets (e.g., 100 types of assets). Once the type of asset is identified, the weights and configuration files may be changed and more (e.g., four) convolutional layers are added followed by two fully connected layers to output probabilities and bounding boxes of objects or areas of interest that may be present on the asset. The images uploaded from the drone may be increased in size (e.g., to 448×448) as this type of classification requires more granular detail to be present. The degree of size increase may be dynamically controlled, e.g., scaled up if sufficient detail is not detected for reliable classification.

The fully connected layers predict the class probabilities and bounding boxes (i.e. cracked antenna, rust and corrosion, etc.). As an example, the final layer may use linear activation whereas the convolutional layers may use leaky ReLu activation.

Once the back-end logic of the compute engine825detects the presence of class and bounding box coordinates, it may switch to and trigger a centroid tracker function to bring that specific classification into the center of the field of view of the drone's image-acquisition device. The back-end logic cooperates with a ranging compute engine to resolve the safest flight path for the drone to approach and position the image-acquisition device for high-resolution scans.

Accordingly, the preliminary flight path establishes the type of asset in view and registers the position of the asset in 3D space relative to the drone to account for any GPS drift vectors. If there are any obstacles or hazards present in the operating area they are classified and their position in 3D space is registered. The centroid tracker is activated once a classification in area of interest is detected and keeps the object of interest centered in the field of view. The ranging compute engine controls forward and backwards movement of the drone. Before any of these commands are executed, the position of the drone in 3D space relative to the asset and any obstacles present in the operating area is obtained. This data runs through back-end logic that resolves a safe GPS waypoint flight path that will bring the drone to the area of interest—in GPS-denied areas, this flight path can still be resolved and executed using Kalman filtering of inertial data in conjunction with centroid and ranging functionality. The flight path is fine-tuned in real time via the centroid tracker and ranging compute engine. It should be noted that the centroid tracker can be run by a HAPSNN800rather than the DINN812.

In step855, the HAPSNN CNN (“HP-CNN”) processes each image to detect objects therein using a standard object-detection routine (e.g., YOLO), and attempts to identify (i.e., classify) all detected objects based on its prior training (discussed further below). Detected objects that cannot be identified are stored in a database and made available to personnel for identification and labeling (step857). The HP-CNN is then retrained on an augmented dataset including the newly identified and labeled object data (step859), resulting in generation of new CNN weights (step862). If the real-time neural network825resident on the drone118is also a CNN (“RT-CNN”), the HAPSNN800may push these weights to the RT-CNN, which receives and loads them. That is, the HP-CNN and RT-CNN may be identical or substantially similar so that CNN weights generated for the HP-CNN may be propagated across a fleet of drones.

The HP-CNN (or, in some embodiments, an RT-CNN on its own) may be trained in a conventional fashion. In particular, the CNN is trained on labeled images of objects likely to be encountered by a drone as it executes its missions, yielding a CNN capable of analyzing and classifying the objects most likely to be encountered by drones in their typical flight paths. Because no training set can be exhaustive and drones will inevitably encounter unknown objects during use, the above-described process of spotting unrecognized objects, storing them for manual labeling, and thereafter retraining the CNN on the augmented dataset helps minimize the risk of mishap by constantly enriching the drones' visual vocabularies and action repertoires. Although this is most efficiently accomplished using a HAPSNN as a central training hub that receives unclassifiable objects from many drones and can keep all of their neural networks updated to reflect the latest classification capabilities, it is nonetheless possible to implement this training and retraining function on individual DINNs.

FIGS.9A-13Eillustrate various drone applications and the manner in which a DINN may be used to control and simplify drone operation. InFIG.9A, the DINN812guides the drone118around an antenna900to be inspected in accordance with a flight pattern910that may change in real time as the drone118detects anomalies or structures requiring closer inspection. For example, as shown inFIG.9B, the flight path910may be altered to keep the drone118clear of power lines915, which will have been recognized as an obstacle.

With reference toFIG.10, an antenna1000may have a crack1005, which is identified by a DINN-equipped drone. The back end may execute an openCV function to fix a centroid on the antenna1000and move the drone to center the antenna in the field of view to facilitate image acquisition. This can be coupled with a depth map generated by stereo RGB cameras of known sensor size, lens properties, and camera focal point separation to position the drone118close enough to the antenna for the camera to resolve sufficient detail to pick up the cracks well. Other faults that may be recognized by the drone118include structural corrosion1010and a paint coating anomaly1015. The drone118payload may include additional sensors, such as an EM sensor and an infrared camera, and the DINN812may include additional modules such as a spectral analysis and/or electromagnetic compute engine. These enable the drone118to detect a PIM signature at a coax connector1025and other EM anomalies such as magnetic fields emanating from a lightning arrestor1030(indicating the possibility of arcing), as well as an abnormal heat signature1035. These conditions may be diagnosed as well as observed by the drone118or using onboard compute engines and/or in cooperation with a HAPSNN800as described above. The DINN812may further include a ranging compute engine to run data fusion between a depth map obtained as discussed above and laser rangefinder/radar/ultrasonic or any other ranging sensors that the drone payload may contain to derive the most accurate range readings; a computer vision compute engine to perform image-processing and analysis functions (such as centroid placement, as noted above); and/or an infrared and multispectral compute engine to analyze infrared and other images obtained at wavelengths outside the visible spectrum.

As shown inFIG.11, if the drone118detects an unusual EM signature from an asset1100—such as abnormal non-vertical magnetic field lines1105, whereas under normal operation the field lines are vertical as indicated at1110—it may record the anomaly or, in embodiments where the drone118is in communication with a HAPSNN, report it to the HAPSNN, which may autonomously summon a drone with a more specialized inspection payload to better diagnose the condition. The drone118may also communicate or otherwise interact with an asset to be inspected. InFIG.12, a drone118reads data stored in a transformer1200indicating an abnormality. That is, the transformer1200has self-diagnostic capability and, when an operating anomaly was earlier detected, data indicative of the detection was stored internally. If the transformer1200is inspected regularly by a drone118and the anomalous condition does not require urgent attention, it may not be necessary for the transformer to communicate its presence to supervisory personnel upon detection. Rather, when the drone118makes an inspection flight, it can interrogate the memories of all transformers and thereby acquire previously stored data indicating transformer anomaly detections. This may prompt the drone118to perform a closer inspection (as indicated in the figure) in response to the classification of the condition and database lookup. Once again, a HAPSNN may recognize the condition and send new weights to the DINN of the drone118, enabling it to make condition-specific classifications as it inspects the transformer1200.

FIGS.13A-13Eshow how a DINN812can establish an optimal flight path through a complex inspection environment.FIGS.13A and13Billustrate a power distribution substation1300including a series of outgoing transmission lines collectively indicated at1310, a voltage regulator1315, a stepdown transformer1320, a series of switches1325, one or more lightning arrestors1330, and incoming transmission lines collectively indicated at1335. Any of these components can develop anomalies or malfunction, and all may be inspected by a drone118. As shown inFIGS.13C and13D, the drone118may execute a preliminary flight pattern1350to gather images for analysis by the DINN, alone or in concert with a HAPSNN. The DINN812analyzes images acquired by the drone's onboard camera and classifies the various components1310-1335. Based on these classifications, the navigation module832(seeFIG.8) or back-end code computes an optimized flight plan1355that permits all of the components1310-1335to be properly and efficiently inspected by the drone118. In greater detail, each component present in the substation1300is classified during the preliminary inspection flight path1350, which is farther away from the substation1300to accommodate GPS drift vectors and other unknowns relating to obstacles or inaccuracies about the initial flight path. Once a component is classified, its position in the image, as well as that of the drone, is registered. This process repeats multiple times throughout the preliminary inspection and enables the back-end code to triangulate and position each of the classified assets in a 3D space so a more precise inspection flight path, which will bring the drone and payload closer to the assets, can be calculated. This new flight plan1355is then executed and again the DINN812classifies assets as the flight path is executed. In the preliminary inspection, the drone is farther away so the DINN812can only classify large items as the resolution of the image-acquisition device(s) is fixed. Once the closer flight path1355is executed, more asset detail will be detected, enabling the DINN812to classify new items and adjust the path of the drone again as needed. The ranging compute engine calculates the closest allowable approach distance between drone and an asset consistent with an acceptable safety margin.

Still another application in which a complex inspection operation may be optimized by a DINN is illustrated inFIG.14. Here the objective is to detect and classify the “rad center”—i.e., the center of radiation1410R,1415R,1420Rof each of the antennas1410,1415,1420, which are mounted to the telecommunication structure1400at different elevations. A DINN can classify a rad center and then identify, for each antenna, a centroid to bring the rad center into the center of the field of view of the drone camera so that a barometer reading (elevation) can be recorded. The navigation module832can then use this elevation to generate an orbit flight path that enables the drone118to obtain 3D model data for reconstruction of the antenna structure.