Image-based anomaly detection for aerial inspection using self-supervised learning

A method of automatically detecting anomaly from aerial images of an object of interest is provided. The method may include generating a data coding model corresponding to a category of assets by training a neural network with a training set of digital images depicting an asset in a state that is free from anomalies. The method may further include receiving a target digital image depicting a target asset, and reconstructing the target digital image using the data coding model to generate a decoded target digital image associated with the state that is free from anomalies. The data coding model may be self-supervised to learn to reconstruct itself to an anomaly-free state. The method may also include comparing the target digital image to the decoded target digital image to generate a difference map and, in response to a determination that the difference map depicts any anomaly, generating anomaly alert data.

FIELD OF THE DISCLOSURE

This disclosure is related to image-based anomaly detection using an aircraft such as an unmanned aerial vehicle (UAV) for inspection, survey and surveillance, and in particular to image-based anomaly detection using self-supervised learning models.

BACKGROUND

An important task for aerial inspection, covering a wide geographical area, is detecting when a target structure, i.e., an asset, includes any anomalies. For example, when inspecting a fence, an open gate or a broken portion of the fence may be considered an anomaly. Likewise, when inspecting an enclosed structure, such as a power substation, an animal having become trapped inside may also be considered an anomaly. Generally, an inspection anomaly may be anything detected during inspection that deviates from what is standard, normal, or expected. In wide geographical areas, anomalies may be rare, however, they may lead to significant problems if not attended to promptly. Conducting regular on-site inspections may be costly and inefficient.

Overhead inspection through aerial photographs may significantly ease the costs of regular inspection. For example, drone aircraft may be used to obtain aerial photos of individual assets. In order to detect an anomaly, three types of existing solutions may be used. First, an image change-based approach may be used to compare a current image of the asset to a previous image of the asset. Inherent in this process is the requirement that a previous image exist. This may not be the case for new assets. Further, it can be inconvenient and potentially problematic to store both previous images and new images of the assets. Also, using an image change-based approach may not be reliable in detecting an anomaly if the anomaly already existed in the prior image.

A second solution includes a feature-based classification approach. In this approach, certain features may be associated with a particular type of anomaly. The images may be analyzed to detect the features, and thereby find an anomaly. However, in this approach, each category of anomalies being searched from usually has a corresponding database of features associated with the category of anomalies. Further, only a finite list of known and predetermined anomalies may be detected. New types of anomalies, or anomalies having different visual features may pass undetected.

A third solution includes an outlier detection approach. Outlier detection has been used for intrusion detection in analyzing access data and fraud detection in analyzing financial data. However, a statistical outlier analysis does not apply to image data because image data has many more dimensions than access data or financial data. Further, each image analyzed is unique, making it difficult to define any kind of outlier.

Therefore, typical anomaly detection systems are not sufficient to efficiently and inexpensively detect anomalies without historical imagery and without pre-assumptions. Other disadvantages may exist.

SUMMARY

Disclosed is an anomaly detection system that overcomes at least one of the described disadvantages of the typical anomaly detection systems. The system may create a generative model associated with an anomaly free-state of a category of assets. The generative model may be used to deconstruct a digital image of a target asset and reconstruct a new image that resembles the expected normal state of the target asset. By comparing the reconstructed image and the original image, any anomaly may be detected. Such detection may be performed without any previous images of the target asset.

In a typical anomaly detection system, a neural network may be trained to identify particular anomalies in a feature-based classification approach. However, the disclosed system fundamentally changes the underlying functionality of the typical neural network-based approaches by developing and using a data coding model that corresponds to a “normal,” or anomaly free, state instead of corresponding to one or more particular anomalies. Thus, as the neural network of the disclosed system learns to reconstruct an image, it does so by encoding image data to a block of code, then decoding the block of code to a new image that matches the “normal” state, and eliminating features associated with an anomaly instead of attempting to identify and reconstruct the anomalous features themselves. The learning of the data coding model, for example an autoencoder, does not require additional information to learn what to output, it learns to encode, decode and output itself, therefore it's unsupervised, or namely self-supervised machine learning.

In an embodiment, a method includes receiving a set of digital images, each digital image of the set of digital images depicting a distinct asset corresponding to a category of assets, and each digital image depicting the distinct asset in a state that is free from anomalies associated with the category of assets. The method further includes generating a data coding model corresponding to the category of assets by training a neural network with the set of digital images, the data coding model associated with the state that is free from anomalies. The method also includes receiving a target digital image depicting a target asset corresponding to the category of assets. The method includes generating encoded target data by transforming the target digital image using the data coding model. The method further includes generating a decoded target digital image by further transforming the encoded target data using the data coding model, the decoded target digital image associated with the state that is free from anomalies. The method includes comparing the target digital image to the decoded target digital image to generate a difference map. The method further includes in response to a determination that the difference map depicts an anomaly, generating anomaly alert data.

In some embodiments, the neural network is an autoencoder, where generating the encoded target data includes encoding the target digital image using one or more encoding layers of the autoencoder, and where generating the decoded target digital image includes decoding the encoded target data using one or more decoding layers of the autoencoder. In some embodiments, the autoencoder is a deep autoencoder having multiple encoding layers and multiple decoding layers, a convolutional autoencoder, or both.

In some embodiments, the method includes the anomaly alert data includes an annotated digital image depicting the target asset. In some embodiments, the method includes sending the anomaly alert data to a user output device, the user output device including a display device, a speaker device, or both. In some embodiments, the method includes sending the anomaly alert data to a local, or remote database, or both.

In some embodiments, the method includes generating a set of decoded digital images corresponding to the set of digital images using the data coding model, comparing the set of decoded digital images to the set of digital images to generate a set of difference maps, converting the set of difference maps to a binary classification model, and applying the binary classification model to the difference map to categorize the difference map as depicting any anomaly or not depicting any anomaly. In some embodiments, the binary classification model is a support vector machine model.

In some embodiments, the target digital image includes an aerial or satellite photo of the target asset, where the target asset is a physical structure, and where the category of assets includes other assets having a shape, layout, or function that corresponds to a shape, layout, or function of the target asset.

In some embodiments, the method includes generating additional data coding models corresponding to additional categories of assets by training additional neural networks with additional sets of digital images, the additional data coding models associated with additional states that are free from anomalies associated with the additional categories of assets. In some embodiments, the method includes receiving additional target digital images depicting additional target assets corresponding to the additional categories of assets, generating additional encoded target data by encoding the additional target digital images using the additional data coding models, generating additional decoded target digital images by decoding the additional encoded target data using the additional data coding models, comparing the additional target digital images to the additional decoded target digital images to generate additional difference maps, and in response to a determination that at least one of the additional difference maps depicts any anomaly, generating anomaly alert data.

In some embodiments, the method includes altering one or more digital images of the set of digital images to remove a feature that does not correspond to any anomaly associated with the category of assets.

In an embodiment, a system includes one or more processors and one or more memories, the one or more memories including instructions that, when executed by the one or more processors, cause the one or more processors to perform or initiate operations. The instructions may cause the processor to receive a data coding model corresponding to a category of assets, the data coding model associated with a state that is free from anomalies associated with the category of assets. The instructions may further cause the processor to receive a target digital image depicting a target asset corresponding to the category of assets. The instructions may further cause the processor to generate encoded target data by encoding the target digital image using the data coding model. The instructions may further cause the processor to generate a decoded target digital image by decoding the encoded target data using the data coding model. The instructions may further cause the processor to compare the target digital image to the decoded target digital image to generate a difference map. The instructions may also cause the processor to, in response to a determination that the difference map depicts any anomaly, generate anomaly alert data.

In some embodiments, instructions cause the one or more processors to receive a set of digital images, each digital image of the set of digital images depicting a distinct asset corresponding to the category of assets, and each digital image depicting the distinct asset in the state that is free from anomalies associated with the category of assets, and generate the data coding model by training a neural network with the set of digital images.

In some embodiments, the neural network is an autoencoder, where generating the encoded target data includes encoding the target digital image using one or more encoding layers of the autoencoder, and where generating the decoded target digital image includes decoding the encoded target data using one or more decoding layers of the autoencoder, and where the autoencoder is a deep autoencoder having multiple encoding layers and decoding layers, a convolutional autoencoder, or both.

In some embodiments, the instructions cause the one or more processors to generate a set of decoded digital images corresponding to the set of digital images using the data coding model, compare the set of decoded digital images to the set of digital images to generate a set of difference maps, convert the set of difference maps to a binary classification model, and apply the binary classification model to the difference map to categorize the difference map as depicting any anomaly or not depicting any anomaly. In some embodiments, one or more digital images of the set of digital images is artificially altered to remove a feature that does not correspond to any anomaly associated with the category of assets.

In some embodiments, the system includes a user output device, the user output device including a display device, a speaker device, or both, where the instructions cause the at least one processor to send the anomaly alert data to the user output device. In some embodiments, the method includes sending the anomaly alert data to a local, or remote database, or both. In some embodiments, the target digital image includes an aerial or satellite photo of the target asset, where the target asset is a physical structure, and where the category of assets includes other assets having a shape, layout, or function that corresponds to a shape, layout, or function of the target asset.

In an embodiment, a method includes receiving a target digital image depicting a target asset corresponding to a category of assets. The method further includes generating encoded target data by encoding the target digital image using a data coding model corresponding to the category of assets, the data coding model associated with a state that is free from anomalies associated with the category of assets. The method also includes generating a decoded target digital image by decoding the encoded target data using the data coding model. The method includes comparing the target digital image to the decoded target digital image to generate a difference map.

In some embodiments, the method includes receiving a set of digital images, each digital image of the set of digital images depicting a distinct asset corresponding to the category of assets, and each digital image depicting the distinct asset in the state that is free from anomalies associated with the category of assets, and generating the data coding model by training a neural network with the set of digital images.

DETAILED DESCRIPTION

Referring toFIG. 1, an embodiment of a system100for anomaly detection is depicted. The system100may include one or more processors102and one or more memories104coupled to the one or more processors102. The one or more memories104may store instructions106to be executed by the one or more processors102. For example, the instructions106, when executed by the one or more processors102may initiate or perform any of the operations described herein. The one or more processors102and the one or more memories104may include a single processor and a single memory at a personal computing device or, alternatively, may include multiple processors and multiple memories distributed within a network (e.g., a local area network, a wide area network, the internet, etc.). A distributed example of the system100is further described within the context ofFIG. 2.

In some embodiments, the one or more processors102may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), a peripheral interface controller (PIC), another type of microprocessor, and/or combinations thereof. The one or more processors102may be implemented as integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuit (ASICs), combinations of logic gate circuitry, other types of digital or analog electrical design components, or combinations thereof. The one or more memories104may include a memory such as random-access memory (RAM), read only memory (ROM), magnetic disk memory, optical disk memory, flash memory, another type of memory capable of storing data and processor instructions, or the like, or combinations thereof. In some embodiments, the one or more memories104, or portions thereof, may be located externally or remotely from the one or more processors102.

In some embodiments, the system100includes a neural network108. Although depicted as distinct and separate, the neural network108may be implemented using the one or more processors102and the one or more memories104. Alternatively, the neural network108may be implemented remotely. In some embodiments, the neural network108may include an autoencoder having one or more encoding layers and one or more decoding layers. The autoencoder may be a deep autoencoder having multiple encoding layers and multiple decoding layers. Further, the autoencoder may be a convolutional autoencoder, relying on a convolution operation to perform transformations of input data. Other types of neural networks may also be used.

In some embodiments, the system100operates in two phases: first, using the neural network108for training data coding model170, and second, using the neural network108to load the learned data coding model170for anomaly detection. Using the neural network108enables the system100to apply the data coding model170to reconstruct images that fall within a same category of images (e.g., having some features or shapes in common) as described further herein, despite differences that may exist between the images themselves. This technical benefit also enables the system100to operate on images of a target asset without having any prior images of the target asset, as explained in detail in the following paragraphs. In order to train the neural network108, the one or more processors102may first receive a set of digital images110.

The set of digital images110may depict distinct assets112corresponding to a category114of assets. For example, the assets112may correspond to a physical structure having a particular shape, layout, or function. The category114of assets may include all assets that have the same or a similar shape, layout, or function. For example, in some cases the asset112may include a fence enclosure having a rectangular layout and the category114of assets may include any fence enclosure having the same rectangular layout. As another example, the asset112may include a vent structure having a particular shape and the category114of assets may include all vent structures having the same particular shape. In yet another example, the asset112may include a barrier that operates as a gate and the category114of assets may include all barriers that operate as a gate. Other properties (e.g., structures, objects, commodities, goods, or stock) may be used to determine a category of assets. All the assets112received at the processor102may correspond to the category114of assets. In some embodiments, the set of digital images110may include aerial or satellite photos of the assets112.

As an illustrative example, the set of digital images110may include a set of aerial photos. The category114of assets may correspond to a type of utility structure (e.g., power substations, gas vents, electric transmission poles, oil rigs, etc.). Each type of utility structure may have a similar shape, layout, or function and all of the assets112depicted in the set of digital images110may have the shape, layout, or function associated with the category114. A substantial portion of the system100may be associated with the category114of assets as denoted by the dotted line118. The system100may then be replicated in order to detect anomalies associated with additional categories164of assets.

The category114of assets may be associated with various anomalies. For example, with respect to a power substation, it may be possible for an animal to get within a fence surrounding the substation. The animal would be a visible anomaly in an aerial photo. Another example of an anomaly may be an open gate, where normally the gate should be closed. As explained herein, typical anomaly detection system may attempt to identify a particular anomaly from a predetermined list of anomaly categories. However, given the high number of different possible anomalies and the many different ways in which any anomaly can present itself in aerial photos, it may not be practical to use such a system. As such, rather than focusing on the individual anomalies associated with the category114of assets, the system100focuses on a state116of the categories of assets that is free from anomalies.

Each of the digital images110may be associated with the state116that is free from anomalies. The one or more processors102may use the set of digital images to train the neural network108and thereby generate a data coding model120. As such the data coding model120may also be associate with the state116of the category114of assets that is free of anomalies. This means that using the neural network108in conjunction with the data coding model120to encode and then subsequently decode, or reconstruct, any input image will result in an attempt at reconstructing the input image without any anomalies. Any digital images that are not free of anomalies may be filtered out from the digital images110, in order to ensure accurate training.

After the neural network108is trained, and the data coding model120is generated, it may be desirable for the system100to further have the capability to distinguish between actual anomalies and noise. As such, the set of digital images110may be encoded using the neural network108and the data coding model120and may subsequently be decoded in order to generate a set of decoded digital images122depicting the assets112. Because the set of digital images110depicts the assets112in the state116that is free from anomalies, and because the data coding model120is configured to reconstruct images in the state116that is free from anomalies, any differences between the set of digital images110and the set of decoded digital images122may be considered as noise, or otherwise inconsequential to the determination of anomalies. As such, the set of digital images110and the set of decoded digital images122may be compared to generate a set of difference maps124, representing the noise.

In some embodiments, the set of difference maps124are converted into a binary classification model126. The binary classification model may be usable to categorize generated difference maps as depicting an anomaly or not depicting an anomaly, which is further described in relation to the second phase of using the system100to identify anomalies. In some embodiments, the binary classification model126may be a support vector machine model. Other types of binary classification models may be used.

In some embodiments, during the second phase of operation, the system100is used to detect an anomaly associated with the category114of assets. As an initial operation, the one or more processors102may receive a target digital image128depicting a target asset130. The target digital image may be an aerial or satellite photo of the target asset130.

In some embodiments, in order to determine whether an anomaly exists with respect to target asset130, the target digital image128is transformed, or encoded, using the data coding model120to generate encoded target data132. After the encoded target data132has been generated, the one or more processors102may generate a decoded target digital image134by further transforming, or decoding, the encoded target data132using the data coding model120. And the encoded target data132may be further transformed to the decoded target digital image134using the data coding model120. Because the data coding model120is a generative representation of the set of digital images110that depict the assets112in the state116that is free from anomalies, the decoded target digital image134may be transformed from the encoded target data132and associated with the state116that is free from anomalies. This means that the decoded target digital image134is a reconstructed version of the target digital image128. It has been transformed using the neural network108and the data coding model120that learned to ignore any anomalies that may exist in the target digital image128. The one or more processors102may then compare the target digital image128to the decoded target digital image134to generate a difference map136.

In some embodiments, in order to ensure that the any potential anomalies shown on difference map136are not due to noise, the binary classification model126is applied to the difference map136. The difference map136may then by classified as either depicting an anomaly or not depicting any anomaly. In response to a determination that the difference map136depicts an anomaly, the one or more processors102may generate anomaly alert data138. The anomaly alert data138may include an annotated digital image140depicting the target asset130with an annotation calling attention to any anomalies.

The system100may further include a user output device142. The user output device142may include a display device144and a speaker device146. The display device144may be used to output the anomaly alert data138to a user. The system100may further include a database148. The anomaly alert data138may be stored at the database148, which may be a remote or a local database.

The system100may also detect anomalies associated with additional categories164of assets by replicating the operations described herein for those additional categories164. For example, additional sets of digital images160may be received by the one or more processors102. The additional sets of digital images160may be used to train additional neural networks158to generate additional data coding models170. The additional data coding models170may correspond to additional categories164of assets.

In addition to the additional data coding models170, additional binary classification models176may also be generated by using the additional neural networks158and the additional data coding models170to encode and decode the additional sets of digital images160to generate additional sets of decoded digital images172. The additional sets of decoded digital images172may be compared to the additional sets of digital images160to generate additional difference maps174. The additional difference maps174may be converted into the additional binary classification models176.

The one or more processors102may receive additional target digital images178depicting additional target assets (not shown inFIG. 1) corresponding to the additional categories164of assets. Additional encoded target data182may be generated by encoding the additional target digital images178using the additional data coding models170. The one or more processors102may then decode the additional encoded target data182using the additional data coding models170to generate additional decoded target digital images184. The additional decoded target digital images184may be compared to the additional target digital images178to generate additional difference maps186. In response to a determination that at least one of the additional difference maps186depicts any anomaly the anomaly alert data138may be generated to alert a user of the anomalies.

Because the system100relies on the neural network108to determine whether an anomaly exists, a benefit of the system100is that an anomaly may be detected in a target digital image128of a target asset130without relying on previously taken images of the target asset130. For example, in an image change-based approach, described in the Background section, an anomaly may detected by comparing a new image of the target asset130to a previously taken image of the target asset130. In contrast, because the data coding model120“knows” what the target asset130should look like, based on its training, it does not need any previously taken images of the target asset130, even in cases where the set of digital images110used to train the data coding model120does not include any images of the target asset130. As an example,FIG. 10depicts an example of a digital image progression1000in which a target digital image1028has been reconstructed without any previous non-anomalous images. The target digital image1028may include an anomaly1032within a depicted target asset1030. The data coding model120may not have been trained with any images of the target asset1030, yet the system100is able to generate a decoded target digital image1034free from the anomaly1032. A difference map1036may then be generated that depicts the anomaly1032. Further, the system100uses a data coding model120that is associated with an anomaly free state to fundamentally change the functionality of the neural network108, as compared to typical anomaly detection systems that would use the neural network108to identify or recreate the anomalies themselves instead of the assets112in the state116that is free of anomalies. Other advantages may exist.

FIG. 2depicts an embodiment of a system200that applies a multiphase approach for anomaly detection. For example, the operations described with reference toFIG. 1may be performed in a training operation202and a prediction operation204. In some embodiments, the system200includes a neural network108, a training portion208of a local system, and a prediction portion220of the local system. The neural network108, the training portion208, and the prediction portion220may all correspond to, and be implemented using, the one or more processors102ofFIG. 1.

In some embodiments, during operation, a set of digital images110may be received by the neural network108. The set of digital images110may depict distinct assets corresponding to a category of assets. Further, each of the digital images110may be associated with a state that is free from anomalies.

In some embodiments, the neural network108includes a machine learning engine206to produce a data coding model120, such as an autoencoder model. The machine learning engine206may train the data coding model120using the set of digital images110. As such the data coding model120may also be associate with the state of the category of assets that is free of anomalies. This means that using the neural network108to encode and then subsequently decode, or reconstruct, any input image will result in an attempt at reconstructing the input image without any anomalies.

In some embodiments, a separated training portion208receives negative samples216and positive samples218that will use the data coding model120to generate reconstructed images. The negative samples216may correspond to samples that correspond to a normal state of a depicted asset. That is, the negative samples may depict the asset in the state that is free from anomalies. The positive samples218may correspond to samples that correspond to an anomalous state of the depicted asset. That is, the positive samples may depict the asset as having an anomaly (e.g., an animal within a usually fenced region, an open gate, another type of anomaly, etc.). In some cases, additional steps may be performed to diagnose the anomaly to determine what specific anomaly exists. Generally, an anomaly may include any condition where an asset that should remain in a constant state has some sort of change.

Some examples of anomalies that may be detected include a vehicle present at a target asset, the absence of a vehicle that should always be present at a target asset, animals within a fence, fallen poles or other fallen structures, open gates, land erosion, flooding. An open gate may be an anomaly at a fenced enclosure, where it is typical for all enclosures in the category of fence enclosures to have their gates closed.

In some embodiments, in addition to training the data coding model120, the training portion208of the local system further train a classification model, such as a binary classification model126that classifies images as depicting an anomaly or not depicting an anomaly. In order to generate the binary classification model126, the training portion208of the local system may receive a set of decoded digital images122, or reconstructed digital images. A change detect algorithm210may be applied to the set of decoded digital images122in order to generate a set of difference maps124. The set of difference maps may be fed into a local machine learning engine212, which may use a support vector machine214to generate the binary classification model126. Once both the data coding model120and the binary classification model126are generated and trained, they may be used during the prediction operation204to detect anomalies.

In some embodiments, the prediction portion220of the local system includes a local server222, which corresponds to at least one of the one or more processors102ofFIG. 1. The local server222may be configured to receive the data coding model120from the neural network108. Alternatively, the local server222may access the data coding model120remotely. The local server222may further be configured to receive a target digital image128depicting a target asset. The local server222may use the data coding model120to generate encoded target data and then generate a decoded target digital image134by decoding the encoded target data using the data coding model120. The local server222may compare the target digital image128to the decoded target digital image134to generate a difference map136. In response to a determination that the difference map136depicts any anomaly, the local server222may generate anomaly alert data that includes an annotated digital image140. For example, the annotated digital image140may include an annotation250showing a position of a detected anomaly. The annotation250may help a user locate an anomaly within the annotated digital image140.

A benefit of the system200is that the data coding model120may be generated and trained and subsequently shared with a local server222. Thus, the local server222does not need access to any training data, such as the set of digital images110in order to determine whether an anomaly exists. As such, a previous image of the target asset depicted in the target digital image128may not be needed. Other advantages may exist.

Referring toFIG. 3, a digital image progression300is depicted. The digital image progression300may correspond to a process that uses a binary classification model to determine the existence of anomaly, such as the binary classification model126. The first image is a digital image302depicting an asset312. The digital image302may correspond to a single image taken from the set of digital images110and the asset may correspond to a single asset taken from the assets112. In particular, the digital image302may depict the asset312in a state that is free from anomalies, however contains noise. While the data coding model120may be trained in an unsupervised or self-supervised process, the binary classification model126may be trained in a supervised process, receiving feedback in the form of the negative samples216and the positive samples218.

The digital image302may encoded and subsequently decoded, as described herein, to produce a decoded digital image304. The decoded digital image304may correspond to a single image taken from the set of decoded digital images122. As seen inFIG. 3, the decoded digital image304may be slightly altered compared to the digital image302. This is because the decoded digital image304has been artificially reconstructed based on the learned parameters of a data coding model (e.g., the data coding model120) of what the asset312should look like when it excludes any anomalies.

The digital image302may be compared to the decoded digital image304to generate a difference map306, which may correspond to a single difference map taken from the set of difference maps124. The difference map306depicts some noise308. The noise308represents the difference between an actual version of the asset312in a state that is free from anomalies and an abstracted version of the asset312in the state that is free from anomalies, with the abstracted version being abstracted by the data coding model used. In order to avoid inadvertently classifying the noise as an anomaly, the difference map306may be used, along with other difference maps from the set of difference maps124, to generate a binary classification model, such as the binary classification model126, to identify noise in a difference map.

Referring toFIG. 4, a digital image progression400is depicted. The digital image progression400may correspond to a process for altering a digital image for use with training a neural network. The first image is a digital image402depicting an asset412and a feature404. The feature may be an anomaly. Alternatively, the feature may not constitute an anomaly, but may, if used to train a neural network, reduce the effectiveness of the training. In the particular example ofFIG. 4, the feature404may correspond to a crack in the ground due to earth erosion that has formed near the asset412. The digital image402may correspond to a single image taken from the set of digital images110and the asset412may correspond to a single asset taken from the assets112.

In some embodiments, in order to perform effective training, the feature404is removed artificially from the digital image402to generate an altered digital image406. The altered digital image406may then replace the digital image402in the set of digital images110. The altered digital image406may be encoded and decoded by the neural network108to generate a decoded digital image408. The decoded digital image408may be compared to the digital image402to generate a difference map410. The difference map410may clearly show detection414corresponding to the feature404. The difference map410may then make up part of the set of difference maps124used to generate the binary classification model126.

Referring toFIG. 5, a target digital image progression500is depicted. The progression500may corresponding to a process for detecting an anomaly532. The first image is a target digital image528depicting a target asset530. For example, the target asset530may correspond to a fenced-in piece of equipment. The anomaly532may be a portion of the fence that is missing or otherwise broken or obstructed. The target asset530may be associated with a category of target assets. For example, the category of assets may include similar assets that include a similarly shaped structure positioned within a rectangular fence area as depicted in the target digital image528. The target digital image528may correspond to the target digital image128and the target asset530may correspond to the target asset130.

Using a neural network, such as the neural network108, along with a data coding model, such as the data coding model120, the target digital image528may be encoded and subsequently decoded to generate a decoded target digital image534. Because the data coding model120is trained to reconstruct the digital images depicting assets within the category of assets as being without any anomalies, the decoded target digital image534may be reconstructed without the anomaly532.

The target digital image528may be compared to the decoded target digital image534to generate a difference map536. The difference map536may correspond to the difference map136. As shown inFIG. 5, the difference map536depicts the anomaly532. In response to a determination that the anomaly532is visible in the difference map536, anomaly alert data538may be generated. The anomaly alert data538may include an annotated digital image540that depicts the target asset530and the anomaly532along with an annotation550indicating a position, or otherwise highlighting, the anomaly532. The anomaly alert data538may be sent to the user output device142for display to a user. In other cases, where an anomaly does not exist, the anomaly alert data may not be generated.

Referring toFIG. 6, a target digital image progression600is depicted. The progression600may corresponding to a process for detecting an anomaly632in a for a target asset630that is different from, but in the same category as, the target asset530. The first image may be a target digital image628depicting the target asset630.

Using the neural network108, along with the data coding model120, the target digital image628may be encoded and subsequently decoded to generate a decoded target digital image634depicting the target asset630as not including any anomalies. The target digital image628may be compared to the decoded target digital image634to generate a difference map636, which may depict the anomaly632.

In response to a determination that the anomaly632is visible in the difference map636, anomaly alert data638may be generated. The anomaly alert data638may include an annotated digital image640that depicts the target asset630and the anomaly632along with an annotation650indicating a position, or otherwise highlighting, the anomaly632. The anomaly alert data638may be sent to the user output device142for display to a user. As shown inFIG. 6, a single data coding model, such as the data coding model120, can be used to detect anomalies in many different assets belonging to the same category of assets. Further, every type of anomaly that may exist, need not be documented ahead of time because instead of being trained to reconstruct any particular anomaly, the data coding model120may be trained to reconstruct a depiction of assets in a state that is free from anomalies. Anomalies may then be detected through comparison.

Referring toFIG. 7a block diagram of an embodiment of a high level architecture of neural network108is depicted. The neural network108may be an autoencoder including multiple encoding layers702and multiple decoding layers704. At each of the multiple encoding layers702and the multiple decoding layers704, a data coding model120may be learned as a set of mathematical variables and operations from the set of training images110constraint by the architecture of neural network108. The data coding model120may be associated with a state of a category of assets that is free from anomalies.

During operation, the neural network108may receive a target digital image128and may encode the target digital image128using the multiple encoding layers702in conjunction with the data coding model120. The process may result in encoded target data132. The encoded target data132may then be decoded, or reconstructed, using the multiple decoding layers704to generate a decoded target digital image134. However, because the data coding model120is associated with the state that is free from anomalies, any anomalies depicted in the target digital image128may not be depicted in the decoded target digital image134. By comparing the target digital image128with the decoded target digital image134, anomalies may be detected automatically.

Referring toFIG. 8, an embodiment of a method800for training a neural network for anomaly detection is depicted. The method800may include receiving a set of digital images, each digital image of the set of digital images depicting a distinct asset corresponding to a category of assets, and each digital image depicting the distinct asset in a state that is free from anomalies associated with the category of assets, at802. For example, the set of digital images110may be received at the one or more processors102.

The method800may further include generating a data coding model corresponding to the category of assets by training a neural network with the set of digital images, the data coding model associated with the state that is free from anomalies, at804. For example, the data coding model120may be generated based on the set of digital images110.

A benefit of the method800is that a data coding model may be trained to reconstruct images depicting assets in a state that is free from anomalies, which, when compared to the original image enables detection of an anomaly in the original image. The data coding model may then be distributed for anomaly detection at systems that may not have the capacity to maintain large databanks of previous image data. Other benefits may exist.

Referring toFIG. 9, an embodiment of a method900for anomaly detection is depicted. The method900may be a continuation of, and can be used together with, the method800. The method900may include receiving a data coding model corresponding to a category of assets, the data coding model associated with a state that is free from anomalies associated with the category of assets, at902. For example, the one or more processors102may receive the data coding model120. As another example, the local server222may receive the data coding model120from the neural network108, which may be remote from the local server222.

The method900may further include receiving a target digital image depicting a target asset corresponding to the category of assets, at904. For example, the target digital image128may be received at the one or more processors102.

The method900may also include generating encoded target data by encoding the target digital image using the data coding model, at906. For example, the encoded target data132may be generated.

The method900may include generating a decoded target digital image by decoding the encoded target data using the data coding model, at908. For example, the decoded target digital image134may be generated.

The method900may further include comparing the target digital image to the decoded target digital image to generate a difference map, at910. For example, the target digital image128may be compared to the decoded target digital image134to generate the difference map136.

The method900may also include, in response to a determination that the difference map depicts any anomaly, generating anomaly alert data, at912. For example, the one or more processors102may determine that the difference map depicts an anomaly and may generate the anomaly alert data138.

A benefit of the method900is that an anomaly may be detected in a target digital image of a target asset without relying on previously taken images of the target asset. Further, the method900uses a data coding model that is associated with an anomaly free state to fundamentally change the functionality of a neural network, as compared to typical anomaly detection methods that would use the neural network to identify or recreate the anomalies themselves instead of the assets in a state that is free of any anomalies. Other advantages may exist.

Although various embodiments have been shown and described, the present disclosure is not so limited and will be understood to include all such modifications and variations as would be apparent to one skilled in the art.