INCREMENTAL LEARNING FOR ANOMALY DETECTION AND LOCALIZATION IN IMAGES

Anomalies in multiple different scenes or images can be detected and localized in a single training flow of a neural network. In various examples, incremental learning can be applied to a given system or network, such that the system or network can learn the distribution of new scenes in a single training flow. Thus, in some cases, when an anomalous image from a new scene is given as input to the network, the network can detect and localize the anomaly.

BACKGROUND

An anomaly can generally be defined as an event or occurrence that does not follow expected or normal behavior. In the context of neural networks or machine learning, an anomaly can be difficult to define, but the definition can be critical to the success and effectiveness of a given anomaly detector. An efficient anomaly detector should be capable of differentiating between anomalous and normal instances with high precision, so as to avoid false alarms. With respect to identifying anomalies in images or scenes, it can be impractical or, in some cases, infeasible using existing approaches, to train a neural network to identify anomalies of different types that can occur in all sorts of scenes. Thus, current neural network approaches to identifying anomalies lack capabilities in terms of efficiency and accuracy, among others

BRIEF SUMMARY

Embodiments of the invention address and overcome one or more of the described-herein shortcomings by providing methods, systems, and apparatuses that improve anomaly detection. For example, in various embodiments, anomalies in multiple different scenes or images can be detected and localized in a single training flow of a neural network. In various examples, incremental learning can be applied to a given system or network, such that the system or network can learn the distribution of new scenes in a single training flow. Thus, in some cases, when an anomalous image from a new scene is given as input to the network, the network can detect and localize the anomaly.

In an example aspect, a system can include a first neural network system and a second neural network system. A first encoder module of the first neural network system can receive a plurality of input images that each include scenes that appear as expected, such that the plurality of input images defines a plurality of non-anomalous images. The first neural network system can be trained on the plurality of non-anomalous images so as to learn information associated with the plurality of non-anomalous images. The first neural network system can transfer information associated with the plurality of non-anomalous images to the second neural network system, for instance using knowledge distillation loss, such that the first neural network system defines a teacher model, and the second neural network system defines a student model. After training the first neural network system, a second encoder module of the second neural network system can receive a given input image that includes a scene that includes an anomaly so as to not appear as expected, such that the input image defines an anomalous image. Based on the information associated with the plurality of non-anomalous images, the system can detect the anomaly of the anomalous image. In some cases, based on the information associated with the plurality of non-anomalous images, the system can determine a region of the anomalous image that corresponds to the anomaly, so as to localize the anomaly.

DETAILED DESCRIPTION

In various embodiments, anomalies in multiple different scenes or images can be detected and localized in a single training flow of a neural network. In various examples, incremental learning can be applied to a given system or network, such that the system or network can learn the distribution of new scenes in a single training flow. System and network can be used interchangeably herein without limitation, unless otherwise specified. Because of the training described herein, in some cases, when an anomalous image from a new scene is given as input to the network, the network can detect and localize the anomaly. To do so, as described herein, the network can be trained on non-anomalous images using unsupervised learning.

In particular, referring toFIG.1, a system or network100can be trained on a plurality of input images102. The input images102can define respective scenes, for instance industrial scenes that include one or more machines or components, or medical scenes that include medical devices or bodily structures. It will be understood that that the input images are not limited to the examples described herein. That is, the input images102can vary as desired, and all such input images are contemplated as being within the scope of this disclosure. Further, in various examples, the input images102can define a vectorized input, RGB image, CAD image, or the like. The input images102can include images that are captured by various sensors or camera, and all such captured images are contemplated as being within the scope of this disclosure. By way of example, a given input image102of a given machine can be captured by a camera positioned to capture images of all or part of the machine. As described further herein, in accordance with various embodiments the system100can be trained on input images102that are non-anomalous images. Generally non-anomalous images are images that define a scene that is ordinary, or is consistent with an expectation for the scene. By way of example, a non-anomalous image of a particular machine, such as a turbine or computer numerical control (CNC) machine, might depict the machine in its normal operating state that is consistent with its design. Continuing with the example, an anomalous image of the same machine might depict a tool that was left in the machine, or an additional or damaged component of the machine. In an example, a set of non-anomalous images is defined, and then images that depict a scenario that is not covered in the set of non-anomalous images can be considered to be anomalous images.

With continuing reference toFIG.1, the network100can define an adversarial variational autoencoder (AVAE) system, for instance a convolutional AVAE. The network100can include an encoder convolutional network or module104, a decoder convolutional network or module106, and a discriminator convolutional network or module108. Thus, each of the encoder module104, decoder module106, and discriminator module108can include a plurality of layers. Layers of the encoder module104, decoder module106, and discriminator module108can be fully connected or convolutional. Fully connected layers may include neurons that communicate their respective output to every neuron in an adjacent layer. In contrast to the fully connected layers, the convolutional layers may be locally connected, such that, for example, the neurons in a given layer might be connected to a limited number of neurons in an adjacent layer. The convolutional layers can also be configured to share connection strengths associated with the strength of each neuron. It will be understood that the network100is simplified for purposes of example. For example, the network100may include any number of layers as desired, in particular any number of intermediate layers, and all such models are contemplated as being within the scope of this disclosure.

The encoder module104, in particular an input layer of the encoder module104, can receive the input images102. The encoder module104can process the input images102to generate encoder outputs110. In particular, a given encoder output110, which can also be referred to as z, can represent its corresponding input image102, which can also be referred to as x, in the latent space. For example, the encoder module104can include a neural network that receives an input image (or datapoint x), and outputs its hidden (latent) representation z. In particular, a given input image102can define a first number of dimensions, and the encoder module104can encode the data of the input image102into the encoder output110(or latent representation space z) that defines a second number of dimensions that is less than the first number of dimensions. Thus, the encoder module104can learn an efficient compression of the data of the input images102into the lower dimensional space of the data of the encoder output110. The encoder output110defines a data distribution that can be characterized by parameters112, in particular a mean μ and standard deviation σ. Using the parameters112, the latent activations can be sampled so as to define a decoder input114. Using the decoder input114that is based on the latent space z or encoder output110, the decoder module106can reconstruct the input images102so as to generate corresponding reconstructed images116, which can be referred to as z. The discriminator module108can receive the input images102and the reconstructed images116as input, and can compare the respective data distributions of the input images102and reconstructed images, so as to make a determination or prediction, or to generate a label such as related to whether the input x is real or fake. Thus, a discriminator output118can indicate whether the corresponding input to the system100is real or fake. A goal during training of the system100is to make the reconstructed images116sharper and more accurate as compared to the corresponding input images102. To do so, an attention map120, which can be referred to as â can be normalized ∈ (0, 1), can be computed by backpropagating the gradients from the encoder output110(z).

With respect to attention maps, it is recognized herein that state-of-the-art deep learning based classifiers typically generate the attention map by backpropagating the gradients corresponding to a specific class to the input image. In various embodiments the input images102are unlabeled, and thus activation maps can be obtained from the latent space z. The activation maps can be used to generate the attention maps. In various examples, the generated attention maps indicate or describe regions of the input images102that are discriminative. Discriminative regions can refer to regions on which the ultimate label or classification of the discriminator output118is based. In an example, Grad-CAM can be used to compute the attention map120using gradient backpropagation.

With respect to training the system100, in accordance with some embodiments, the system100is trained only on non-anomalous images. Thus, during training, the input images102are non-anomalous such that, during inference time, or when the system100is used to make a prediction or generate a classification, the input images102can include anomalous images, and the system will not reconstruct regions of the anomalous images that pertain to the anomaly. The network or system100trained and used in this manner can define a teacher model202, which is described further herein with reference toFIG.2. In particular, the discriminator module108can compute a pixel-wise difference between a given reconstructed image116and its corresponding anomalous input image102. Based on the pixel-wise difference, the discriminator module108, and thus the network100, can generate a class score in the discriminator output118. Thus, the score can represent the pixel-wise difference between an input image and a reconstructed image that corresponds to the input image. Generally, in some cases, the score is higher as the reconstruction is less accurate. A high score can indicate that the input image comes from a different distribution than the non-anomalous image, such that input image can be considered to be anomalous. In an example, when the class score (or pixel-wise difference) associated with an input image is greater than a predetermined or pre-specified threshold, the input image can be defined as anomalous. Similarly, when the class score (or pixel-wise difference) associated with an input image is less than a predetermined or pre-specified threshold, the input image can be defined as non-anomalous. The predetermined threshold can vary in accordance with, or be based on, the use case of the particular operation.

In some cases, training data is limited, such as, for example, when training on images of industrial machines. Further, anomalous data (images) can be difficult or expensive to collect. Thus, in various examples, the network100is trained on a limited amount of non-anomalous images so to supplement that training, the network100can be pre-trained, for example on ImageNet. In an example, the encoder module104can then be fine-tuned on original data or non-autonomous images. Such original data can include images with a large amount of high frequency components, such that decoder module106can be configured to employ skip connections and an inter-leaved convolutional layer to preserve local information. The discriminator module108positioned at the output of the decoder module106can maintain the distribution of the input images102and reconstructed images116, hereby enabling a sharper reconstruction. To preserve the spatial relation, the system100can define an AVAE that is end-to-end convolutional.

As described further, the network100can be configured as a teacher model202or student model204. In particular, a system or network200can include multiple networks100arranged such that one network100defines the teacher model202, and one network defines the student model204. Thus, as described herein, the system200can define a student-teacher model capable of learning the distributions of new sets of scenes or images, while retaining previous scenes or images without increasing the network's memory footprint.

In an example, the system200can localize an anomaly of one of the input images102without any prior information as to where or what the anomaly is. By way of example, a given input image102might define a scene that includes a particular machine, such as a turbine or CNC machine. The given input image102might further define an anomaly. By way of example, a tool left inside of the particular machine can be included in the given input image102, so as to define the anomaly. In various examples, the system200is trained on only non-anomalous images, and can localize the anomaly of an anomalous image based on that training. In some cases, the training might not include an image of the particular machine, such that the system200can identify an anomaly of a scene in which it has not been trained. In particular, in some cases, attention maps of the encoder output110can define the latent space attention by using GradCAM, which is a gradient-based class activation map generation mechanism. In various examples, because the latent space corresponds to the distribution of non-anomalous images, the resulting attention maps that are generated can be considered to be the non-anomalous attention map. The areas of attention in an attention map120can be maximized, so as to encourage the network100to attend to the entire image because the training, in some cases, might involve only non-anomalous images. To maximize the attention, extra supervision can be provided to the network100to better attend to the non-anomalous regions of the image. In particular, for example, the attention map, which can be normalized from0to1, can be compared and tuned to a target attention map having all 1's, thereby providing extra supervision by encouraging attention to attend to the entire image. During testing of the system200, given an anomalous image (an image having an anomaly) as an input image102, the attention map120that is generated can indicate or highlight the non-anomalous regions of the image, such that inverting the attention map120can results in an inverted attention map that indicates or highlights the abnormal region of the image. That is, the inverted attention map can indicate the one or more anomalous regions of the image, thereby localizing the anomaly. In some cases, the attention map of the normal class is calculated and inverted, so as to generate the attention map that highlights the anomaly of an anomalous image. The inversion can be generated by subtracting the normalized attention map from an image (e.g., 2D matrix) that defines all 1's.

Further, the input images102with which the system200is trained can each define respective scenes that are different from each other. Thus, the network200can be trained to detect and localize anomalies in a single training flow, rather than training the network200on each scene individually. In various examples, to facilitate the learning of a new scene that was not previously defined in the input images102, the student model204can acquire information (e.g., a scene that was previously defined in the input images102) from the teacher model202, so as to learn the distribution of non-anomalous images of a new scene without losing information of the previous scene which it acquires from the teacher model202. Thus, the system200, in particular the student model204, can retain the information of the non-anomalous distribution of a previous (old) scene, while the teacher model202learns information of a new scene, in a single training flow.

In another example aspect, with continuing reference toFIG.2, during training, a knowledge distillation loss206can be shared, for instance shared by the teacher model202to the student model204, so as to enable the student model204to retain information from previous input images or scenes (or classes) while learning an input image102that includes a new scene208. Thus, the system200can be trained so as to learn the distribution of a new set of scenes in a single training flow. In particular, the student model204and the teacher model202can define or have the same model parameters as each other. In various examples, the student model204learns information associated with previously learned scenes from the teacher model202. In addition, the student model204can learn the non-anomalous distribution of a new scene while retaining the information associated with the previously learned scenes acquired from the teacher model202. In some cases, a memory footprint of the system200does not grow with respect to the number of classes (or number of scenes) that the system200encounters. The learned information from the teacher model202of a previous scene can be transferred to the student model204with the help of knowledge distillation loss206. For example, the student model204can be trained with the knowledge distillation loss206and the same objective function in which the teacher model202is trained, so as to define a trained student model. The knowledge distillation loss can refer to modifications of the cross-entropy loss that can encourage the raw outputs of the student networks to be similar to the raw outputs of the teacher network. Training with this distillation loss can transfer the knowledge learned in the teacher network202to the student network204. After that training, given another new scene210defined by another new input image102, the trained student model can become the teacher model202, and an untrained network100can define the student model204. Thus, the untrained student model204can learn the information from the other new scene210with (while learning) the previously learned information. Consequently, during inference time, given an anomalous image from any class, the student model204can detect and localizing the anomaly included in the anomalous image.

With continuing reference toFIG.2, in an example, x can represent the input image of a scene, and x′ can represent the input image a new scene. The input image x can be input into the encoder module104of the teacher model202, and the input image x′ can be input into the encoder module104of the student model204. In response to the respective inputs, the decoder module106of the teacher model can generate the reconstructed image {circumflex over (x)}, and the decoder module106of the student model204can generate the reconstructed image {circumflex over (x)}′. The discriminator module108of each model can define a respective convolution network that can determine whether the reconstructed images x hat and {circumflex over (x)}′ defines the same distribution as that of the input images x and z, respectively. Based on the determination of the discriminator module108, the discriminator output can indicate a class score, or can indicate whether the respective input is real or fake, for example, thereby resulting in a Real/Fake decision as output. In various examples, a goal is to make the reconstruction sharper. The teacher model202can compute an attention map â and the student model204can compute an attention map â′. The attention maps can be by backpropagating the gradients from the latent space. The knowledge distillation loss206can represent the transfer of information of the previous (old) scene on which the teacher network was trained (e.g., the scene from image x), to the untrained student network. In various examples, the student network204retains the information of the previous scene (e.g., from image x) along with learning information from the new scene (e.g., from image x′).

Referring now toFIG.3, example operations300are shown that can be performed by the system200, which can include a first neural network system100and a second neural network system100. At302, a first encoder module of the first neural network system can receive a plurality of input images that each include scenes that appear as expected, such that the plurality of input images defines a plurality of non-anomalous images. At304, the first neural network system can be trained on the plurality of non-anomalous images so as to learn information associated with the plurality of non-anomalous images. In some examples, training the first neural network system can include computing, by the first encoder module, a latent space representation of each of the plurality of non-anomalous images. Based on the respective latent space representation, the first neural network system can generate an attention map that indicates one or more discriminative regions of the respective non-anomalous image. Further data associated with the attention map can be back-propagated through the first encoder module, for instance toward an input layer of the encoder module104.

Still referring toFIG.3, the first neural network system can transfer information associated with the plurality of non-anomalous images to the second neural network system, such that the first neural network system defines a teacher model (e.g., teacher model202), and the second neural network system defines a student model (e.g., student model204). In some examples, transferring the information associated with the plurality of non-anomalous images to the second neural network system can include computing, by a decoder module of the first neural network system, a knowledge distillation loss. Further, the knowledge distillation loss can be transferred to the second neural network system.

At308, for instance after training the first neural network system, a second encoder module of the second neural network system can receive a given input image that includes a scene that includes an anomaly so as to not appear as expected, such that the input image defines an anomalous image. By way of example, and without limitation, the scenes defined by the plurality of non-anomalous images might include a plurality of industrial machines, and the scene defined by the anomalous image might include a machine different from the plurality of industrial machines. By way of further example, the scene defined by the anomalous image might further include a tool within the machine.

With continuing reference toFIG.3, at310, based on the information associated with the plurality of non-anomalous images, the system200can detect the anomaly of the anomalous image. In some cases, based on the information associated with the plurality of non-anomalous images, the system can determine a region of the anomalous image that corresponds to the anomaly, so as to localize the anomaly. In particular, for example, the first encoder module can compute a latent space representation of each of the plurality of non-anomalous images. Based on the respective latent space representation of each of the plurality of non-anomalous images, a first decoder module of the first neural network system, can reconstruct the plurality of non-anomalous images. The second encoder module can compute a latent space representation of the anomalous image. Based on the latent representation of the anomalous image, a second decoder module of the second neural network system can reconstruct regions of the anomalous image other than the region that corresponds to the anomaly. Additionally, or alternatively, based on the latent space representation of the anomalous image, the second neural network can generate an attention map that indicates one or more discriminative regions of the anomalous image, and the second neural network can invert the attention map so as to localize the anomaly of the anomalous image.

Thereafter, by way of further example, information associated with the plurality of non-anomalous images and the anomalous image can be transferred to a third neural network system, such that the second neural network system defines the teacher model, and the third neural network system defines the student model. Thus, a third encoder module of the third neural network system can receive another input image that includes another scene that includes another anomaly so as to not appear as expected, such that the other input image defines another anomalous image. Based on the information associated with the plurality of non-anomalous images and the anomalous image, the third neural network system can detect the other anomaly of the other anomalous image. This process can continue as many times as desired so as define incremental learning, wherein during each iteration or increment the student model becomes the teacher model, and a new untrained network becomes the student model Consequently, the system200can be trained to detect and localize anomalies, based on non-anomalous images, in a single training flow.

FIG.4illustrates an example of a computing environment within which embodiments of the present disclosure may be implemented. A computing environment400includes a computer system510that may include a communication mechanism such as a system bus521or other communication mechanism for communicating information within the computer system510. The computer system510further includes one or more processors520coupled with the system bus521for processing the information. The system200may include, or be coupled to, the one or more processors520.

The processors520may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as described herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may include any type of suitable processing unit including, but not limited to, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s)520may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor may be capable of supporting any of a variety of instruction sets. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

The system bus521may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may permit exchange of information (e.g., data (including computer-executable code), signaling, etc.) between various components of the computer system510. The system bus521may include, without limitation, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, and so forth. The system bus521may be associated with any suitable bus architecture including, without limitation, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnects (PCI) architecture, a PCI-Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, and so forth.

Continuing with reference toFIG.4, the computer system510may also include a system memory530coupled to the system bus521for storing information and instructions to be executed by processors520. The system memory530may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM)531and/or random access memory (RAM)532. The RAM532may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM531may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory530may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors520. A basic input/output system533(BIOS) containing the basic routines that help to transfer information between elements within computer system510, such as during start-up, may be stored in the ROM531. RAM532may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors520. System memory530may additionally include, for example, operating system534, application programs535, and other program modules536. Application programs535may also include a user portal for development of the application program, allowing input parameters to be entered and modified as necessary.

The operating system534may be loaded into the memory530and may provide an interface between other application software executing on the computer system510and hardware resources of the computer system510. More specifically, the operating system534may include a set of computer-executable instructions for managing hardware resources of the computer system510and for providing common services to other application programs (e.g., managing memory allocation among various application programs). In certain example embodiments, the operating system534may control execution of one or more of the program modules depicted as being stored in the data storage540. The operating system534may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

The computer system510may also include a disk/media controller543coupled to the system bus521to control one or more storage devices for storing information and instructions, such as a magnetic hard disk541and/or a removable media drive542(e.g., floppy disk drive, compact disc drive, tape drive, flash drive, and/or solid state drive). Storage devices540may be added to the computer system510using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire). Storage devices541,542may be external to the computer system510.

The computer system510may also include a field device interface565coupled to the system bus521to control a field device566, such as a device used in a production line. The computer system510may include a user input interface or GUI561, which may comprise one or more input devices, such as a keyboard, touchscreen, tablet and/or a pointing device, for interacting with a computer user and providing information to the processors520.

The computer system510may perform a portion or all of the processing steps of embodiments of the invention in response to the processors520executing one or more sequences of one or more instructions contained in a memory, such as the system memory530. Such instructions may be read into the system memory530from another computer readable medium of storage540, such as the magnetic hard disk541or the removable media drive542. The magnetic hard disk541and/or removable media drive542may contain one or more data stores and data files used by embodiments of the present disclosure. The data store540may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores in which data is stored on more than one node of a computer network, peer-to-peer network data stores, or the like. The data stores may store various types of data such as, for example, skill data, sensor data, or any other data generated in accordance with the embodiments of the disclosure. Data store contents and data files may be encrypted to improve security. The processors520may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory530. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

The computing environment400may further include the computer system510operating in a networked environment using logical connections to one or more remote computers, such as remote computing device580. The network interface570may enable communication, for example, with other remote devices580or systems and/or the storage devices541,542via the network571. Remote computing device580may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system510. When used in a networking environment, computer system510may include modem572for establishing communications over a network571, such as the Internet. Modem572may be connected to system bus521via user network interface570, or via another appropriate mechanism.

Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and/or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”