Patent Description:
In industrial production environments, detecting a fault (e.g., a defect) in a product is significant for reliable product production. A typical approach for detecting a defect in a product may include using an image (or a video) of the product. In addition, the production process and facility abnormalities may also be monitored in real time through images that capture a product's state or a state of the various manufacturing processes.

<CIT> discloses a leather flaw detection method, a system, a storage medium and computer equipment. A triple network is used for quickly filtering out images without flaws, so that the processing efficiency can be improved, the detection cost can be saved, and the method is more suitable for the practical operation condition. <NPL>) discloses a siamese defect-aware attention network (SDANet) with a template comparison detection strategy that improves the defect detection technique for matching new samples without rapidly collecting new data and retraining the model.

In a general aspect here is provided an apparatus according to claim <NUM>. The apparatus includes a processor configured to execute a plurality of instructions and a memory storing the plurality of instructions, wherein execution of the plurality of instructions configures the processor to output a defect prediction score of an input image through the use of a neural network provided reference image, the input image, and an enhanced image, and the neural network includes an attention map modulator configured to adaptively adjust an intensity of an attention map generated during the use of the neural network.

The processor is further configured to generate the enhanced image based on the input image and the reference image, in which a defective area is emphasized.

The generating of the enhanced image further may include obtaining a differential image based on the input image and the reference image and outputting the enhanced image by adjusting an intensity of a defective area included in the differential image.

The neural network further includes a feature extractor configured to receive the reference image and the input image and extract a feature map and an attention modulator configured to receive the feature map and the enhanced image and output a modulated feature map.

The attention modulator may operate as reflecting that the attention modulator was trained to increase weights of defect-associated values included in the feature map.

The attention modulator may include an attention map generator configured to receive the enhanced image and output a modulated attention map. The attention modulator includes a feature modulator configured to receive the modulated attention map and the feature map and output the modulated feature map.

The attention map generator includes a generator configured to receive the enhanced image and output the attention map. The attention map modulator may be further configured to receive the attention map and output the modulated attention map, and the modulated attention map may be obtained by adaptively adjusting an intensity of the attention map.

The feature modulator may be configured to output the modulated feature map by applying the modulated attention map to the feature map.

The feature modulator is configured to apply the modulated attention map to a portion of the feature map and output the modulated feature map by concatenating a result of applying the modulated attention map and a remaining portion of the feature map excluding the portion.

The feature modulator may be configured to perform an elementwise operation on at least a portion of the feature map and the modulated attention map.

The neural network further may include a defect classifier configured to receive the modulated feature map and output the defect prediction score of the input image.

In a general aspect there is provided a processor-implemented method according to claim <NUM>. The method includes generating a defect prediction score of an input image through the use of a neural network provided reference image, the input image, and an enhanced image, the neural network including an attention map modulator configured to adaptively adjust an intensity of an attention map generated during the use of the neural network.

The method includes generating the enhanced image based on the input image and the reference image, in which a defective area is emphasized.

The generating of the of the enhanced image further may include obtaining a differential image based on the input image and the reference image and outputting the enhanced image by adjusting an intensity of a defective area included in the differential image.

The attention map generator includes a generator configured to receive the enhanced image and output the attention map. The attention map modulator may further be configured to receive the attention map and output the modulated attention map, the modulated attention map may be obtained by adaptively adjusting an intensity of the attention map.

The feature modulator may be configured to apply the modulated attention map to a portion of the feature map and output the modulated feature map by concatenating a result of applying the modulated attention map and a remaining portion of the feature map excluding the portion.

In an example, here is provided a processor-implemented method, the method including comparing an input image to a reference image to determine a defective area on an object, emphasizing the defective area to generate an enhanced image of the input image, and implementing at least a portion of a neural network based on an attention map generated based on the enhanced image, the reference image, and the input image.

The neural network may adaptively adjust attention map intensities corresponding to the defective area in the generating of the attention map.

The method may include generating, by the neural network, a defect prediction score for the input image indicating a likelihood of whether the input image contains a defect within the determined defective area.

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals may be understood to refer to the same or like elements, features, and structures.

For example, the sequences within and/or of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, except for sequences within and/or of operations necessarily occurring in a certain order. As another example, the sequences of and/or within operations may be performed in parallel, except for at least a portion of sequences of and/or within operations necessarily occurring in an order, e.g., a certain order.

As non-limiting examples, terms "comprise" or "comprises," "include" or "includes," and "have" or "has" specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof, or the alternate presence of an alternative stated features, numbers, operations, members, elements, and/or combinations thereof. Additionally, while one embodiment may set forth such terms "comprise" or "comprises," "include" or "includes," and "have" or "has" specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, other embodiments may exist where one or more of the stated features, numbers, operations, members, elements, and/or combinations thereof are not present.

The phrases "at least one of A, B, and C", "at least one of A, B, or C', and the like are intended to have disjunctive meanings, and these phrases "at least one of A, B, and C", "at least one of A, B, or C', and the like also include examples where there may be one or more of each of A, B, and/or C (e.g., any combination of one or more of each of A, B, and C), unless the corresponding description and embodiment necessitates such listings (e.g., "at least one of A, B, and C") to be interpreted to have a conjunctive meaning.

As used in connection with various example embodiments of the disclosure, any use of the terms "module" or "unit" means processing hardware, e.g., configured to implement software and/or firmware to configure such processing hardware to perform corresponding operations, and may interchangeably be used with other terms, for example, "logic," "logic block," "part," or "circuitry". As one non-limiting example, an application-predetermined integrated circuit (ASIC) may be referred to as an application-predetermined integrated module. As another non-limiting example, a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC) may be respectively referred to as a field-programmable gate unit or an application-specific integrated unit. In a non-limiting example, such software may include components such as software components, object-oriented software components, class components, and may include processor task components, processes, functions, attributes, procedures, subroutines, segments of the software. Software may further include program code, drivers, firmware, microcode, circuits, data, database, data structures, tables, arrays, and variables. In another non-limiting example, such software may be executed by one or more central processing units (CPUs) of an electronic device or secure multimedia card.

Due to manufacturing techniques and/or tolerances, variations of the shapes shown in the drawings may occur. Thus, the examples described herein are not limited to the specific shapes shown in the drawings, but include changes in shape that occur during manufacturing.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term "may" herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

<FIG> illustrate examples of typical defect inspection approaches.

Referring to <FIG>, a first typical defect detection algorithm may include three operations. In operation <NUM>, an input image (e.g., a video of a potentially defective product) that may include defect-related information may be acquired by capturing an image of a product to be inspected (e.g., a product subject to defect detection). In an example, some or all aspects of the input image may be enhanced. In operation <NUM>, a differential image (e.g., a differential video) may be obtained from an image of a normal product (e.g., a reference video) and an image of the product to be inspected (e.g., a video of the defect), and defect-related information included in the differential image may be enhanced. In operation <NUM>, features for each image may be estimated based on the image of the normal product, the image of the product to be inspected, and the differential image. The feature estimation operation may be performed by organic combination with human interaction, such as in process where the product image is also observed by a human. In operation <NUM>, it may be determined whether the product to be inspected is defective. In operation <NUM>, a decision tree may be constructed based on the features (e.g., the features estimated in operation <NUM>), and a product having one or more features corresponding to a preset defect condition may be determined to be defective.

Referring to <FIG>, in another typical defect detection example, operations <NUM> and <NUM> described above with reference to <FIG> may be replaced by a neural network. The image of the normal product (e.g., the reference video), the image of the product to be inspected (e.g., the video of the defect), and the differential image (e.g., the differential video) may be input into the neural network, to determine whether the product to be inspected is defective.

In an example of a method of industrial fault detection there may be instances where changes in the defect observation environment may occur. In addition, in some instances, there may be times where a fault detection employing a neural network takes place within a suboptimal scenario where there is a small amount of training data. In a non-limiting example, one or more embodiments of an image processing system implementing an industrial fault detection system may be stable in a defect observation environment where changes occur, including noise, distortion, and interference, as well as other external factors that may occur. In a non-limiting example, one or more embodiments of an image processing system implementing an industrial fault detection system may overcome suboptimal scenarios in which there is a small amount of training data available.

<FIG> illustrates an example of an electronic device according to one or more embodiments.

A typical inspection approach that is based on a human interaction, such as the approach of <FIG>, may be inferior to the performance of features based on the use of an artificial intelligence model, e.g., machine learning. In the following examples, which will be described below with reference to <FIG>, features based on a use of a machine learning model may be used to inspect for and detect defects in products.

In the example described with reference to <FIG>, a typical neural network is used, but the characteristics of an input image for defect detection are not considered. The differential image used in the example of <FIG> is obtained by enhancing defect-related information, and the typical neural network may depend on the differential image. In the example of <FIG>, when the characteristics of the differential image change, an overfitting issue may arise, resulting in a decrease in a detection accuracy of the typical detection approach using the typical neural network. The following examples, which will be described below in greater detail in <FIG>, may use an attention mechanism in a machine learning model, which may improve generalization performance even in a suboptimal scenario, or other examples, where there is a small amount of training data for the machine learning model that may be employed, and may also maintain a predetermined detection accuracy.

Referring to <FIG>, in a defect observation environment(e.g., an environment for observing defects), an electronic device <NUM> may obtain a reliable defect detection result by utilizing an enhanced image in which an intensity of a signal related to a defect is adjusted based on factors within a defect observation environment, as a non-limiting example. In an example, the electronic device <NUM> may improve defect detection accuracy by utilizing an attention mechanism, e.g., connected to or in the machine learning model, that increases a weight of defect-associated information (e.g., a defect-associated value included in a generated feature map). The electronic device <NUM> may perform a defect detection method that may adopt to, or function in, a change in a suboptimal scenario, by utilizing an attention map whose intensity (e.g., contrast) is adaptively adjusted according to the defect observation environment.

In an example, the electronic device <NUM> may detect a defect in an input image (e.g., an image of a product to be inspected) based on the input image and a reference image (e.g., an image of a normal product).

The electronic device <NUM> may detect the defect using a neural network. The neural network may be a model that is trained for a particular purpose, e.g., to infer information from or about an input to the neural network. The neural network may include nodes (artificial neurons) forming the network through trained synaptic connections. Training includes adjusting connection strengths of these synaptic connections.

Each node of the neural network may include, or be connected by, a combination of weights or biases. The neural network may include one or more layers, each including one or more nodes. The training of the neural network may be performed through supervised learning, as a non-limiting example, to infer a known result from a corresponding input by changing the weights of the nodes through iterations of training.

The machine learning model may include a deep neural network (DNN). The neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), a perceptron, a multilayer perceptron, a feed forward (FF), a radial basis network (RBF), a deep feed forward (DFF), a long short-term memory (LSTM), a gated recurrent unit (GRU), an auto encoder (AE), a variational auto encoder (VAE), a denoising auto encoder (DAE), a sparse auto encoder (SAE), a Markov chain (MC), a Hopfield network (HN), a Boltzmann machine (BM), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a deep convolutional network (DCN), a deconvolutional network (DN), a deep convolutional inverse graphics network (DCIGN), a generative adversarial network (GAN), a liquid state machine (LSM), an extreme learning machine (ELM), an echo state network (ESN), a deep residual network (DRN), a differentiable neural computer (DNC), a neural turning machine (NTM), a capsule network (CN), a Kohonen network (KN), an attention network (AN), or other non-limiting machine learning models. While embodiments refer to neural networks, this is for convenience of explanation and embodiments exist where the machine learning model is other than a neural network.

The electronic device <NUM> may be a personal computer (PC), a data server, or a portable device.

The portable device may be a laptop computer, a mobile phone, a smart phone, a tablet PC, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device or portable navigation device (PND), a handheld game console, an e-book, or a smart device, as non-limiting examples. The smart device may be a smart watch, a smart band, or a smart ring, as non-limiting examples.

The electronic device <NUM> may include a processor <NUM> and a memory <NUM>. The electronic device may include an imaging sensor <NUM>, e.g., a CCD or CMOS image sensor, to capture the input image.

The processor <NUM> may process data stored in the memory <NUM>. The processor <NUM> may execute a computer-readable code (for example, software) stored in the memory <NUM> and instructions triggered by the processor <NUM>. The memory <NUM> may also store the reference image.

The processor <NUM> may further execute programs, and/or may control other operations or functions of the electronic device <NUM>, and may include any one or a combination of two or more of, for example, a central processing unit (CPU), a graphic processing unit (GPU), a neural processing unit (NPU) and tensor processing units (TPUs), but is not limited to the above-described examples.

The processor <NUM> may be a hardware-implemented data processing device having a circuit that is physically structured to execute desired operations. The hardware-implemented data processing apparatus may include, for example, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

When the instructions are executed by the processor <NUM>, the processor <NUM> may be configured to perform a plurality of operations. The processor <NUM> obtains an enhanced image in which a defective area included in an input image is emphasized, based on the input image and a reference image. The processor <NUM> generates a defect prediction score of the input image by inputting the reference image, the input image, and the enhanced image into a neural network. The neural network includes an attention map generator and/or an attention map modulator configured to adaptively adjust an intensity (e.g., a contrast) of an attention map generated during implementation of the neural network. Herein, illustrations, and corresponding or related discussions, of different portions of the neural network are also representative of each illustrated portion, any one or more various combinations of such illustrated portions, or the entire neural network illustration being (or otherwise being implemented by) one or more processors, or one or more processors configured to execute instructions and one or more memories storing the instructions that when executed by the one or more processor configure the one or more processors to perform the respectively described operations herein. The operations performed by the processor <NUM> and the configuration of the neural network will be described in greater detail below with reference to <FIG>.

The memory <NUM> may be implemented as a volatile memory device or a non-volatile memory device. The memory <NUM> may include computer-readable instructions. The processor <NUM> may be configured to execute computer-readable instructions, such as those stored in the memory <NUM>, and through execution of the computer-readable instructions, the processor <NUM> is configured to perform one or more, or any combination, of the operations and/or methods described herein.

The memory <NUM> may be a volatile or nonvolatile memory. The volatile memory device may be implemented as a dynamic random-access memory (DRAM), a static random-access memory (SRAM), a thyristor RAM (T-RAM), a zero capacitor RAM (Z-RAM), or a twin transistor RAM (TTRAM).

The non-volatile memory device may be implemented as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic RAM (MRAM), a spin-transfer torque (STT)-MRAM, a conductive bridging RAM(CBRAM), a ferroelectric RAM (FeRAM), a phase change RAM (PRAM), a resistive RAM (RRAM), a nanotube RRAM, a polymer RAM (PoRAM), a nano floating gate Memory (NFGM), a holographic memory, a molecular electronic memory device), or an insulator resistance change memory.

<FIG> illustrates an example of a defect inspection method based on a neural network according to one or more embodiments.

A processor (e.g., the processor <NUM> of <FIG>) utilizes an enhanced image. In an example, the enhanced image may be an image in which a defective area included in a differential image is emphasized. In an example, when employed in the defect observation environment, the processor <NUM> may obtain a reliable defect detection result by adjusting an intensity of a signal related to the defective area found in the defect observation environment. In addition, the processor <NUM> may obtain relation defect detection results according to an amount of training data the neural network may have been trained on, such as in a suboptimal scenario where there was a limited number of training data. In a suboptimal scenario, the machine learning model may not have had sufficient training data on a product or type of defect.

In a non-limiting example, the processor <NUM> applies an attention mechanism to, or within, a trained machine learning model configured to perform a defect detection algorithm. The attention mechanism may increase a weight of defect-associated information generated by the machine learning model (e.g., a defect-associated value included in a feature map). One or more embodiments improve defect detection accuracy by using the attention mechanism. By using the attention mechanism, examples of the processor <NUM> may improve generalization performance even in a suboptimal scenario that has a small amount of training data, and examples of the processor <NUM> may also improve and maintain a predetermined detection accuracy that may have been available if more training data was accessible.

In a non-limiting example, the processor <NUM> utilizes an attention map that is unaffected by a change in the defect observation environment (or a change in the defect). Thus, in an example, changes in facility or environment in which the defect detection takes place, (e.g., noise, distortion, interference, or other external factors) may not affect the attention map or defect detection method. The processor <NUM> may provide a defect detection method resistant to potential negative effects from a change in training model of the neural network, such as in an where a neural network model has low training data by utilizing an attention map whose intensity (e.g., contrast) is adaptively adjusted according to the defect observation environment.

Referring to <FIG>, the processor <NUM> may utilize a processor <NUM> implement (e.g., execute) a neural network <NUM>. Operations of processor <NUM> may be performed by processor <NUM>, or processor <NUM> may be separate from the processor <NUM> in the electronic device <NUM>. For example, the processor <NUM> may be a digital signal processor, though examples are not limited thereto.

The processor <NUM> may receive a reference image <NUM> (e.g., an image of a normal product) and an input image <NUM> (e.g., an image of a product to be inspected) and output an enhanced image <NUM> in which a defective area is emphasized. The processor <NUM> may be configured to perform signal processing. The processor <NUM> may obtain a differential image based on the reference image <NUM> and the input image <NUM>. The processor <NUM> may output the enhanced image <NUM> in which an intensity of a defective area included in the differential image is adjusted.

The neural network <NUM> is configured to output a defect prediction score <NUM> to determine whether the input image <NUM> includes a defect (e.g., whether the product to be inspected included in the input image <NUM> is defective), based on the reference image <NUM>, the input image <NUM>, and the enhanced image <NUM> provided to the neural network <NUM>. Portions of the neural network <NUM> may include a feature extractor <NUM>, an attention mechanism <NUM>, and a defect classifier <NUM>.

The feature extractor <NUM> receives the reference image <NUM> and the input image <NUM> and extract a feature map <NUM>.

The attention mechanism <NUM> may be trained to increase weights of defect-associated values included in the feature map <NUM>. The attention mechanism <NUM> may be intended to improve the quality of features (e.g., the feature map <NUM>). The attention mechanism <NUM> may include an attention map generator <NUM>-<NUM> and a feature modulator <NUM>-<NUM>.

The attention map generator <NUM>-<NUM> may receive the enhanced image <NUM> and output a modulated attention map <NUM>. The attention map generator <NUM>-<NUM> may generate an attention map whose intensity (e.g., contrast) is adaptively adjusted (e.g., the modulated attention map <NUM>) according to a strength of a neural network's available training data. The attention map generator <NUM>-<NUM> may generate an attention map (e.g., the modulated attention map <NUM>) immune to a change in the defect observation environment (or a change in the defect). The attention map generator <NUM>-<NUM> may include a generator <NUM> and an attention map modulator <NUM>.

The generator <NUM> receives the enhanced image <NUM> and output an attention map <NUM>. The attention map modulator <NUM> receives the attention map <NUM> and output the modulated attention map <NUM>. The attention map modulator <NUM> adaptively adjusts the intensity (e.g., contrast) of the attention map <NUM> according to changes within or other factors that may arise in the defect observation environment. In an example, the modulated attention map <NUM> may be derived through Equation <NUM>.

In Equation <NUM>, A denotes the attention map <NUM>, Amodulated denotes the modulated attention map <NUM>, and σ and u denote parameters for adjusting the intensity (e.g., contrast) of the attention map <NUM>. The parameters σ and u may be trained to contribute to improving detection accuracy. The number of modulated attention maps <NUM> is not limited to "<NUM>" and may vary depending on the configuration of the feature modulator <NUM>-<NUM>.

The feature modulator <NUM>-<NUM> may receive the modulated attention map <NUM> and the feature map <NUM> and output the modulated feature map <NUM>. The feature modulator <NUM>-<NUM> may obtain the modulated feature map <NUM> as a result of performing an elementwise operation on at least a portion of the feature map <NUM> and the modulated attention map <NUM>. The modulated feature map <NUM> may be derived through Equation <NUM>.

In Equation <NUM>, y denotes the modulated feature map <NUM>, Aw and Ab denote modulated attention maps <NUM>, and x denotes the feature map <NUM>. The configuration of the feature modulator <NUM>-<NUM> will be described in greater detail below with reference to <FIG>.

The feature extractor <NUM> and the attention mechanism <NUM> may be connected in series, and the feature extractor330 and the attention mechanism <NUM> connected in series may be repeated a plurality of times. The attention mechanism <NUM> may be designed in a plug-in form, and may be compatible with a neural network model not shown in <FIG>.

The defect classifier <NUM> may receive the modulated feature map <NUM> and output a defect prediction score indicating whether the input image <NUM> includes a defect (e.g., whether the product to be inspected included in the input image is defective).

The electronic device <NUM> may be applied in various manners in an industrial environment for producing products (e.g., elements, components, and finished products) using an image (or video)-based defect inspection algorithm. In a non-limiting example, the electronic device <NUM> may be applied to a semiconductor process. The electronic device <NUM> may monitor the production process and for facility abnormalities in real time based on an input image (or an input video) and predict a product yield rate. The electronic device <NUM> may be used in an industrial environment for producing products and also in the field of automatically detecting an abnormal state found in a security camera image.

<FIG> illustrates examples of a configuration and an operation of a feature modulator shown in <FIG> according to one or more embodiments.

Referring to <FIG>, examples of the configuration and the operation of the feature modulator <NUM>-<NUM> are shown. Two examples, <NUM> and <NUM> are illustrated. The feature modulator <NUM>-<NUM> may perform an elementwise operation on at least a portion of the feature map <NUM> and the modulated attention map <NUM>.

In example <NUM>, the feature modulator <NUM>-<NUM> may be configured to output the modulated feature map <NUM> by applying the modulated attention map <NUM> to the feature map <NUM>. In example <NUM>, the feature modulator <NUM>-<NUM> may be configured to output the modulated feature map <NUM> by applying the modulated attention map <NUM> to a portion <NUM>-<NUM> of the feature map <NUM> and concatenating a result (e.g., a modulated feature map <NUM>) of applying the modulated attention map and a remaining portion <NUM>-<NUM> of the feature map <NUM>. By utilizing the modulated feature map <NUM> including the modulated feature map <NUM> and the remaining portion <NUM>-<NUM> of the feature map <NUM> including unique information of the input image, an electronic device (e.g., the electronic device <NUM> of <FIG>) may provide an improved defect detection technique.

<FIG> illustrates an example of a defect inspection method according to one or more embodiments.

Referring to <FIG>, operations <NUM> and <NUM> may be performed sequentially. However, examples are not limited thereto. For example, two or more operations may be performed in parallel.

In operation <NUM>, a processor (e.g., the processor <NUM> of <FIG>) may obtain an enhanced image (e.g., the enhanced image <NUM> of <FIG>) in which a defective area included in an input image (e.g., the input image <NUM> of <FIG>) is emphasized, based on the input image <NUM> and a reference image (e.g., the reference image <NUM> of <FIG>). The enhanced image <NUM> may be obtained from a processor (e.g., the processor <NUM> of <FIG>) configured to receive the input image <NUM> and the reference image <NUM> and output an image in which a defective area is emphasized. The processor <NUM> may obtain a differential image based on the input image <NUM> and the reference image <NUM> and output the enhanced image <NUM> by adjusting an intensity of a defective area included in the differential image.

In operation <NUM>, the processor <NUM> may output a defect prediction score (e.g., the defect prediction score <NUM> of <FIG>) of the input image by inputting the reference image <NUM>, the input image <NUM>, and the enhanced image <NUM> into a neural network (e.g., the neural network <NUM> of <FIG>). The neural network <NUM> may include an attention map modulator (e.g., the attention map modulator <NUM> of <FIG>) configured to adaptively adjust an intensity (e.g., a contrast) of an attention map.

The neural network <NUM> may include a feature extractor (e.g., the feature extractor <NUM>) and be configured to receive the reference image <NUM> and the input image <NUM> and extract a feature map (e.g., the feature map <NUM> of <FIG>). The neural network <NUM> may also include an attention mechanism (e.g., the attention mechanism <NUM> of <FIG>) configured to receive the feature map <NUM> and the enhanced image <NUM> and to output a modulated feature map (e.g., the modulated feature map <NUM> of <FIG>).

In a non-limiting example, the attention mechanism <NUM> may be trained to increase weights of defect-associated values included in the feature map <NUM>. The attention mechanism <NUM> may include an attention map generator (e.g., the attention map generator <NUM>-<NUM> of <FIG>) configured to receive the enhanced image <NUM> and to output a modulated attention map (e.g., the modulated attention map <NUM> of <FIG>). The attention mechanism <NUM> may include a feature modulator (e.g., the feature modulator <NUM>-<NUM> of <FIG>) configured to receive the modulated attention map <NUM> and the feature map <NUM> and output the modulated feature map <NUM>.

In an example, the attention map generator <NUM>-<NUM> may include a generator (e.g., the generator <NUM> of <FIG>) configured to receive the enhanced image <NUM> and to output an attention map <NUM>. The attention map generator <NUM>-<NUM> may also include an attention map modulator (e.g., the attention map modulator <NUM>) configured to receive the attention map <NUM> and to output the modulated attention map <NUM>.

In an example, the feature modulator <NUM>-<NUM> may output the modulated feature map <NUM> by applying the modulated attention map <NUM> to the feature map <NUM>. In another example, the feature modulator <NUM>-<NUM> may output the modulated feature map <NUM> by applying the modulated attention map <NUM> to a portion of the feature map (e.g., the portion <NUM>-<NUM> of the feature map of <FIG>) and concatenating a result (e.g., the modulated feature map <NUM> of <FIG>) of applying the modulated attention map and a remaining portion of the feature map (e.g., the remaining portion <NUM>-<NUM> of the feature map of <FIG>). In a non-limiting example, the feature modulator <NUM>-<NUM> may perform an elementwise operation on at least a portion of the feature map <NUM> and the modulated attention map <NUM>.

The neural network <NUM> may further include a defect classifier (e.g., the defect classifier <NUM> in <FIG>) configured to receive the modulated feature map <NUM> and output a defect prediction score of the input image <NUM>.

The electronic device, processors, memories, neural networks, processor <NUM>, memory <NUM>, electronic device <NUM>, and processor <NUM>, as well as other components described and disclosed herein and described with respect to <FIG> are implemented by or representative of hardware components. As described above, or in addition to the descriptions above, examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. As described above, or in addition to the descriptions above, example hardware components may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, and thus, not a signal per se. As described above, or in addition to the descriptions above, examples of a non-transitory computer-readable storage medium include one or more of any of read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD- Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks , and/or any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the scope of the appended claims. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.

Claim 1:
A processor-implemented method, the method comprising:
generating an enhanced image (<NUM>) based on an input image (<NUM>) and a reference image (<NUM>), in which a defective area is emphasized;
generating a defect prediction score (<NUM>) of the input image (<NUM>) through the use of a neural network that is provided with the reference image (<NUM>), the input image (<NUM>), and the enhanced image (<NUM>),
wherein the neural network comprises:
a feature extractor (<NUM>) configured to receive the reference image (<NUM>) and the input image (<NUM>) and to extract a feature map (<NUM>); characterized in that the neural network further comprises
an attention mechanism (<NUM>) configured to receive the feature map (<NUM>) and the enhanced image (<NUM>) and to output a modulated feature map (<NUM>), said attention mechanism (<NUM>) comprising:
a generator (<NUM>) configured to receive the enhanced image (<NUM>) and to output an attention map (<NUM>);
an attention map modulator (<NUM>) configured to adaptively adjust an intensity of the attention map (<NUM>) generated during the use of the neural network; and
a feature modulator (<NUM>-<NUM>) configured to receive the modulated attention map (<NUM>) and the feature map (<NUM>) and to output the modulated feature map (<NUM>).