Adversarial patches including pixel blocks for machine learning

Systems, apparatuses, and methods are directed towards identifying that an adversarial patch image includes a plurality of pixels. The systems, apparatuses, and methods include dividing the adversarial patch image into a plurality of blocks, that each include a different group of the pixels in which the pixels are contiguous to each other, and assigning a first plurality of colors to the plurality of blocks to assign only one of the first plurality of colors to each pixel of one of the plurality of blocks.

FIELD OF THE DISCLOSURE

This disclosure generally relates to methods, systems, apparatuses, and computer readable media for generation of adversarial patches to train deep learning models, neural network models and/or for machine learning applications.

BACKGROUND

Machine learning systems (e.g., neural networks or deep learning systems) may suffer from small guided adversarial perturbations. The adversarial perturbations may be intentionally inserted to robustly train the machine learning systems. That is, a more robust machine learning system may be generated by testing the machine learning system against adversarial patches. The machine learning system may misclassify an image (that includes the adversarial patch) and then the machine learning system retrains to eliminate or minimize such misclassifications.

SUMMARY

Consistent with the disclosure, exemplary embodiments of systems, apparatuses, and methods thereof for generating enhanced adversarial patches, are disclosed.

According to an embodiment, an adversarial patch generation system, includes a memory, and a processor coupled to the memory, wherein the processor is configured to identify that an adversarial patch image includes a plurality of pixels, divide the adversarial patch image into a plurality of blocks that each include a different group of the pixels in which the pixels are contiguous to each other, and assign a first plurality of colors to the plurality of blocks to assign only one of the first plurality of colors to each pixel of one of the plurality of blocks.

In an embodiment of the system, the processor is to assign a first color of the first plurality of colors to a first block of the plurality of blocks, and modify each pixel of the group of the pixels of the first block to be the first color.

In an embodiment of the system, the processor is to execute an iterative process to during a first iteration, assign a second color to the first block, during a second iteration, change the second color to a third color and assign the third color to the first block, and during a final iteration, change the third color to the first color to assign the first color to the first block.

In an embodiment of the system, the processor is to for each respective block of the plurality of blocks, modify the group of the pixels of the respective block to be a same color of the first plurality of colors that is assigned to the respective block.

In an embodiment of the system, the processor is to execute an iterative process to generate first pixel values for the plurality of pixels during a final iteration, wherein the first pixel values are to be associated with the first plurality of colors.

In an embodiment of the system, the processor is to execute the iterative process to generate during a previous iteration before the final iteration, second pixel values for the plurality of pixels, wherein the second pixel values correspond to a second plurality of colors, and generate the first pixel values based on the second pixel values.

In an embodiment of the system, the processor is to identify, during the final iteration, a first loss that is a measure that the adversarial patch image having the first pixel values causes a machine learning system to misclassify an object, identify, during the previous iteration, a second loss that is a measure that the adversarial patch image having the second pixel values causes a machine learning system to misclassify the object, and determine that the adversarial patch image is to have the first pixel values and not the second pixel values based on an identification that first loss meets a threshold and the second loss does not meet the threshold.

In an embodiment, a method includes identifying that an adversarial patch image includes a plurality of pixels, dividing the adversarial patch image into a plurality of blocks that each include a different group of the pixels in which the pixels are contiguous to each other, and assigning a first plurality of colors to the plurality of blocks to assign only one of the first plurality of colors to each pixel of one of the plurality of blocks.

In an embodiment of the method, further includes assigning a first color of the first plurality of colors to a first block of the plurality of blocks, and modifying each pixel of the group of the pixels of the first block to be the first color.

In an embodiment of the method, further includes executing an iterative process that includes during a first iteration, assigning a second color to the first block, during a second iteration, changing the second color to a third color and assigning the third color to the first block, and during a final iteration, changing the third color to the first color to assign the first color to the first block.

An embodiment of the method further includes for each respective block of the plurality of blocks, modifying the group of the pixels of the respective block to be a same color of the first plurality of colors that is assigned to the respective block.

An embodiment of the method further comprises executing an iterative process to generate first pixel values for the plurality of pixels during a final iteration, wherein the first pixel values are to be associated with the first plurality of colors.

In an embodiment of the method, the executing the iterative process includes generating during a previous iteration before the final iteration, second pixel values for the plurality of pixels, wherein the second pixel values correspond to a second plurality of colors, and generating the first pixel values based on the second pixel values.

In an embodiment, a non-transitory computer readable medium includes a set of instructions, which when executed by one or more processors of a device, cause the one or more processors to identify that an adversarial patch image includes a plurality of pixels, divide the adversarial patch image into a plurality of blocks that each include a different group of the pixels in which the pixels are contiguous to each other, and assign a first plurality of colors to the plurality of blocks to assign only one of the first plurality of colors to each pixel of one of the plurality of blocks.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to assign a first color of the first plurality of colors to a first block of the plurality of blocks, and modify each pixel of the group of the pixels of the first block to be the first color.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to execute an iterative process to, during a first iteration, assign a second color to the first block, during a second iteration, change the second color to a third color and assign the third color to the first block, and during a final iteration, change the third color to the first color to assign the first color to the first block.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to for each respective block of the plurality of blocks, modify the group of the pixels of the respective block to be a same color of the first plurality of colors that is assigned to the respective block.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to execute an iterative process to generate first pixel values for the plurality of pixels during a final iteration, wherein the first pixel values are to be associated with the first plurality of colors.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to execute the iterative process to generate during a previous iteration before the final iteration, second pixel values for the plurality of pixels, wherein the second pixel values correspond to a second plurality of colors, and generate the first pixel values based on the second pixel values.

In an embodiment of the non-transitory computer readable medium, the set of instructions, which when executed by the one or more processors, cause the one or more processors to identify, during the final iteration, a first loss that is a measure that the adversarial patch image having the first pixel values causes a machine learning system to misclassify an object, identify, during the previous iteration, a second loss that is a measure that the adversarial patch image having the second pixel values causes a machine learning system to misclassify the object, and determine that the adversarial patch image is to have the first pixel values and not the second pixel values based on an identification that the first loss meets a threshold and the second loss does not meet the threshold.

DESCRIPTION OF THE EMBODIMENTS

The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a machine readable (e.g., computer-readable) medium or machine-readable storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

A training process of a machine learning architecture may include capturing images of a target and feeding the captured images to the machine learning architecture. The machine learning architecture may then identify a probability that the captured images include a particular object (e.g., a stop sign, human being, stop light, right turn sign, etc.). The machine learning architecture may then be trained (e.g., supervised, unsupervised or semi-supervised learning) to increase correct identification of objects in the captured images.

Some machine learning architectures may appear to accurately detect objects during test settings. In real-world environments, the same machine learning architectures may be easily “fooled” into misclassifying an object. For example, some machine learning architectures may misclassify objects based on unforeseen environmental conditions (e.g., foggy, rainy, bright or low-light conditions). In some cases, nefarious actors may intentionally place physical patches over objects to cause machine learning architectures to misclassify the objects. For example, an actor may place a patch over a stop sign to cause a machine learning architecture to misclassify the stop sign as a speed limit sign.

Therefore, it may be beneficial to test the robustness of machine learning architectures prior to public use and release of the machine learning architectures. For example, a machine learning model may be tested against an adversarial patch. If the adversarial patch causes the machine learning model to misclassify an object, the machine learning model may need to be retrained with the adversarial patch. Thus, some embodiments may include an enhanced adversarial patch generation system to generate adversarial patches to be used to train and test the robustness of machine learning architectures. For example, some embodiments may generate adversarial patches that, during testing or training, cause machine learning architectures to misclassify an image to a desired target class despite variations in positions of the adversarial patches, camera angles and camera distances.

Some embodiments of the adversarial patch generation system may further modify characteristics (e.g., intensity, color, etc.) of blocks of contiguous pixels collectively rather than each individual pixel. Doing so may be more efficient than an individual analysis and modification of each pixel. Furthermore, the resulting adversarial patch that is generated through modification of blocks of pixels may be more likely to cause misclassification when captured through an imaging device than an adversarial patch that is generated through a pixel-by-pixel modification. For example, the resulting adversarial patch may include an easily discernible pattern by an average camera and that is able to be cause misclassifications from different angles, rotations and distances.

The enhanced adversarial patch generation system may result in several technical advantages, including raised testing standards to enhance efficiency, robustness and safety. For example, the adversarial patches described herein may be used to test the machine learning architectures to increase reliability of the machine learning architectures and determine the effectiveness of the machine learning architectures. Therefore, it is less likely that poor performing machine learning architectures will be publicly released. Doing so may lower the probability that the machine learning architectures will misclassify targets when used in operation, enhancing safety and robustness. Moreover, modifying contiguous blocks of pixels rather than each individual pixel results in a more efficient process.

Turning now toFIG. 1A, an enhanced adversarial patch100is illustrated. The enhanced adversarial patch100may include a series of blocks that are each M pixels by M pixels. The pixels of each block may be contiguous with each other. In the example ofFIG. 1A, the blocks are illustrated as having different patterns. Each different pattern may correspond to a different color. Identical patterns may be a same color. Within each block, each of the pixels of the block may be a same color. For example, characteristics of the pixels of a block may be identical aside from position. Thus, each block may be entirely one color so that a first block is a first color, a second block is a second color, a third block is a third color, etc. Therefore, the enhanced adversarial patch100may be consistently colored for blocks of pixels to avoid and/or reduce concern of physical world transformation effects on the patch. As a result, each respective pixel in the enhanced adversarial patch100will be a same color as at least one other pixel that is contiguous to the respective pixel. As described below, an adversarial patch generation system may apply an iterative process that includes forward-propagation and backward-propagation to generate the enhanced adversarial patch100.

FIG. 1Ashows a conventional patch102as well. The conventional patch102may be generated through a pixel-by-pixel analysis. That is, there is no requirement that contiguous blocks of pixels are the same color. Rather, each pixel may be sequentially analyzed to determine a color for the pixel, which may be different than all pixels contiguous to the pixel. Such a process is time-consuming, requires increased power relative to generation of the block-colored enhanced adversarial patch100and is less effective at causing machine learning models to misclassify. For example, the conventional patch102may not cause a machine learning model to misclassify at different zoom levels of an image capture device, different image capture angles of an image capture device relative to the conventional patch102and at different positions of the conventional patch102relative to an object. Moreover, different environmental conditions (e.g., bright light, low light, reflections, etc.) may further deteriorate the effectiveness of the conventional patch102to cause misclassification. In contrast, the enhanced adversarial patch100may accurately cause misclassifications regardless of characteristics of the image capture device and environmental conditions.

FIG. 1Billustrates a real-world scenario106of machine learning model (e.g., neural network) classifications of an image and the enhanced adversarial patch100. In the example ofFIG. 1B, the image is a stop sign104. A series of Tests 1-4 are illustrated. In each of Tests 1-4, an imaging device112(discussed below) captures an image of the enhanced adversarial patch100and the stop sign104. The machine learning model analyzes the images to identify and/or classify objects in the images. The machine learning model classifications are presented inFIG. 1. In some embodiments, the machine learning model may be trained and/or retrained using the adversarial patch100.

As illustrated, in Test 1, the enhanced adversarial patch100causes the machine learning model to misclassify the stop sign104. In detail, the machine learning model misclassifies the stop sign104as likely (82%) being a 100 KM speed sign.

In Test 2 the enhanced adversarial patch100causes the machine learning model to misclassify the stop sign104as likely (79%) being a 100 KM speed sign. As illustrated, the enhanced adversarial patch100is at a same position relative to the stop sign104in Test 1 and Test 2. In Test 2 however, the imaging device112has zoomed out and/or is positioned farther away from the stop sign104and the adversarial patch100. Thus, in the image of Test 2, the stop sign104and the adversarial patch100appear smaller than in the image of Test 1. Nonetheless, the adversarial patch100still causes the machine learning model to misclassify the stop sign104.

In Test 3 the enhanced adversarial patch100causes the machine learning model to misclassify the stop sign104as likely (88%) being a 100 KM speed sign. As illustrated, the enhanced adversarial patch100is at a same position relative to the stop sign104in Test 1, Test 2 and Test 3. In Test 3 however, the imaging device112has rotated relative to the stop sign104and the adversarial patch100. Thus, in the image of Test 3, the stop sign104and the adversarial patch100appear rotated. Nonetheless, the adversarial patch100still causes the machine learning model to misclassify the stop sign104.

In Test 4 the enhanced adversarial patch100causes the machine learning model to misclassify the stop sign104as likely (92%) being a 100 KM speed sign. As illustrated the enhanced adversarial patch100is moved and rotated relative to the stop sign104in Test 1, Test 2 and Test 3. Nonetheless, the adversarial patch100still causes the machine learning model to misclassify the stop sign104.

Thus, regardless of the position and rotation of the adversarial patch100relative to the stop sign104, and the rotation, position and zoom of the imaging device112relative to the stop sign104and/or the adversarial patch100, the machine learning model still misclassifies the stop sign104as likely being a 100 KM sign. As such, the adversarial patch may be used to test robustness of machine learning models and/or train the machine learning models. In contrast, the conventional patch102may cause misclassification under some circumstances (e.g., Test 1), but not cause misclassification under other circumstances (Tests 2, 3 and 4). That is, the conventional patch102may not cause misclassification at certain distances, camera angles, camera rotations and/or environmental conditions.

The above enhanced adversarial patch100has adversarial patterns that are simple enough to be detected from different distances and will remain robust (detectable and cause misclassification) with respect to varying cameras distances and/or angles. In some embodiments, the adversarial patch100has large adversarial perturbations in the subparts of an image by choosing random subparts, scale and rotations.

Minute pixel level changes such as those of conventional patch102are not captured with enough detail to cause misclassification. As such, a method (discussed below) of forward-propagation to generate a patch from a block of pixels is introduced. The forward-propagation method forces a contiguous M×M block of pixels to have a same value (e.g., color and/or intensity), which decreases the visual complexity of the enhanced adversarial patch. The enhanced adversarial patch100may include simple enough patterns to be classified to a particular target class, even when the image is viewed at varying distances and angles. While other adversarial examples may not robustly transfer to the physical world, some embodiments of the present application generate adversarial patches that are simple enough to avoid physical world transformation effects on the enhanced adversarial patch100.

FIG. 1Cillustrates different positions110a,110b,110cand/or rotations of the imaging device112relative to the stop sign104and the adversarial patch100. The lines of sight108a,108b,108cbetween the imaging device112and the stop sign104and/or the adversarial patch100are illustrated. As described above, the machine learning model may misclassify the stop sign104in each position of the imaging device112due to the influence of the adversarial patch100. The imaging device112may be any type of suitable imaging device (e.g., mobile camera, sensor array, night vision, etc.).

Each of the apparatus, methods and/or processes described herein may be implemented in hardware, in software, or in some combination thereof. For example, each of the method blocks described herein may be implemented in configurable logic such as, for example, programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), or in fixed-functionality logic hardware using circuit technology such as, for example, application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof Alternatively or additionally, these components may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as random access memory (RAM), read only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., to be executed by a processor or computing device. For example, computer program code to carry out the operations of the modules may be written in any combination of one or more operating system applicable/appropriate programming languages, including an object oriented programming language such as JAVA, SMALLTALK, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.

FIG. 2illustrates a method200of generating a color pattern for an adversarial patch. Processing block202may identify that an adversarial patch image includes a plurality of pixels. Processing block204may divide the adversarial patch image into a plurality of blocks that each include a different group of the pixels in which the pixels are contiguous to each other. Processing block206may assign a first plurality of colors to the plurality of blocks to assign only one of the first plurality of colors to each pixel of one of the plurality of blocks. The colors may be different from each other. Processing block206may include assigning a first color of the first plurality of colors to a first block of the plurality of blocks, and modifying each pixel of the group of the pixels of the first block to be the first color. Processing block206may further include for each respective block of the plurality of blocks, modifying the group of the pixels of the respective block to be a same color of the first plurality of colors that is assigned to the respective block.

FIG. 3shows an iterative process300of generating final colors302efor an adversarial patch. It will be understood that adversarial patch may include an adversarial patch image. In Iteration Zero, the adversarial patch is initialized to include all pixels at a same color302a.Iteration Zero may further include dividing the adversarial patch into a plurality of blocks that each include a group of contiguous pixels (e.g., each block may be M×M pixels). The boundaries of the blocks may remain the same throughout the process300.

As illustrated, the blocks of the adversarial patch in Iteration Zero are a same color302a, but in some embodiments the blocks of the adversarial patch may be initialized to different, random colors. For example, the patch may be initialized randomly. In some embodiments, the adversarial patch may be randomly initialized so that each block may have a different color.

The process300may then modify the colors304in Iteration One. In detail, the process300may modify the color302ato generate first colors302bfor the adversarial patch. For example, the colors302aare modified to various different first colors302b.As described above, the different patterns represent different colors and/or characteristics of the pixels of the blocks, such as intensity. Thus, different first colors302bmay be written into the different blocks of the adversarial patch. For example, one block may be blue, a second red, a third yellow, etc.

To enhance the reliability and accuracy of misclassification, the process300may test to make sure that misclassification is increasing. For example, the process300may superimpose the adversarial patch having the color302aover a target image through software and/or hardware processes. A machine learning model may then classify the target image with the superimposed initial adversarial patch having the color302a. The process300may record the classification. As an example, suppose that the target image is an image of a stop sign. The process300would superimpose the adversarial patch having the color302aover the stop sign to generate a first classification image and the machine learning model would classify the first classification image. For example, the machine learning model may classify the first classification image, for example as being 95% a stop sign. In contrast, if the adversarial patch having the color302awas not superimposed over the stop sign, the machine learning model may classify the stop sign as being 99% a stop sign.

The process300may identify whether misclassification (loss) increases. For example, an increase in misclassification may occur when the machine learning model reduces a probability that the target image belongs to a correct category, increases a probability that the target image belongs to an incorrect category and/or identifies the target image as being in an incorrect category. For example, in the above example, a probability of misclassification increases with the superimposition of the adversarial patch having the color302aover the stop sign because the probability of a correct classification decreases from 99% to 95%.

Process300may then modify the colors304in Iteration One to increase the misclassification. For example, the modification of the colors304may include a determination of whether the first colors302bincrease misclassification as described above. For example, the machine learning model may classify an image that includes the adversarial patch having the first colors302bsuperimposed on the target image. Continuing with the above example, suppose that the machine learning model classifies an image that includes the stop sign and the adversarial patch, that has the first colors302b, as being 100 KM sign. In the present example, the misclassification increases, and therefore the first colors302bare stored as the new colors.

The process300may continue to modify colors306,308,310,312to generate colors302c,302d,302euntil the last iteration to increase misclassification. In some embodiments, the number of iterations is predetermined. At the last iteration, the process300may identify that there are no further modifications that may increase misclassification, and adversarial patch is to include the last colors302e. In some embodiments, the process300may conclude that the misclassification has reached a threshold of misclassification (e.g., the target image is 95% classified into an incorrect category) to determine that no more iterations are necessary and the last colors302eare to form the final adversarial patch. For example, if the machine learning model misclassifies an image that includes the target image and the adversarial patch with the first colors302bat a probability that meets a threshold, the process300may determine that no further iterations are needed.

As a more detailed example, the threshold may be set to 95% misclassification. If the target image is a stop sign and the adversarial patch with the last colors302ecauses the machine learning model to classify the stop sign as a 100 KM sign with 95% confidence, the threshold has been met. If however the machine learning model classified the stop sign as a 100 KM sign with 94% confidence, the threshold may not have been met. In some embodiments, if the process300determines that the threshold is unable to be met, the last colors302emay still be output as assigned to the adversarial patch along with a notification that the adversarial patch does not meet the threshold.

In some embodiments, the adversarial patch having the last colors302emay be used to train a machine learning model through computer implemented modifications of existing images to superimpose the adversarial patch of the images. In some embodiments, the adversarial patch having the last colors302emay be physically printed to train a machine learning model.

FIG. 4illustrates a process400to generate pixel values for an adversarial patch.FIG. 4may be utilized for example in the process300ofFIG. 3, or any of the methods or apparatuses described herein. For simplicity, process400uses grayscale with each the values of the various matrices indicative of intensity. It will be understood however that process400is applicable to different representations of color (e.g., R, G, B values). In such different representations, the process400may include various matrices that each correspond to a different component (e.g., one matrix for red, one for blue, one for green, etc.) of the representation. The matrices of process400may correspond to pixels. In cases where different matrices are used to represent the different representations of color (e.g., R, G, B values), then the process400is repeated each time for each matrix.

The process400may force contiguous m×m pixels to have the same characteristics to generate patches that are less complex and simple enough to be captured well by low quality cameras (e.g., camera phones). As will be discussed in further detail below, process400may modify colors through a smaller patch ps402. The smaller patch psmay have dimensions H/m×W/m if the adversarial patch is intended to be of dimensions H×W. From small patch ps402the process400may expand the small patch to a large patch404to obtain a large (e.g., blown up) patch p1406of dimensions H×W by copying each pixel of smaller version ps402m×m times to create blocks of colors in the adversarial patch. In some embodiments, process400may use forward-propagation to convert the smaller patch psinto the larger patch pl.

Thus, in some examples the small patch ps402is a subset of the large patch pl406. The large patch pl406may include similar values in a plurality of quadrants to correspond to colors in various blocks of contiguous pixels. The large patch pl406may correspond to pixels of the adversarial patch. That is, each value in the matrix of the large patch pl406may correspond to a pixel of the adversarial patch. For example, the matrix may approximate x-y coordinates of pixels so that a position of an element corresponds to an x-y coordinate of a pixel and represents an intensity at that x-y coordinate. Pixels which are contiguous may be represented by adjacent values in the matrix.

Process400may then modify the values in the large patch pl406to trend towards increased loss through gradient descent, and identify changes in pixel values408. The loss may be a measure of incorrectness of classification of a target image that includes an adversarial patch by a machine learning model. Thus, as the misclassification increases, the loss increases to decrease the probability of a correct classification and/or increase the probability of an incorrect classification. A difference or an approximate difference between the modified values and the original values in the large patch pl406is calculated and stored in a difference patch pd410. That is, an amount each element from the large patch pl406is modified to generate the modified values is stored in the difference patch pd410.

To identify loss, the process400may randomly cover a part of a target image with an adversarial patch that includes the modified values at a random location, scale and rotations. For example, process400may modify the target image (e.g., a software and/or hardware implemented modification) to include the adversarial patch. The process400may thus update the large patch plso as to minimize cost with respect to a target class using gradient descent to generate the modified values.

For example, suppose T is a transformation function that takes the large patch version pl, image x, location loc, scaling s and rotation r and gives a transformed image {circumflex over (x)}=T(pl, x, loc, s, r), such that {circumflex over (x)} is obtained after applying the patch plscaled by s and rotated by r at location loc of the image x. Some embodiments may obtain a patch {circumflex over (p)}, that maximizes the probability of target class ŷtover the data sampled using the above transformation function T Some embodiments therefore optimize the following equation 1, where X is training data, R is the set of rotations in range [−20 degree, 20 degree], S is the set of scales i.e. the fraction of image covered by patch, and L is the set of locations possible in the image x
{circumflex over (p)}=arg maxpx∈X,r∈R,s∈S,loc∈L[logPr(ŷt|T(pl, x, loc,r, s))]  EQUATION 1

The process400may continue to use the differences from the difference patch pd410to calculate corresponding small patch differences412and store the differences in a small difference patch414. For example, values associated with one block may be added together and stored in the small difference patch414at corresponding locations. That is, the four values (0.2, 0.1, 0.4, −0.3) from the upper left quadrant may correspond to a first block, are added together to determine a first value (0.4) and the first value is stored in the upper left position of the small difference patch414. The four values (1.7, 2.4, 0.1, −0.4) from the upper right quadrant may correspond to a second block, are added together to determine a second value (3.8) and the second value is stored in the upper right position of the small difference patch414. The four values (0.3, −0.6, 0.3, 1.1) from the lower left quadrant may correspond to a third block, are added together to determine a third value (1.1) and the third value is stored in the lower left position of the small difference patch414. The four values (0.2, 0.2, −0.4, 0.8) from the lower right quadrant may correspond to a fourth block, are added together to determine a fourth value (0.8) and the fourth value is stored in the lower right position of the small difference patch414. The first-fourth blocks are different blocks of pixels of the adversarial patch.

In some embodiments, a small change in settings of large patch generation may change the way that the small patch differences are calculated. Thus, some embodiments may automatically utilize the chain rule of derivatives to convert from large patch differences to small patch differences.

Process400may then update the small patch416to generate an updated small patch418. For example, the process400may update the small patch ps402patch based on the first-fourth values in the small difference patch414. In the present example, the first-fourth values may be multiplied by a constant (e.g., 0.1) and subtracted from corresponding values in the small patch ps402patch that are in the same position and/or correspond to the same block as the first-fourth values. The process418may repeat420using the values in the updated small patch418.

FIG. 5illustrates a method500of a flow of generating an adversarial patch. Illustrated processing block502initializes a small patch ps. The patch initialization may be determined according to the following algorithm 1:

Illustrated processing block504forward-propagates the small patch to generate a larger patch. For example, processing block504may generate an enlarged patch using the below forward-prop algorithm 2, and then generates the patched training data XB* using the transformation function T.

In some embodiments, processing block504follows algorithm 2 to execute the forward-propagate process:

Illustrated processing block506evaluates how much a loss changes by changing the large patch pixels and identifies a difference between original pixel values and modified pixel values to increase the loss. For example, processing block506may then compute a derivative of Loss L with respect to (w.r.t.) the enlarged patch, where L =loss(M(XB*), t). The loss may be a cross-entropy loss between the prediction M(XB*) with respect to a target label t. M(x) is the softmax-prediction of Model M for input x.

Illustrated processing block508calculates how much the loss changes by changing small patch pixels using backward-propagation. For example, processing block508may compute a derivative of loss with respect to small patch psusing an algorithm 3 (back-prop) and update the small patch psin the direction of negative gradient thus obtained. In some embodiments, processing block508follows algorithm 3 below to execute the backward-propagation operation:

Processing block510updates the small patch based on the change in loss calculated by block508. Processing block512may determine whether any further iterations are needed (e.g., whether loss should be increased). If not, processing block514generates a final large patch from the updated small patch. If further iterations are needed, processing block504may execute again.

FIG. 6illustrates a computer-implementable graphical user interface600according to some embodiments. Input602may allow a user to upload a sample folder of target images (e.g., stop signs, ostrich images, 100 KM signs) that are to be misclassified by a machine learning model (e.g., neural network). Input604allows the user to upload a machine learning model that is to be caused to misclassify the target images. Input606may allow the user to select a particular misclassification. So for example, if the target images are 100 KM signs, the user may select that the patch class to be stop signs. In other words, the user may select that the machine learning model is to be caused to misclassify the 100 KM signs as stop signs. Thus, inputs602,604,606are user modifiable inputs.

Based on the inputs604,606,602, a patch may be generated to achieve the user's objectives. In the present example, area612illustrates the original image which are tractors. Area610illustrates the patch overlaid on the images. As is apparent from area610, the tractor was unable to be misclassified and so area610provides a warning along with a patch that includes colors most likely to drop probability of correct classification. The patch is shown in detail at area608. Area614illustrates the effectiveness of adversarial patches based on a comparison of the machine learning network classifying the images without the adversarial patch and with the adversarial patches.

Referring now toFIG. 7, an exemplary computing device800(e.g., an intermediate level server122) for generating the adversarial patch100and performing the process106ofFIGS. 1A, 1B and 1C, method200ofFIG. 2, process300ofFIG. 3, process400ofFIG. 4, method500ofFIG. 5, and the graphical user interface600ofFIG. 6. The computing device800may include a processor804, a memory810, a data storage814, a communication subsystem806(e.g., transmitter, receiver, transceiver, etc.), and an I/O subsystem812. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory810, or portions thereof, may be incorporated in the processor804in some embodiments. The computing device800may be embodied as, without limitation, a mobile computing device, a smartphone, a wearable computing device, an Internet-of-Things device, a laptop computer, a tablet computer, a notebook computer, a computer, a workstation, a server, a multiprocessor system, and/or a consumer electronic device.

The processor804may be embodied as any type of processor capable of performing the functions described herein. For example, the processor804may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit.

The memory810may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory810may store various data and software used during operation of the computing device800such as operating systems, applications, programs, libraries, and drivers. The memory810is communicatively coupled to the processor804via the I/O subsystem812, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor804the memory810, and other components of the computing device800.

The data storage device814may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, non-volatile flash memory, or other data storage devices. With respect to generation of adverbial patches and machine learning models, the data storage device814may store data (e.g., computer code) to execute the processes and methods described herein. Alternatively, such data may be stored remotely. In some embodiments, the processor804or other hardware components may be configured to execute the processes and methods. Regardless, the computing device800may identify that an adversarial patch image includes a plurality of pixels, divide the adversarial patch image into a plurality of blocks that each include a different group of the pixels in which the pixels are contiguous to each other, and assign a first plurality of colors to the plurality of blocks.

The computing device800may also include a communications subsystem806, which may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the computing device800and other remote devices over a computer network (not shown). The communications subsystem806may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, Bluetooth®, WiMAX, LTE, etc.) to affect such communication.

As shown, the computing device800may further include one or more peripheral devices816. The peripheral devices816may include any number of additional input/output devices, interface devices, and/or other peripheral devices. For example, in some embodiments, the peripheral devices816may include a display, touch screen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and/or other input/output devices, interface devices, and/or peripheral devices. The computing device800may also perform one or more of the functions described in detail above and/or may store any of the databases referred to below.

It will be understood that the foregoing description is applicable to any size image or adversarial patches. For example, the adversarial patches may include any number of blocks and any number of pixels.

The methods shown herein may generally be implemented in a computing device or system. The computing device or system may be a user level device or system or a server-level device or system. More particularly, the methods may be implemented in one or more modules as a set of logic instructions stored in a machine or computer-readable storage medium such as random access memory (RAM), read only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as, for example, programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), in fixed-functionality logic hardware using circuit technology such as, for example, application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.

For example, computer program code to carry out operations shown in the methods and processes of any of the figures herein may be written in any combination of one or more programming languages, including an object-oriented programming language such as JAVA, SMALLTALK, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. Additionally, logic instructions might include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, and/or other structural components that are native to hardware (e.g., host processor, central processing unit/CPU, microcontroller, etc.).

Example sizes/models/values/ranges may have been given, although embodiments are not limited to the same. Where specific details are set forth in order to describe example embodiments, it should be apparent to one skilled in the art that embodiments can be practiced without, or with variation of, these specific details. The description is thus to be regarded as illustrative instead of limiting.