Patent ID: 12254077

DETAILED DESCRIPTION

FIGS.1-3illustrate systems and techniques for generating artificial intelligent (AI)-resistant verification images using adversarial perturbations in an image-based CAPTCHA. The AI-resistant verification images are generated using a pixel-by-pixel bounds on each pixel of a source image that places a minimum and maximum threshold value on an adversarial perturbation that is added to each pixel. The adversarial perturbations that are generated using the pixel-by-pixel bounds are based on the limitations of human visual perception (e.g., a “Weber-Fechner” relation described in detail with reference toFIGS.1-3) and AI image recognition limitations that dictate the perturbation range of the adversarial perturbations.

As used herein, the term verification image refers to an image-based mechanism by which a query is issued to elicit an answer from an end user that enables a determination of whether the end user is a human. The adversarial perturbation is defined as a change or distortion in the image data of the verification image that is deliberately designed to prevent a machine-learning or AI system from correctly classifying the image. The adversarial perturbation that is added to the source image by the data processing system forces the AI system to misclassify the image (i.e., to mistake the image for a different image). The adversarial perturbations thereby prevent the AI network from selecting the correct image while being visually imperceptible to the human eye.

FIG.1illustrates an exemplary network environment100for implementing an image-based CAPTCHA using artificial intelligence resistant verification images in accordance with some embodiments. In various embodiments, network environment100includes a computing device106, a network140, a server110, and an image database130. In some embodiments, the server110provides web pages, web services, and associated data to one or more remote clients via the network140. At least some of these services and associated data are specified to be secure services and data that are to be secured from unauthorized access. For example, in some embodiments the server110provides banking or other financial services to remote users via the network140, wherein the banking and financial data is expected to be secured from unauthorized access.

For the example illustrated inFIG.1, the computing device106is configured to employ a malicious entity169, such as malicious software executing at computing device106, that utilizes an AI unit169to attempt to gain unauthorized access to the secure services and data of server110. Computing device106includes a display for displaying an AI-resistant image-identification CAPTCHA144. Although for the embodiment ofFIG.1, the computing device106is depicted as a desktop computer with a display, in some embodiments, computing device106is a headless server that communicates with computers and other devices via a network, a cellular-phone, a tablet, or other computing device that is used to access software services and applications from server110via network140. In various embodiments, network140is any type of communication network, such as, for example, wireless networks (e.g., cellular or satellite), wire-based networks (e.g., cable), cellular telecommunication networks, and IP-based telecommunication network(s) (e.g., Voice over Internet Protocol networks). In other implementations, the network(s)140also include traditional landline networks, or combinations thereof.

In some embodiments, AI unit112is an artificially intelligent autonomous program executing on computing device106that, used in conjunction with the malicious entity169, is designed to gain unauthorized access to computer systems on network140(e.g., server110). AI unit112essentially serves as an image classification system that utilizes artificial intelligence to attempt to correctly classify an AI-resistant verification image142presented in AI-resistant image-identification CAPTCHA144. In different embodiments, AI unit112is hardware, software, or a combination thereof, that utilizes convolutional neural network (CNNs), recurrent neural networks (RNN), or other artificially intelligent self-learning software operating program to access and analyze services and applications provided by server110via network140. For example, in some embodiments, the AI unit112is a neural network that has been trained to classify image data provided by servers connected to the network140in order to identify conventional CAPTCHA images. The malicious entity169executing at the computing device110uses the classified images to provide a response (referred to herein as a CAPTCHA response) to access secure services and data.

To illustrate generally, in order to access secure data, malicious entity169at the computing device106sends, via the network140, a request to the server110that stores the secure data. In response, the server110sends a set of CAPTCHA images to the server110for identification. To access the secure data, the malicious entity169must properly identify a specified subset of the CAPTCHA images, such as identifying all the images with a specified feature, and provide a CAPTCHA response indicating the specified subset. The malicious entity169employs the AI unit112to identify the images having the specified feature, and thus to generate the CAPTCHA response to access the secure data. To prevent unauthorized access to the secure data and services from such unauthorized access by the entity employing the AI unit112, server110includes an AI-resistant CAPTCHA generator195configured to generate the AI-resistant image-identification CAPTCHA144.

During an attempt to gain unauthorized access to server110by computing device106, server110receives, via network140, an autonomous service request from computing device106to utilize services or applications provided by server110. In various embodiments, the autonomous service request is from the malicious entity169that employs AI-unit112of computing device106. For example, in different embodiments the autonomous service request is a request to access secure data (e.g. a web page, user data, and the like) stored at the server110. To prevent the unauthorized access to the secure data, AI-resistant CAPTCHA generator195of server110presents the AI-resistant image-identification CAPTCHA144to computing device106.

In order to present the AI-resistant image-identification CAPTCHA144, AI-resistant CAPTCHA generator195generates an AI-resistant verification image142that is based on a Weber-Fechner relation that limits the perceptibility of an adversarial perturbation that is being added to source image196to generate the AI-resistant verification image142. The Weber-Fechner relation (discussed further in detail below with reference toFIGS.2-3) is a human perception visual relationship that expresses the perceived intensity of the image as being proportional to the logarithm of the actual intensity of the image. The Weber-Fechner relation allows AI-resistant CAPTCHA generator195to generate the AI-resistant verification image142by providing a threshold by which a human cannot perceive the adversarial perturbation that is added to the source image196. That is, the adversarial perturbation is not perceivable by a human user, but is perceivable by the AI unit112.

In order to generate AI-resistant verification image142and the corresponding adversarial perturbation value by which each pixel of the source image196is to be distorted, the AI-resistant CAPTCHA generator195is configured to generate a pixel-by-pixel bounds for each pixel of the source image196. The pixel-by-pixel bounds is a limitation for each pixel of the source image196that represents the maximum and minimum amount by which each pixel can be distorted. The maximum value is calculated by AI-resistant CAPTCHA generator195using source image196and a threshold stimuli, which forms the intensity change threshold by which a human cannot perceive the distortion that is generated using the Weber-Fechner relation. The minimum value of the pixel-by-pixel bound is calculated by the AI-resistant CAPTCHA generator195also utilizing the threshold stimuli of the Weber-Fechner relation that is used to generate the maximum value. The derivation of the minimum and maximum threshold values of the pixel-by-pixel bounds is described further in detail below with reference toFIG.2.

After the pixel-by-pixel bounds has been generated by the AI-resistant CAPTCHA generator195, the AI-resistant CAPTCHA generator195uses a standard adversarial example generation technique, that includes an adversarial perturbation generation technique, to generate an initial value of the adversarial perturbation, and, in some embodiments, a corresponding initial AI-resistant verification image. An adversarial example is an image with intentional pixel adversarial perturbations that causes a classification system to make a false classification. In some embodiments, AI-resistant CAPTCHA generator195uses at least one of the following adversarial perturbation generation techniques to generate the adversarial example: the Fast Gradient Sign Method (FGSM), the Deep Fool Method (DFM), or the Jacobian-based Saliency Map Attack (JSMA). For illustrative purposes of some of the embodiments, the FGSM is used to provide an initial approximation of the adversarial perturbation values for each pixel of the source image196, although, either of the DFM method or JSMA adversarial perturbation generation techniques can be used to determine the initial approximation of the adversarial perturbation values for each pixel of the source image196.

The perturbation values provided by the standard adversarial perturbation generation technique are than adjusted based upon the minimum and maximum values provided by AI-resistant CAPTCHA generator195. Thus, a first adversarial perturbation (and corresponding AI-resistant verification image) is generated using a standard method of generating perturbations (i.e., a standard adversarial perturbation generation technique). Then, using the pixel-by-pixel bounds, the first adversarial perturbation (and corresponding AI-resistant verification image) is adjusted to ensure that the perturbation lies within the minimum value and maximum threshold values to ensure that the perturbation is not visible by a human. Thus, the AI-resistant CAPTCHA generator195relies on the generation of the minimum and maximum threshold values to limit the selection of the adversarial perturbation to perturbation values that generate an AI-resistant verification image142that are both recognizable by a human user of the AI-resistant image-identification CAPTCHA144and prevent the AI unit112from accessing the services provided by server110.

After generation of the AI-resistant verification image142by AI-resistant CAPTCHA generator195, the AI-resistant verification image142is provided as part of the AI-resistant image-identification CAPTCHA144to malicious entity169of computing device106. Because of the adversarial perturbations included in the image142, AI unit112used for classification by malicious entity169is unable to correctly classify the AI-resistant verification image142that is presented by AI-resistant image-identification CAPTCHA144. That is, malicious entity169encounters the AI-resistant verification image142and attempts to determine the appropriate response to the CAPTCHA query provided the AI-resistant image-identification CAPTCHA144, but the adversarial perturbations result in the AI unit112providing an incorrect classification for the image, and therefore cause the malicious entity169to provide an incorrect CAPTCHA response. As a result, the malicious entity169employing AI unit112is unable to access the services and applications provided by server110.

FIG.2illustrates image database130server110with AI-resistant CAPTCHA generator195ofFIG.1that is used to generate AI-resistant verification images142in accordance with some embodiments. Server110includes a central processing unit (CPU)274, a graphical processing unit (GPU)294, and a memory280. GPU294includes an AI-resistant CAPTCHA generator195that is configured to generate the AI-resistant verification image142using a pixel-by-pixel bounds. The pixel-by-pixel bounds is a bounds (i.e., maximum and minimum limitation) on the adversarial perturbation that is added to each pixel of the source image196. In some embodiments, AI-resistant CAPTCHA generator195is implemented in hardware or software. In some embodiments, AI-resistant CAPTCHA generator195is implemented in CPU274. Alternatively, in some embodiments AI-resistant CAPTCHA generator195is implemented as a separate block (such as an FPGA). Image database130is configured to include source images196that are utilized to generate AI-resistant verification image142of AI-resistant image-identification CAPTCHA144are developed using a database of known images. For example, AI-resistant verification image142is constructed using source images accessed from, for example, ImageNet or CityScapes.

In various embodiments, CPU274or GPU294is configured to execute operations for the server110, including generation of the AI-resistant verifications images142, as described herein. It will be appreciated that, although the CPU274and GPU294are illustrated as single processors, in some embodiments the CPU274and GPU294each represent multiple processing units, with each processing unit including one or more processor cores or other compute units. For example, in at least one embodiment the CPU274represents at least one CPU having multiple processor cores and the GPU294represents at least one GPU having multiple single-instruction multiple data (STMD) compute units configured to execute graphical operations.

Some embodiments of the GPU294are used for general purpose computing. In the illustrated embodiment, the GPU294implements multiple processing elements (not shown inFIG.2in the interest of clarity) that are configured to execute instructions concurrently or in parallel. In the illustrated embodiment, the CPU274and the GPU294communicate with the memory280over a bus (not shown). However, some embodiments of the GPU294communicate with the memory280over a direct connection or via other bridges, switches, routers, and the like. The GPU294executes instructions stored in the memory280and stores information in the memory280such as the results of the executed instructions. For example, the memory280stores a copy of instructions from a program code that is to be executed by the GPU294. As described further herein, both CPU274and GPU294are configured to execute instructions, which include the instructions used to generate the AI-resistant verification image142described below.

In order to generate the AI-resistant verification image142of AI-resistant image-identification CAPTCHA144(depicted inFIG.1), AI-resistant CAPTCHA generator195is configured to use a pixel-by-pixel bounds to generate the AI-resistant verification image142. As stated previously, the pixel-by-pixel bounds is a bounds (i.e., maximum and minimum limitation) on the adversarial perturbation that is added to each pixel of the source image196. The pixel-by-pixel bounds is based on the principles of human vision and the limitations of AI image detection capabilities with regards to adversarial attacks. Adversarial attacks are a process or processes that change the solution or classifier output of a learning system. To be generally useful, many adversarial attacks must also be imperceptible to the human eye, or humans might detect and foil the attack. The process of human vision begins with photons from the visual field striking the retina. The retina is composed of photoreceptors that are spectrally sensitive, where the photoreceptors are activated based on the wavelength of light that impinges the photoceptors. Based on the wavelength of light photons incident from different parts of the visual field during a saccade, different numbers of various types of photoreceptors respond, which allows higher regions of the visual pathway to determine the colors being seen, the object being detected, and the like.

The ratio of the largest to smallest amount of light intensity (i.e., dynamic range) that the human visual system can perceive is large (approximately 1011of ambient light). This perception is performed by adjusting the sensitivity of the photoreceptors of the human visual system based on the background light level, which leads to the notion of a “just noticeable difference” (JND) in the change in stimuli that can be perceived. The JND is governed by what is known as Weber's Law or the Weber-Fechner relation. The Weber-Fechner relation states that the perceived intensity is proportional to the logarithm of the actual intensity. Thus, in some embodiments, perception is represented as

p=k⁢⁢ln⁡(ss0),with⁢⁢S<(1+μ)⁢S0,ln⁡(ss0)<ln⁡(1+μ).
If an intensity change threshold, p=ln(1+μ) is defined as a threshold below which humans cannot perceive, then, in some embodiments, “invisible perturbations” are constructed (by the AI-resistant CAPTCHA generator195as shown below) based on the Weber-Fechner relation that would formally be difficult or even impossible for a human eye to detect.

In various embodiments, the process for generating AI-resistant verification images142includes AI-resistant CAPTCHA generator195providing the pixel-by-pixel bounds (i.e., a bounds on each pixel) of the perturbed image S as:

Smin<S<SmaxwithSmin=s01+μandSmax=(1+μ)⁢S0

where μ is a threshold stimuli and the pixel-by-pixel bounds described above is utilized to generate the human-invisible perturbed AI-resistant verification image142.

For example, during operation, in order to facilitate generation of the AI-resistant verification image142, AI-resistant CAPTCHA generator195receives the input image S0, and determines an initial value of μ that is required to solve for Sminand Smax. In various embodiments, the initial value of μ is determined using a table of μ values (not shown) indicative of threshold values below which a human eye cannot perceive changes in the light intensity of the image S0.

Based on the selected value of μ and the input image S0received by AI-resistant CAPTCHA generator195, AI-resistant CAPTCHA generator195determines the values of Sminand Smaxusing

Smin=s01+μ⁢⁢and⁢⁢Smax=(1+μ)⁢S0.
After AI-resistant CAPTCHA generator195determines the pixel-by-pixel bounds, AI-resistant CAPTCHA generator195generates an initial AI-resistant verification image S. That is, after AI-resistant CAPTCHA generator195of GPU294receives the input image S0, AI-resistant CAPTCHA generator195generates S using the equation:
S=S0+η

where S0is the input image from image database130and η is the adversarial perturbation that is added to the input source image S0by AI-resistant CAPTCHA generator195.

In some embodiments, in order to generate an initial adversarial perturbation η of AI-resistant verification image142, the AI-resistant CAPTCHA generator195uses, for example, a standard adversarial perturbation generation method such as, the Fast Gradient Sign Method (FGSM). As is known in the art, given an image classification network with parameters θ and loss J(θ, x, y)), the FGSM is used to generate the adversarial perturbation η by taking the sign of the gradient J(θ, x, y)), where x is the input image S0, y is the label of x, and ϵ represents the size of the perturbation.
η=ϵ sign(Δx/(θ,x,y))

The output of the FGSM method is an adversarial perturbation η that is used to generate the initial image Sinitusing S=S0+i. After solving for the initial AI-resistant verification image Sinit, AI-resistant CAPTCHA generator195determines whether the initial AI-resistant verification image Sinitis within the pixel-by-pixel bounds, i.e., Sminand Smax, by comparing S to the Sminand Smaxvalues. In one embodiment, when AI-resistant CAPTCHA generator195determines that Sinitis within the pixel-by-pixel bounds, Sinitis output by AI-resistant CAPTCHA generator195as the AI-resistant verification image142. When, for example, AI-resistant CAPTCHA generator195determines that Sinitis not within Sminand Smax, AI-resistant CAPTCHA generator195adjusts the pixel values of Sinitto reside within the pixel-by-pixel bounds by determining a new Weber-based adversarial perturbation ηweberand a new Weber-based AI-resistant verification image142SWeberthat is based on μ. In one embodiment, for example, when AI-resistant CAPTCHA generator195determines that Sinitis above Smax, AI-resistant CAPTCHA generator195determines the adversarial perturbation using

ηWeber=SWeber-SinitwhereSWeber=(smax+smin)2

SWeberis the AI-resistant verification image142output by AI-resistant CAPTCHA generator195. After AI-resistant CAPTCHA generator195generates AI-resistant verification image142, server110provides the AI-resistant verification image142over network140to as part of the AI-resistant image-identification CAPTCHA144to computing device106.

As a result of using the pixel-by-pixel bounds to generate the AI-resistant verification image142, the AI unit112of computing device106is unable to correctly classify the AI-resistant verification image142in the AI-resistant image-identification CAPTCHA144. Thus, server110is protected from the unauthorized access by AI unit112using the AI-resistant verification image142generated based on the Weber-Fechner relation. That is, by modifying existing adversarial perturbations based on the Weber-Fechner relation, contrast of the source image196is maintained below the bound known to trigger human detection.

FIG.3illustrates a method300for generating artificial intelligence resistant verification images142in the network environment ofFIG.1andFIG.2in accordance with some embodiments.

Method300commences at block310with reference toFIGS.1-2. At block305, a source image196is retrieved by server110from image database130. At block310, AI-resistant CAPTCHA generator195determines a threshold stimuli μ based on a table of μ values indicative of threshold values below which a human eye cannot perceive changes in the light intensity of the source image196(i.e., the Weber-Fechner relation). At block315, based on the threshold stimuli μ and the source image196, AI-resistant CAPTCHA generator195determines the pixel-by-pixel bounds (Sminand Smax) for each pixel of the source image196. At block320, AI-resistant CAPTCHA generator195generates an initial adversarial perturbation and corresponding verification image using a standard adversarial generation technique, such as, for example, the Fast Gradient Sign Method (FGSM), the Deep Fool Method (DFM), or the Jacobian-based Saliency Map Attack (JSMA). At block325, a determination is made by AI-resistant CAPTCHA generator195as to whether the verification image and the adversarial perturbation reside within the pixel-by-pixel bounds. Based on whether the verification image and the adversarial perturbation reside within the pixel-by-pixel bounds, AI-resistant CAPTCHA generator195adjusts the verification image and corresponding adversarial perturbations to reside within the pixel-by-pixel bounds (Sminand Smax). At block330, the AI-resistant verification image142is generated by AI-resistant CAPTCHA generator195based on the adjustment of the verification image and the adversarial perturbations. At block335, AI-resistant CAPTCHA generator195provides the AI-resistant verification image142to AI-resistant image-identification CAPTCHA144to prevent unauthorized access to the applications provided by server110.

In various embodiments, additionally, it should be recognized that the processes described herein to generate the AI-resistant verification images142are not limited to human vision. The Weber-Fechner relation also relates to animal vision and would have similar impact on a variety of artificial vision systems, such as cameras. In various embodiments, an iterative approach, such as, for example, a quadratic program is used to find the adversarial perturbation as close to possible to an optimal perturbation that maps to an optimal S that lies within the pixel-by-pixel bounds, Sminand Smax.

In some embodiments, the apparatus and techniques described above are implemented in a server including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as in the server described above with reference toFIGS.1-3. Electronic design automation (EDA) and computer aided design (CAD) software tools may be used in the design and fabrication of these IC devices. These design tools typically are represented as one or more software programs. The one or more software programs include code executable by a computer system to manipulate the computer system to operate on code representative of circuitry of one or more IC devices so as to perform at least a portion of a process to design or adapt a manufacturing system to fabricate the circuitry. This code can include instructions, data, or a combination of instructions and data. The software instructions representing a design tool or fabrication tool typically are stored in a computer readable storage medium accessible to the computing system. Likewise, the code representative of one or more phases of the design or fabrication of an IC device may be stored in and accessed from the same computer readable storage medium or a different computer readable storage medium.

Embodiments of the present disclosure include methods for generating adversarial perturbations which fool networks (e.g., pre-trained CNNs) while being substantially visually imperceptible to the human eye. Generally, the methods for generating adversarial perturbations include generate misclassifications for image recognition tasks and measuring noticeable difference of the images with and without perturbations by the human eye to generate human-friendly, AI-resistant verification images. As used herein, the term “verification image” refers to any image-based mechanism by which a query is issued to elicit an answer from an end user that enables determinization of whether the end user is a human.

In some embodiments, the verification image refers to a CAPTCHA displaying text as a graphic (which is partially obscured and/or distorted) and requires the end user to respond with numerical digits and/or text characters. In other embodiments, the verification image refers to a reCAPTCHA displaying known and unknown graphic words simultaneously for solving for unknown tests through end user input. If the end user successfully identifies the known word, the end user's answer for the unknown word is used by the reCAPTCHA system to “solve” the unknown word.

CAPTCHAs and reCAPTCHAs have been used to prevent spam and abuse. These image verification systems use distorted images of text that an end user needs to identify or employ image-identification tasks, along with behavioral analysis techniques, to determine whether the interacting entity is a human or a bot. CAPTCHA-based systems have become increasingly difficult for humans to use due to an arms-race against AI techniques that can extract the relevant features from images to evade detection by CAPTCHA and other challenge-response tests.

A computer readable storage medium may include any non-transitory storage medium, or combination of non-transitory storage media, accessible by a computer system during use to provide instructions and/or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

In some embodiments, certain aspects of the techniques described above may implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.