Patent Description:
Various approaches to detecting fake images have been created. For example, an algorithm can be utilized to detect subtle inconsistencies (e.g., lighting patterns) in the image indicative of tampering. While such approaches have been created, the reliable detection of fake images has proved to be a challenging and difficult problem. New frameworks are needed for verifying the authenticity of images to address limitations in the current techniques.

<CIT> describes a method of protecting a digital image. The method comprises extracting feature values from the digital image based on a selected authentication bit-rate; embedding data corresponding to the feature values as a watermark into the digital image; and creating an image signature based on the data corresponding to the feature values.

<NPL>), XP080838639 describes a convolutional neural network based encoder-decoder architecture for embedding of images as payload.

<CIT> describes a system including a feature extraction engine.

Techniques and apparatuses are defined in the appended claims for verifying the authenticity of images. In particular, the techniques and apparatuses include or otherwise leverage one or more machine-learned models to verify the authenticity of received images to detect the manipulation of an image. The verification is based on comparing detected image features to recovered image features to determine if an image is fake or manipulated. The comparison of detected features and recovered features can be used to identify characteristics indicative of an inauthentic image that may not be detectable by a human observer.

In all implementations, a feature extraction process is utilized to extract determined features from an input image and a message embedding process is utilized to embed a signature including the determined features in the input image to generate an output image. A feature-extraction process is utilized to extract an embedded signature, including features, from a received image and recovering features from the signature. An authenticity process is then utilized to verify the authenticity of the image, for example, by comparing determined features and recovered features.

Aspects described below include a method performed by a system that includes an encoder system and a decoder system. In the method, the system (e.g., the decoder system) receives an image to be verified. The system performs feature recognition on the received image to determine a plurality of determined features of the received image. The system generates a first output that defines values that represent the determined features of the received image. The system includes a message decoding neural network that decodes the received image to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The system generates a second output that defines values representing the recovered features. The system provides the first output and the second output to a manipulation detection neural network. The manipulation detection neural network generates an estimation of the authenticity of the received image utilizing at least the first output and the second output.

Aspects described below also include a computing device having a processor and a computer-readable storage medium having stored thereon instructions that responsive to execution by the processor, cause the processor to execute procedures for verifying the authenticity of images. Aspects described below include methods of generating images and methods of verifying the authenticity of images. Optional features of one aspect, such as the method described above, may be combined with other aspects.

Techniques and apparatuses for verifying the authenticity of images utilizing machine-leamed models are described with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:.

As described above, the present invention is directed to techniques and apparatuses for verifying the authenticity of images. In particular, the systems and methods of the present invention include or otherwise leverage one or more machine-leamed models to verify the authenticity of an image.

As used herein, the phrase "image" includes, but is not limited to, still images, moving images (e.g., video), moving images associated with audio, moving images associated with text, and the like. As used herein, the phrase "video" means a series or time sequence of moving images, which may or may not also be associated with other types of information, such as audio or text. For example, a video may include a time sequence of moving images that includes a stream of audio information. As used herein, the phrase "fake image" means an image that is presented as an original image but is a manipulated copy of the original image or is a fake copy of the original image. As used herein, the phrase "image feature," includes, but is not limited to, image-independent matrices of measurements throughout an image, facial expressions, landmarks, face landmarks of a person in the image, landmarks, key points, user-provided features, edges, comers, blogs, ridges, motion, optical flow, raw pixels from the image, and the like.

<FIG> illustrates, an example system <NUM> that uses a feature recognition engine <NUM> (e.g., a feature extraction neural network <NUM>) configured to extract image features from an input image <NUM> and a message encoder <NUM> (e.g., a message encoding neural network <NUM>) configured to generate an output image <NUM> from the input image <NUM>. <FIG> illustrates an example system <NUM> that uses a feature recognition engine <NUM> (e.g., a feature extraction neural network <NUM>) configured to extract features <NUM> from a received image <NUM>, a message decoder <NUM> (e.g., a message decoding neural network <NUM>) configured to extract a signature <NUM> (hidden message) including features <NUM> from the received image <NUM>, and/or an authenticity engine <NUM> (e.g., a manipulation detection neural network <NUM>) configured to verify the authenticity of the received image <NUM> based on the processing of the received image <NUM>. In aspects, the system may include one or both of an encoder system <NUM> of <FIG> configured to generate the output image and a decoder system <NUM> of <FIG> configured to decode a received image and to determine the authenticity of the received image. At least one of system <NUM>, the components of system <NUM>, system <NUM>, or the components of system <NUM> may be implemented on one or more of a user computing device, an embedded computing device, a server computing device, a model processing device, a training device, a virtual computing device, other computing devices or computing infrastructure, or combinations thereof, for example as described below with respect to <FIG>.

As illustrated in <FIG>, in system <NUM>, an input image <NUM> is received by the encoder system <NUM> as an input for processing. For example, a user may submit an image (e.g., a photograph) to the system <NUM> as the input image <NUM>. In another example, a user may capture an input image <NUM> on an imaging device (e.g., a digital camera, a camera of a computing device) and the encoder system <NUM> is implemented in the image pipeline of the imaging device. In another example, the input image <NUM> may be stored in an image repository <NUM> and provided to the encoder system <NUM>. The image repository <NUM> may be included in the system <NUM> (e.g., by one or more local wired or wireless connections) or may be remote to the system <NUM> and in communication with the system <NUM> over one or more wired or wireless connections (e.g., a local area network (LAN) or wide area network (WAN) connection). The image repository <NUM> may be, for example, a database stored locally at the system <NUM>, a database hosted at a server remote to the system <NUM>, or a memory device implemented on a computing device.

The system <NUM> (e.g., the encoder system <NUM>) receives the input image <NUM>, selects one or more features <NUM> of the input image <NUM>, generates one or more signatures <NUM> based on the selected feature(s) <NUM>, and embeds the signature(s) <NUM> into the input image <NUM> to create an output image <NUM>. In aspects, the output image <NUM> may be stored on a memory device.

The system <NUM> (e.g., the encoder system <NUM>) includes a feature recognition engine <NUM>. The feature recognition engine <NUM> is configured to perform a feature extraction process on the input image <NUM> to determine a plurality of determined features <NUM> of the input image <NUM>. For example, the feature recognition engine <NUM> may be configured to select a plurality of features from the input image <NUM> and processes the selected image features <NUM> to generate an output <NUM> (e.g., data) that defines values representing the determined features of the input image. In aspects, the feature recognition engine <NUM> selects a plurality of features from the input image <NUM> and processes the selected image features <NUM> to generate an output <NUM> that each represents a selected feature <NUM> of the input image <NUM>. For example, the features <NUM> of the input image <NUM> could include an image-independent matrix of measurements, facial expressions, face landmarks of a person in the image, landmarks, key points, user-provided features, edges, corners, blogs, ridges, motion, optical flow, raw pixels from the image, and the like.

The feature recognition engine <NUM> can include a feature extraction neural network <NUM>. The feature extraction neural network <NUM> can be configured to receive the input image <NUM> and process the input image <NUM> to generate the output <NUM> that defines values that represent a selected feature <NUM> of the input image <NUM>. In aspects, the values each represent a selected feature <NUM> of the input image <NUM>. For example, the feature extraction neural network <NUM> can receive an input image <NUM> and process the input image <NUM>, or a portion thereof, to generate an output <NUM> that defines values that represent one or more image features that correspond to facial expressions, landmarks, key points, edges, corners, blogs, ridges, or the like. In aspects, output <NUM> defines values that each represent one or more image features that correspond to facial expressions, landmarks, key points, edges, corners, blogs, ridges, or the like. In other implementations, the feature recognition engine <NUM> may generate image features through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

The feature recognition engine <NUM> can receive one or more user-provided features <NUM> provided by one or more user input components configured to receive input (selected image features) from a user. The user-provided features <NUM> may include, facial expressions, face landmarks of the person in the image, raw pixels from the image, and the like. The feature recognition engine <NUM> can include a feature extraction neural network <NUM> and user-provided features <NUM>.

The system <NUM> (e.g., the encoder system <NUM>) also performs a message embedding process on the input image <NUM>. The system <NUM> sends an output <NUM> that specifies the image features <NUM> derived from the input image <NUM> to a message encoder <NUM>. The message encoder <NUM> may include a message encoding neural network <NUM>. The message encoding neural network <NUM> may be configured as a message coding neural network (e.g., a message encoding/decoding neural network) utilized to encode and decode messages. The message encoding neural network <NUM> of <FIG> and the message decoding neural network <NUM> of <FIG> are a message encoding/decoding neural network.

The message encoder <NUM> (e.g., message encoding neural network <NUM>), receives the output <NUM> that specifies the image features <NUM> derived from the input image <NUM> from the system <NUM> and generates a signature <NUM> represents image features <NUM> of the input image <NUM> (e.g., includes the output <NUM>). The message encoder <NUM> (e.g., message encoding neural network <NUM>) processes the input image <NUM> by embedding the signature <NUM> (message) into the input image <NUM> as a digital message (e.g., a Steganographic signal) to generate an output image <NUM>. In aspects, the signature <NUM> is a perceptually invisible watermark.

Weights may be shared between the feature recognition engine <NUM> (e.g., first feature extraction neural network <NUM>, user-provided features <NUM>) illustrated in <FIG> and the second feature recognition engine <NUM> (e.g., second feature extraction neural network <NUM>, user-provided features <NUM>) illustrated in <FIG>.

<FIG> illustrates a system <NUM> that includes a decoder system <NUM> configured to decode a received image <NUM> and to verify the authenticity of the received image <NUM>. The received image <NUM> is an image to be verified. In aspects, the received image <NUM> is a copy of the output image <NUM>. In aspects, the received image <NUM> is a fake copy of input image <NUM> or output image <NUM>. In aspects, the received image <NUM> is a manipulated copy of input image <NUM> or output image <NUM>. The decoder system <NUM> determines the authenticity of the received image <NUM>, for example, whether the received image <NUM> is the output image <NUM>, or whether the received image <NUM> is a fake version of input image <NUM>.

The system <NUM> can send the output image <NUM> generated by the encoder system <NUM> to the decoder system <NUM> as the received image <NUM>. In some implementations, the received image <NUM> is provided to the decoder system <NUM> of the system <NUM>; for example, a user may submit a photograph to the system <NUM> as received image <NUM>. In another example, the received image <NUM> may be provided to the decoder system <NUM> by an image repository, such as image repository <NUM> of <FIG>.

In one example, a user utilizes system <NUM> and/or system <NUM> to verify the authenticity of an image. In another example, an online service provider utilizes system <NUM> and/or system <NUM> to verify the authenticity of an image. In another example, a third party utilizes system <NUM> and/or system <NUM> to verify the authenticity of an image.

The system <NUM> provides the received image <NUM> to the decoder system <NUM> as an input, and the decoder system <NUM> determines if the received image <NUM> is a reliable copy (e.g., authentic copy, unaltered copy) or if the received image <NUM> is an unreliable copy, for example, a fake image or an altered copy of the output image <NUM> or input image <NUM>. As illustrated in <FIG>, the system <NUM> (e.g., decoder system <NUM>) includes a feature recognition engine <NUM>, a message decoder <NUM>, and an authenticity engine <NUM>.

The system <NUM> can provide the received image <NUM> as an input to the feature recognition engine <NUM>. The feature recognition engine <NUM> is configured to perform a feature extraction process (feature recognition) on the received image <NUM> to determine a plurality of determined features <NUM> of the received image <NUM>. For example, the determined features <NUM> of the received image <NUM> could include an image-independent matrix of measurements, facial expressions, landmarks, key points, user-provided features, edges, corners, blogs, ridges, motion, optical flow, pixels, and the like.

The feature recognition engine <NUM> selects a plurality of determined features <NUM> from the received image <NUM> and processes the determined features <NUM> to generate an output <NUM> (data) that defines values representing the determined features of the received image <NUM>. In aspects, the feature recognition engine <NUM> selects a plurality of determined features <NUM> from the received image <NUM> and processes the determined features <NUM> to generate an output <NUM> (data) that defines values representing the determined features of the received image that each represents a selected feature of the received image <NUM>. The output <NUM> of determined features <NUM> is sent by the system <NUM> to the authenticity engine <NUM>.

The feature recognition engine <NUM> can include a second feature extraction neural network <NUM>. The feature extraction neural network <NUM> performs the feature extraction process on the received image <NUM>. The feature extraction neural network <NUM> can be configured to receive the received image <NUM> and process the received image <NUM> to generate the output <NUM> (data) that defines values representing the determined features <NUM> of the received image <NUM>. For example, the feature extraction neural network <NUM> can be configured to receive the received image <NUM> and process the received image <NUM> to generate the output <NUM> that defines values that each represent a determined feature <NUM> of the received image <NUM>. In some implementations, the feature extraction neural network <NUM> is the same neural network as the feature extraction neural network <NUM> of <FIG>. In some implementations, the feature extraction neural network <NUM> is a different feature extraction neural network than the feature extraction neural network <NUM> of <FIG>.

The feature recognition engine <NUM> can receive input including one or more user-provided features <NUM> provided by one or more user input components (e.g., user input component <NUM> of <FIG>) configured to receive input (selected image features) from a user. The user-provided features <NUM> may include, for example, facial expressions, face landmarks of the person in the image, raw pixels from the image, and the like. In some implementations, the feature recognition engine <NUM> can include the second feature extraction neural network <NUM> and user-provided features <NUM>.

The received image <NUM> can be provided as an input to the message decoder <NUM> of the decoder system <NUM>. The message decoder <NUM> (e.g., a message decoding neural network <NUM>) decodes the received image <NUM>. For example, the message decoder <NUM> decodes the received image <NUM> by performing a message extraction process on the received image <NUM>. In the message extraction process, the message decoder <NUM> processes the received image <NUM> to extract a signature <NUM> embedded in the received image <NUM>. The signature <NUM> represents recovered features <NUM> that were embedded into the received image <NUM> by a message encoder (e.g., the message encoder <NUM> of <FIG>). In aspects, the system <NUM> detects whether a signature is present in the received image <NUM>.

The message extraction process includes the generation of a second output <NUM> (data) that defines values that represent recovered features <NUM> of the received image <NUM>. In aspects, the message extraction process includes the generation of a second output <NUM> that defines values that each represent a recovered feature <NUM> of the received image <NUM> that were recovered from the signature <NUM>. The output <NUM> of recovered features <NUM> is sent by the system <NUM> to the authenticity engine <NUM>.

The output <NUM> (first output) from the feature recognition engine <NUM> and the output <NUM> (second output) of the message decoder <NUM> are provided to and are utilized by the authenticity engine <NUM> to generate a prediction (e.g., an estimate) of the authenticity of the received image <NUM>. For example, the authenticity engine <NUM> of the decoder system <NUM> determines whether the received image <NUM> is a reliable copy (e.g., authentic copy, unaltered copy) or if the received image <NUM> is an unreliable copy, for example, a fake image or an altered copy.

The authenticity engine <NUM> can include a manipulation detection neural network <NUM> that receives the outputs (e.g., output <NUM>, output <NUM>). The manipulation detection neural network <NUM> uses at least output <NUM> and output <NUM> to generate an estimate (prediction) of the authenticity of the received image <NUM>, for example, by comparing the output <NUM> to the output <NUM>. Based on the estimate generated by the authenticity engine <NUM>, the system <NUM> determines whether the received image <NUM> is a reliable copy (e.g., authentic copy, unaltered copy) or if the received image <NUM> is an unreliable copy, for example, a fake image or an altered copy.

The systems and methods of the present disclosure can be implemented by or otherwise executed on one or more computing systems. Example computing systems in the computing system <NUM> include one or more of a user computing devices (e.g., laptops, desktops, mobile computing devices such as tablets, smartphones, wearable computing devices, cameras, etc.); embedded computing devices (e.g., devices embedded within a vehicle, camera, image sensor, industrial machine, satellite, gaming console or controller, or home appliance such as a refrigerator, thermostat, energy meter, home energy manager, smart home assistant, etc.); server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, web servers, game servers, application servers, etc.); dedicated, specialized model processing or training devices; virtual computing devices; other computing devices or computing infrastructure; or combinations thereof.

For example, <FIG> depicts a block diagram of an example computing system <NUM> that can verify the authenticity of images according to example embodiments of the present disclosure. The computing system <NUM> includes one or more of a user computing system <NUM>, a server computing system <NUM>, or a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The server computing system <NUM> can include or can otherwise be implemented by one or more server computing devices. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The user computing system <NUM> includes one or more processors <NUM> and one or more memory devices <NUM>. The server computing system <NUM> includes one or more processor(s) <NUM> and a memory device(s) <NUM>. The training computing system <NUM> includes one or more processor(s) <NUM> and a memory device(s) <NUM>. The processor(s) <NUM>, <NUM>, <NUM> can be any suitable processing device (e.g., a central processing unit (CPU); a visual processing unit (VPU); a graphics processing unit (GPU); a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU or other processing device; an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); a co-processor; a controller; or combinations of the processing devices described above) and can be one processor or a plurality of processors that are operatively connected. Processor(s) <NUM>, <NUM>, <NUM> can be embedded within other hardware components such as, for example, an image sensor, accelerometer, and the like.

The memory device(s) <NUM>, <NUM>, <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. The memory device(s) <NUM> can store data <NUM> and instructions <NUM>, which are executed by the processor(s) <NUM> to cause the user computing system <NUM> to perform operations. The memory device(s) <NUM> can store data <NUM> and instructions <NUM>, which are executed by the processor(s) <NUM> to cause the server computing system <NUM> to perform operations. The memory device(s) <NUM> can store data <NUM> and instructions <NUM>, which are executed by the processor(s) <NUM> to cause the training computing system <NUM> to perform operations.

One or more of the user computing system <NUM>, the server computing system <NUM>, or the training computing system <NUM> can include an encoder system (e.g., encoder system <NUM> of <FIG>). An example encoder system may include a feature recognition engine (e.g., feature recognition engine <NUM> of <FIG>) and a message encoder (e.g., message encoder <NUM> of <FIG>). In some implementations, one or more of the user computing system <NUM>, the server computing system <NUM>, or the training computing system <NUM> can include a decoder system (e.g., decoder system <NUM> of <FIG>). An example decoder system may include at least one of a feature recognition engine (e.g., feature recognition engine <NUM> of <FIG>), a message decoder (e.g., message decoder <NUM> of <FIG>), or an authenticity engine (e.g., authenticity engine <NUM> of <FIG>).

The user computing system <NUM> can store or include one or more machine-learned models <NUM>. For example, the machine-leamed models <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-leamed models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. More particularly, the machine-leamed models <NUM> can include one or more of a feature extraction network (such as the feature extraction neural network <NUM> of <FIG> and/or the second feature extraction neural network <NUM> of <FIG>), a message encoding neural network (such as the message encoding neural network <NUM> of <FIG>), a message decoding neural network (such as the message decoding neural network <NUM> of <FIG>), or a manipulation detection neural network (such as the manipulation detection neural network <NUM> of <FIG>).

Additionally or alternatively, one or more machine-learned models <NUM> can be included in or otherwise stored and implemented by the server computing system <NUM> that communicates with the user computing system <NUM> according to a client-server relationship. For example, the machine-leamed models <NUM> can be implemented by the server computing system <NUM> as a portion of a web service (e.g., an extreme multiclass or multilabel classification service, a language modeling service, a metric learning service). Thus, one or more models <NUM> can be stored and implemented at the user computing system <NUM> and/or one or more models <NUM> can be stored and implemented at the server computing system <NUM>. In implementations, the machine-learned models <NUM> can include one or more of a feature extraction network (such as the feature extraction neural network <NUM> of <FIG> and/or the feature extraction neural network <NUM> of <FIG>), a message encoding neural network (such as the message encoding neural network <NUM> of <FIG>), a message decoding neural network (such as the message decoding neural network <NUM> of <FIG>), or a manipulation detection neural network (such as the manipulation detection neural network <NUM> of <FIG>).

The user computing system <NUM> can also include one or more user input component <NUM> that receives user input. For example, the user input component <NUM> can be a touch-sensitive component (e.g., a touch-sensitive display screen, a touchpad) that is sensitive to the touch of a user input object (e.g., a finger, a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

As described above, the server computing system <NUM> can store or otherwise include one or more machine-leamed models <NUM>. For example, the models <NUM> can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Examples of neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

The computing system <NUM> may include an image repository <NUM>. For example, the image repository <NUM> may be included in the system <NUM> by one or more local wired or wireless connections or may be remote to the system <NUM> and in communication with the system <NUM> over one or more wired or wireless connections, e.g., a local area network (LAN) or wide area network (WAN) connection. The image repository <NUM> may be, for example, a database stored locally at the user computing system <NUM> or a database hosted at a server remote to the system <NUM> (e.g., server computing system <NUM>, training computing system <NUM>).

The user computing system <NUM> and/or the server computing system <NUM> can train the models <NUM> and/or <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing system <NUM> or can be a portion of the server computing system <NUM>.

The training computing system <NUM> can include a model trainer <NUM> that trains the machine-leamed models <NUM> and/or <NUM> stored at the user computing system <NUM> and/or the server computing system <NUM> using various training or learning techniques, such as, for example, verifying the authenticity of images based on comparing determined features and recovered features with the machine-leamed models <NUM> and/or <NUM> and backward propagation of errors. In some implementations, performing backward propagation of errors can include performing truncated backpropagation through time. The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts) to improve the generalization capability of the models being trained.

The model trainer <NUM> can train the machine-learned models <NUM> and/or <NUM> based on a set of training data <NUM>. The training data <NUM> can include, for example, examples of the input data that have been assigned labels that correspond to the output data.

If the user has provided consent, the training examples can be provided by the user computing system <NUM>. Thus, in such implementations, the model <NUM> provided to the user computing system <NUM> can be trained by the training computing system <NUM> on user-specific data received from the user computing system <NUM>. In some instances, this process can be referred to as personalizing the model.

The model trainer <NUM> includes computer logic utilized to provide the desired functionality. The model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general-purpose processor. For example, in some implementations, the model trainer <NUM> includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

The network <NUM> can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Intemet), or some combination thereof and can include any number of wired or wireless links.

<FIG> illustrates one example of a computing system <NUM> that can be used to implement the present disclosure. Other computing systems can be used, as well. For example, in some implementations, the user computing system <NUM> can include the model trainer <NUM> and the training data <NUM>. In such implementations, the models <NUM> can be both trained and used locally at the user computing system <NUM>. In some such implementations, the user computing system <NUM> can implement the model trainer <NUM> to personalize the models <NUM> based on user-specific data.

The frameworks described herein can be trained in many ways, for example, depending on how the feature extraction neural network, the message coding neural network (e.g., message encoding neural network, message decoding neural network, message encoding/decoding neural network), and/or the manipulation detection neural network are trained. In some implementations, one or more of the neural networks can be trained separately. In some implementations, one or more of the neural networks are co-trained. In other implementations, all of the neural networks are trained together.

<FIG> depicts an example system <NUM> configured to train a first feature extraction neural network <NUM> (such as the first feature extraction neural network <NUM> of <FIG> and/or the second feature extraction neural network <NUM> of <FIG>) to extract features from an image (e.g., input image <NUM>) with improved accuracy. In aspects, the feature extraction neural network <NUM> is utilized in a system configured to verify the authenticity of an image (e.g., system <NUM> of <FIG>, system <NUM> of <FIG>).

The feature extraction neural network <NUM> may be trained separately from one or more of a message encoding neural network, a message decoding neural network, or a manipulation detection neural network. For example, at operation <NUM>, an input image <NUM> is provided to an encoder system <NUM>, including a feature recognition engine <NUM>, for processing. For example, a user may submit an image (e.g., a photograph) to the system <NUM> as the input image <NUM>. In another example, the feature recognition engine <NUM> is a feature extraction neural network <NUM> and an image repository <NUM> (as described in regards to <FIG>) provides the input image <NUM> to the feature recognition engine <NUM>. In implementations, the feature recognition engine <NUM> may generate determined features <NUM> through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

The feature recognition engine <NUM> (e.g., feature extraction neural network <NUM>) is configured to estimate (determine) features of the input image <NUM> and to generate an output of determined features <NUM> (such as features <NUM> of <FIG>). In aspects, the feature extraction neural network <NUM> estimates determined features <NUM> that have characteristics that can be recognized as learned features by image recognition. The learned features should be sensitive to the detection of fake images; for example, if the image was manipulated or is fake, the learned features will be quite different.

Optionally, or additionally, the determined features <NUM> can be extracted from the input image <NUM> through the use of user input, such as user-provided features <NUM> (e.g., user-provided features <NUM> described herein). In implementations, a second feature recognition engine <NUM> may generate recovered features <NUM> through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

After determined features <NUM> are estimated, the feature recognition engine <NUM> sends an output <NUM> of the determined features <NUM> to a discrimination feature loss computation engine <NUM>. The determined features <NUM> are provided to the discrimination feature loss computation engine <NUM> to enable the discrimination feature loss computation engine <NUM> to compute a feature loss <NUM> that is used to train the feature extraction neural network <NUM>. This process may be repeated until the computed loss is equal to or below a predetermined threshold, or until other convergence criteria are met.

The feature recognition engine <NUM> can also send an output <NUM> with the determined features <NUM> to a message encoder <NUM>, including a message encoding neural network <NUM> (e.g., message encoding neural network <NUM> of <FIG>), to generate an output image <NUM>. In aspects, one or more signature(s) are generated based on the determined feature(s) <NUM>, and the signature(s) are embedded into the input image <NUM> as a digital message (e.g., a Steganographic signal) to generate the output image <NUM>. In implementations, the feature recognition engine <NUM> may generate determined features <NUM> through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

A feature extraction process is performed on an image (e.g., the input image <NUM>, a received image <NUM>) by a feature recognition engine, for example by a first feature recognition engine <NUM> (e.g., first feature extraction neural network <NUM> (such as the first feature extraction neural network <NUM> of <FIG>)) and/or by a second feature recognition engine <NUM> (e.g., second feature extraction neural network <NUM> (such as the feature extraction neural network <NUM> of <FIG>)). For example, as illustrated in <FIG>, a decoder system <NUM> includes the second feature recognition engine <NUM> (e.g., feature extraction neural network <NUM>). The second feature recognition engine <NUM> decodes the received image <NUM> to extract a signature from the received image <NUM> (for example, as described above with respect to <FIG>). The signature represents recovered features <NUM> of the received image <NUM>.

The feature recognition engine (e.g., second feature recognition engine <NUM>) sends an output <NUM> with the recovered features <NUM> to the discrimination feature loss computation engine <NUM>. The recovered features <NUM> are provided to enable the discrimination feature loss computation engine <NUM> to compute a feature loss <NUM> that can be used to train the feature extraction neural network <NUM>.

The system may perform a message coding neural network training procedure in which the discrimination feature loss computation engine <NUM> applies a discrimination feature loss function to the recovered features <NUM> and the determined features <NUM> to determine an image feature loss <NUM> caused by the estimation performed by the feature extraction neural network <NUM>. The discrimination feature loss functions compare the determined features <NUM> of the input image <NUM> (such as the selected features <NUM> of the input image <NUM> of <FIG>) to the recovered features <NUM> of the received image <NUM> (such as the recovered features <NUM> of the received image <NUM> of <FIG>). If the tested image (e.g., received image <NUM> of <FIG>) is authentic (e.g., is not a fake copy of the output image <NUM>, is not a manipulated copy of the output image <NUM>), then the discrimination feature loss functions should be small. However, if the tested image (e.g., received image <NUM>) is not authentic (e.g., is a fake copy of the output image <NUM>, is a manipulated copy of the output image <NUM>), then the discrimination feature loss functions should be large.

The discrimination feature loss computation engine <NUM> sends the feature loss <NUM> as an output to the feature recognition engine <NUM> for use in training the feature extraction neural network <NUM>. The loss function determines deviations between the determined features <NUM> estimated by the feature recognition engine <NUM> and the recovered features <NUM> generated by the second feature recognition engine <NUM>. In some implementations, the loss function may represent the deviations as a loss in the accuracy of recovered features <NUM> in the received image <NUM> as a result of the received image <NUM> is a fake image.

After the loss is calculated, the feature loss <NUM> is used to further optimize the feature extraction neural network <NUM>. For example, the feature loss <NUM> may be used to perform parameter optimization for layers of the feature extraction neural network <NUM>.

Such a training loop may be repeated for multiple images from the image repository <NUM> to iteratively optimize the feature extraction neural network <NUM>. Parameter optimization enables the feature extraction neural network <NUM> to more accurately estimate the features depicted in images. Over time and with sufficient training through iterations of the training loop, the feature extraction neural network <NUM> may improve to minimize the loss function such that the recovered features <NUM> provided to the discrimination feature loss computation engine <NUM> by the second feature recognition engine <NUM> converge to the determined features <NUM> provided to the discrimination feature loss computation engine <NUM> by the first feature recognition engine <NUM>. The process illustrated in <FIG> may be repeated until the feature loss is equal to or below a predetermined threshold, or until other convergence criteria are met.

<FIG> depicts an example system <NUM> configured to train a message coding neural network (e.g., a message encoding/decoding neural network). For example, a message encoding neural network (e.g., message encoding neural network <NUM> of <FIG>, message encoding neural network <NUM> of <FIG>) of a message encoder <NUM> and/or a message decoding neural network (e.g., message decoding neural network <NUM> of <FIG>, message decoding neural network <NUM> of <FIG>) of a message decoder <NUM> of a decoder system <NUM> to encode output images and/or decode received images with improved accuracy. In aspects, at least one of the message coding neural network or the message decoding neural network is trained separately from a feature extraction neural network and/or a manipulation detection neural network. In aspects, the message coding neural extraction neural network is co-trained with one or more of a feature extraction neural network or a manipulation detection neural network.

At operation <NUM>, an input image <NUM> is provided to a feature recognition engine <NUM> for processing. For example, as described above with respect to <FIG>, a user may submit an image (e.g., a photograph) to the system <NUM> as the input image <NUM>. In implementations, the input image <NUM> is modified in order to train at least one of the message encoding neural network <NUM> to be robust to such kinds of distortions. For example, the input image <NUM> can be modified (e.g., edited, resized, cropped) or can be a fake image.

In another example, the feature recognition engine <NUM> includes a first feature extraction neural network <NUM> of an encoder system <NUM>. An image repository <NUM> (e.g., as described with respect to the image repository <NUM> of <FIG>) provides the input image <NUM> to the feature recognition engine <NUM>, as described with respect to <FIG>. At operation <NUM>, the input image <NUM> is also provided to an image loss computation engine <NUM> for processing.

The feature recognition engine <NUM> (e.g., feature extraction neural network <NUM>) estimates (determines) features of the input image <NUM> and generates an output of determined features <NUM>. In aspects, the feature extraction neural network <NUM> estimates determined features <NUM> that have characteristics that can be recognized as learned features by image recognition. The learned features should be sensitive to the detection of fake images; for example, if the image was manipulated or is fake, the learned features will be quite different. In aspects, user-provided features (such as user-provided features <NUM> of <FIG> and/or user-provided features <NUM> of <FIG>) may be provided by a user for inclusion with the determined features <NUM>.

Optionally, or additionally, the determined features <NUM> can be extracted from the input image <NUM> through the use of user-provided features, as is described above with respect to <FIG>. In implementations, the feature recognition engine <NUM> may generate determined features <NUM> through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

After estimated determined features <NUM>, the feature recognition engine <NUM> sends an output <NUM> (third output) of the determined features <NUM> to a feature loss computation engine <NUM>. The determined features <NUM> are provided to the feature loss computation engine <NUM> to enable the feature loss computation engine <NUM> to compute a feature loss <NUM> that is provided as an input to a total loss computation engine <NUM>.

The feature recognition engine <NUM> also sends an output <NUM> with the determined features <NUM> to a message encoding neural network <NUM> (e.g., message encoding neural network <NUM> of <FIG>) of a message encoder <NUM> to generate an output image <NUM>. In aspects, one or more signature(s) are generated based on the determined feature(s) <NUM>, and the signature(s) are embedded into the output image <NUM> as a digital message (e.g., a Steganographic signal) to generate the output image <NUM>.

At operation <NUM>, a received image <NUM> is sent to the image loss computation engine <NUM> for processing. At operation <NUM>, a received image <NUM> is sent to the decoder system <NUM>. In aspects, the decoder system <NUM> includes a message decoder <NUM>. In aspects, the message decoder <NUM> includes a message decoding neural network <NUM>. The message decoding neural network <NUM>, such as the message decoding neural network <NUM> of <FIG>, decodes the received image <NUM> to extract a signature from the received image <NUM>. The signature represents recovered features <NUM> of the received image <NUM>. In implementations, the message decoding neural network <NUM> is a message encoding/decoding neural network configured to both encode and decode images.

The message decoding neural network <NUM> is configured to send an output <NUM> (second output) that includes the recovered features <NUM> to the feature loss computation engine <NUM>. The recovered features <NUM> are provided to the feature loss computation engine <NUM> to enable the feature loss computation engine <NUM> to compute a feature loss <NUM> that is provided as an input to a total loss computation engine <NUM>.

The system <NUM> determines a feature loss <NUM>. In implementations, the system <NUM> includes a feature loss computation engine <NUM> that applies a loss function to determine the feature loss <NUM>. The loss function determines deviations between the determined features <NUM>, which were estimated by the feature recognition engine <NUM> in relation to features in the input image <NUM>, and the recovered features <NUM> extracted by the message decoding neural network <NUM> in relation to features in the received image <NUM>. In some implementations, the loss function may represent the deviations as a loss in the accuracy of the recovered features <NUM> in the received image <NUM> as a result of the received image <NUM> is a fake image. After the loss is calculated, the feature loss <NUM> is sent as an output (first loss) to the total loss computation engine <NUM>. The feature loss <NUM> is provided to the total loss computation engine <NUM> to further optimize one or more of the feature extraction neural network <NUM>, the message encoding neural network <NUM>, the message decoding neural network <NUM>, or a message coding neural network (e.g., a message encoding/decoding neural network).

The system <NUM> determines an image loss <NUM>. In implementations, the system <NUM> includes an image loss computation engine <NUM> that receives the input image <NUM> and the received image <NUM> and applies a loss function to determine an image loss <NUM>. The loss function determines deviations between the input image <NUM> and the received image <NUM>. In some implementations, the loss function may represent deviations in the received image <NUM> as a result of a fake image or a manipulated image. After the image loss <NUM> is calculated, the image loss <NUM> is sent to the total loss computation engine <NUM> and can be used to further optimize one or more of the feature extraction neural network <NUM>, the message encoding neural network <NUM>, or the message decoding neural network <NUM>.

The system <NUM> determines a total loss <NUM> based on the image loss <NUM> and the feature loss <NUM>. In aspects, the system <NUM> includes a total loss computation engine <NUM> that applies a loss function to the feature loss <NUM> and the image loss <NUM> to determine the total loss <NUM>. The total loss <NUM> may be caused by the estimation performed by at least one of the first feature extraction neural network <NUM> or the message decoding neural network <NUM>. The loss function determines deviations between the feature loss <NUM> computed by the feature loss computation engine <NUM> and the image loss <NUM> computed by the image loss computation engine <NUM>.

The total loss function to be optimized can be represented by the following equation: <MAT> Where the total loss equals the image loss function (input image, output image) plus the feature loss function (features, recovered features from input image). In the equation, total loss is L, image loss function is Lil, feature loss function is Lfl, input image is Ii, output image is Io, features are f, and recovered features from input image are fii.

After the total loss is calculated, the computed total loss <NUM> is used to further optimize at least one of the feature extraction neural network <NUM>, the message decoding neural network <NUM>, or a message coding neural network (e.g., a message encoding/decoding neural network). For example, the computed total loss <NUM> may be used to perform parameter optimization for layers of one or more of the feature extraction neural network <NUM> or the message decoding neural network <NUM>. The process illustrated in <FIG> may be repeated until the computed total loss is equal to or below a predetermined threshold, or until other convergence criteria are met. Such a training loop may be repeated for multiple images from the image repository <NUM> to iteratively optimize one or more of the feature extraction neural network <NUM>, the message encoding neural network <NUM>, or the message decoding neural network <NUM>. In implementations, parameter optimization enables the feature extraction neural network <NUM> to more accurately estimate the features depicted in images. Over time and with sufficient training through iterations of the training loop, the feature extraction neural network <NUM> may improve such that the recovered features <NUM> provided to the feature loss computation engine <NUM> by the message decoding neural network <NUM> converge to the determined features <NUM> provided to the feature loss computation engine <NUM> by the feature extraction neural network <NUM>.

In implementations, a manipulation detection neural network (e.g., the manipulation detection neural network <NUM> illustrated in <FIG>) is trained separately from one or more of a message encoding neural network (e.g., the message encoding neural network <NUM> of <FIG>), a message decoding neural network (e.g., the message decoding neural network <NUM> of <FIG>, the message decoding neural network <NUM> of <FIG>), or a feature extraction neural network (e.g., the feature extraction neural network <NUM> of <FIG>, the feature extraction neural network <NUM> of <FIG>, the feature extraction neural network <NUM> of <FIG>, the second feature extraction neural network <NUM> of <FIG>, the first feature extraction neural network <NUM> of <FIG>).

To train the manipulation detection neural network, the values of at least one of the feature extraction neural network, message encoding neural network, feature extraction neural network, or message decoding neural network is fixed, and a training image is generated. The training image may include at least one image manipulation. In aspects, the input image may be a training image. In aspects, the received image may be a training image.

The system sends the training image to the manipulation detection neural network. The manipulation detection neural network determines the image manipulation applied to the training image. Utilizing the image manipulation applied to the training image, the manipulation detection neural network computes a normal loss based on a normal loss function. In implementations, the normal loss is a cross-entropy loss. The manipulation detection neural network is trained based at least on the computed normal loss. In aspects, the loss function applied is a normal loss function for classification.

The feature extraction neural network can be co-trained with a message coding neural network (e.g., a message encoding/decoding neural network, one or more of a message encoding neural network or a message encoding/decoding neural network) to extract features from an image and/or to encode/decode messages in images with improved accuracy. For example, through the combination of the separate feature extraction neural network training, described above, with the separate message encoding/decoding neural network training, described above. The total loss function to be optimized can be represented by the following equation: <MAT> Where the total loss equals the image loss function (input image, output image) plus the feature loss function (features, recovered features from input image) plus the discrimination feature loss function (features, features from received image). In the equation, total loss is L, image loss function is Lil, feature loss function is Lfl, input image is Ii, output image is Io, features are f, recovered features from the input image are fii, discrimination feature loss function is Ldfl, and features from the received image arefri.

The input image may be perceptually the same as output image, and the recovered features from the input image are as close as possible to the features extracted (recovered) from the input image, even if some image manipulation (or faking) is present. In this case, the feature learned might be features that are easier for the message encoding neural network to hide and extract. Further, the learned features may be sensitive to image manipulation (or faking), and as a result, if the image is changed (e.g., manipulated, faked), the features extracted will be quite different from that extracted from the input image. In such an aspect, the manipulation detection neural network may be trained separately, as described above.

The feature extraction neural network can be co-trained with the manipulation detection neural network to extract features from an image and/or to determine whether the received image is a fake or manipulated version of the input image with improved accuracy. In such an aspect, the message coding neural network (e.g., the message encoding neural network <NUM> of <FIG> and the message decoding neural network <NUM> of <FIG>) may be trained first. To train the message coding neural network, image manipulation (or faking) that is desirable to detect could be applied to a random input image with random features.

The total loss function to be optimized could be represented by the following equation: <MAT> Where the total loss equals the image loss function (input image, output image) plus the feature loss function (random features, recovered random features). In the equation, total loss is L, image loss function is Lil, feature loss function is Lfl, random features are Fra, and recovered random features are fr.

The input image can be perceptually the same as output image, and random features are recovered from the input image. The weights in the message coding neural network are then fixed, and the feature extraction network and manipulation detection neural network are co-trained. An image manipulation or fake desired to be detected can then be randomly applied or not applied.

The total loss function to be optimized can be represented by the following equation: <MAT> Where the total loss equals the discrimination feature loss function (features, features from received image) plus cross-entropy loss (fake or real). In the equation, total loss is L, discrimination feature loss function is Ldfl, features from the received image are fri, and cross-entropy loss (fake or real) is Lce. Through such a training process, the feature extraction neural network will learn the best features for detecting fake or real images.

A message coding neural network (e.g., the message encoding neural network <NUM> of <FIG> and the message decoding neural network <NUM> of <FIG>) can be co-trained with a manipulation detection neural network to encode/decode messages in images and/or to determine whether a received image is a fake or manipulated version of an input image with improved accuracy. In such an aspect, the feature extraction neural network is separately trained, as described above. After the feature extraction neural network is trained, then the weights are fixed. To co-train the message coding neural network and manipulation detection neural network, an image manipulation or fake desired to be detected is randomly applied or not applied. The total loss function to be optimized can be represented by the following equation: <MAT>.

Where the total loss equals the image loss function (input image, output image) plus the feature loss function (features, recovered features from input image) plus cross-entropy loss (fake or real). In the equation, total loss is L, image loss function is Lil, feature loss function is Lfl, input image is Ii, output image is Io, features are f, recovered features from the input image are fii, and cross-entropy loss (fake or real) is Lce. In aspects, the input image can be perceptually the same as output image, and the recovered features from the input image are as close as possible to the features extracted (recovered) from the input image, even if some image manipulation (or faking) is present.

A feature extraction neural network can be co-trained with a message coding neural network (e.g., a message encoding/decoding neural network, a message encoding neural network and a message decoding neural network), and a manipulation detection neural network to extract features from an image with improved accuracy, to encode/decode messages in images, and/or to determine whether the received image is a fake or manipulated version of the input image with improved accuracy.

A manipulation or fake that is desired to be detected can be randomly applied or not applied. The total loss function to be optimized can be represented by the following equation: <MAT> Where the total loss equals the image loss function(input image, output image) plus the feature loss function(random features, recovered random features) plus discrimination feature loss function(features, features from received image) plus cross-entropy loss (fake or real). In the equation, total loss is L, image loss function is Lii, feature loss function is Lfl, random features are Fra, recovered random features are frr, discrimination feature loss function is Ldfl, features from the received image are fri, and cross-entropy loss (fake or real) is Lce.

In a case where the received image is a fake image, the discrimination feature loss functions will penalize the case features, and features from the received image are close. In a case where the received image is real, the discrimination feature loss functions will penalize the case features, and features from the received image are far. It is desirable for the input image to be perceptually the same as an output image. It is desirable for the message coding neural network to be robust to image manipulation or fake. It is desirable for the feature exacted to be sensitive for image manipulation or fake.

While features and concepts of the described techniques and apparatuses for verifying the authenticity of images can be implemented in any number of different environments, systems, devices, and/or various configurations, aspects of verifying the authenticity of images are described in the context of the following example devices, systems, and configurations.

For instances in which the techniques and/or apparatuses discussed herein may collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect personal information, e.g., information about a user's social network, social actions or activities, profession, preferences, or current location, or to control whether and/or how the system and/or methods can perform operations more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained, such as to a city, ZIP code, or state level, so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about them and used.

<FIG> illustrates an example method <NUM> of generating an image to be verified. The method <NUM> may be performed by a system <NUM> as described above with respect to <FIG>, utilizing one or more of the components described with respect to <FIG>. At <NUM>, an encoder system receives an input image. The encoder system, at <NUM>, performs a feature recognition on the input image to determine a plurality of determined features of the input image. At <NUM>, the encoder system generates a third output defining values representing the determined features of the input image. At <NUM>, the encoder system provides the third output to a message encoding neural network. At <NUM>, the message encoding neural network generates a signature from the third output defining values representing the determined features of the input image. The encoder system, at <NUM>, embeds the second signature in the input image to generate an output image. In aspects, method <NUM> includes verifying the authenticity of the image. The method <NUM> may be performed including additional or fewer operations than what is illustrated or in a different order.

In an example use case for method <NUM>, a user of a user computing device utilizes a camera module of the user computing device to take an input image (photograph). An encoder system implemented on the user computing device receives the input image. The encoder system performs a feature recognition on the input image to determine a plurality of determined features of the input image. The encoder system generates an output defining values representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented on a memory device of the user computing device. The message encoding neural network generates a signature from the output defining values representing the determined features of the input image. The encoder system embeds the signature in the input image to generate an output image that is stored on the memory device of the user computing device.

In another example use case for method <NUM>, images are stored in an image repository. An encoder system implemented on a server computing device receives an input image from the image repository. For example, an image repository can be utilized by the operator of a server computing device to store images uploaded by users of a service provided by the operator. The encoder system performs a feature recognition on the input image to determine a plurality of determined features of the input image. The encoder system generates an output defining values representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented on a memory device of the server computing device. The message encoding neural network generates a signature from the output defining values representing the determined features of the input image. The encoder system embeds the signature in the input image to generate an output image that is stored on the memory device of the server computing device. The embedded signature can be utilized to verify the image at a later time.

<FIG> illustrates an example method <NUM> of verifying the authenticity of an image. The method <NUM> may be performed by a system <NUM> as described above with respect to <FIG>, utilizing one or more of the components described with respect to <FIG>. At <NUM>, a decoder system receives an image to be verified. At <NUM>, the decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system, at <NUM>, generates a first output defining values representing the determined features of the received image. At <NUM>, the decoder system utilizes a message decoding neural network to decode the received image to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The decoder system, at <NUM>, generates a second output that defines values representing the recovered features of the received image. At <NUM>, the decoder system provides the first output and the second output to a manipulation detection neural network. At <NUM>, the manipulation detection neural network generates an estimation of an authenticity of the received image utilizing at least the first output and the second output. In aspects, method <NUM> includes verifying the authenticity of the image. The method <NUM> may be performed including additional or fewer operations than what is illustrated or in a different order.

In an example use case for method <NUM>, a decoder system implemented on a user computing device receives an image to be verified from a memory device implemented on the user computing device. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output defining values representing the determined features of the received image. The decoder system utilizes a message decoding neural network implemented on the user computing device to decode the received image to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The decoder system generates a second output that defines values representing the recovered features of the received image. The decoder system provides the first output and the second output to a manipulation detection neural network implemented on the user computing device. The manipulation detection neural network generates an estimation of an authenticity of the received image utilizing at least the first output and the second output.

In another example use case for method <NUM>, a decoder system implemented on a server computing device receives an image to be verified from an image repository. For example, an image repository can be utilized by the operator of a server computing device to store images uploaded by users of a service provided by the operator. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output defining values representing the determined features of the received image. The decoder system utilizes a message decoding neural network implemented on the server computing device to decode the received image to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The decoder system generates a second output that defines values representing the recovered features of the received image. The decoder system provides the first output and the second output to a manipulation detection neural network implemented on the server computing device. The manipulation detection neural network generates an estimation of an authenticity of the received image utilizing at least the first output and the second output.

<FIG> depicts a block diagram of an example machine-learned model <NUM> according to example implementations of the present disclosure. As illustrated in <FIG>, the machine-leamed model <NUM> is trained to receive input data of one or more types and, in response, to provide output data of one or more types. Thus, <FIG> illustrates the machine-learned model <NUM> performing inference.

The input data can include one or more features that are associated with an instance or an example. In some implementations, the one or more features associated with the instance or example can be organized into a feature vector. In some implementations, the output data can include one or more predictions. Predictions can also be referred to as inferences. Thus, given features associated with a particular instance, the machine-leamed model can output a prediction for such instance based on the features.

The machine-learned model can be or include one or more of various different types of machine-learned models. In particular, in some implementations, the machine-leamed model can perform classification, regression, clustering, association, anomaly detection, recommendation generation, and/or other tasks.

The machine-leamed model can perform various types of classification based on the input data. For example, the machine-leamed model can perform binary classification or multiclass classification. In binary classification, the output data can include a classification of the input data into one of two different classes. In multiclass classification, the output data can include a classification of the input data into one (or more) of more than two classes. The classifications can be a single label or multi-label.

The machine-leamed model can perform discrete categorical classification in which the input data is simply classified into one or more classes or categories.

The machine-leamed model can perform classification in which the machine-learned model provides, for each of one or more classes, a numerical value descriptive of a degree to which it is believed that the input data should be classified into the corresponding class. In some instances, the numerical values provided by the machine-leamed model can be referred to as "confidence scores" that are indicative of respective confidence associated with the classification of the input into the respective class. In some implementations, the confidence scores can be compared to one or more thresholds to render a discrete categorical prediction. In some implementations, only a certain number of classes (e.g., one) with the relatively largest confidence scores can be selected to render a discrete categorical prediction.

The machine-learned model can provide a probabilistic classification. For example, the machine-learned model can be able to predict, given a sample input, a probability distribution over a set of classes. Thus, rather than outputting only the most likely class to which the sample input should belong, the machine-leamed model can output, for each class, a probability that the sample input belongs to such class. In some implementations, the probability distribution over all possible classes can sum to one. In some implementations, a softmax function or layer can be used to squash a set of real values respectively associated with the possible classes to a set of real values in the range (<NUM>, <NUM>) that sum to one.

The probabilities provided by the probability distribution can be compared to one or more thresholds to render a discrete categorical prediction. In some implementations, only a certain number of classes (e.g., one) with the relatively largest predicted probability can be selected to render a discrete categorical prediction.

The machine-leamed model can be trained using supervised learning techniques in implementations in which the machine-learned model performs classification. For example, the machine-learned model can be trained on a training dataset that includes training examples labeled as belonging (or not belonging) to one or more classes. Further details regarding supervised training techniques are provided below.

The machine-learned model can perform regression to provide output data in the form of a continuous numeric value. The continuous numeric value can correspond to any number of different metrics or numeric representations, including, for example, currency values, scores, or other numeric representations. As examples, the machine-leamed model can perform linear regression, polynomial regression, or nonlinear regression. As examples, the machine-leamed model can perform simple regression or multiple regression. As described above, in some implementations, a softmax function or layer can be used to squash a set of real values respectively associated with two or more possible classes to a set of real values in the range (<NUM>, <NUM>) that sum to one.

The machine-leamed model can perform various types of clustering. For example, the machine-leamed model can identify one or more previously-defined clusters to which the input data most likely corresponds. As another example, the machine-learned model can identify one or more clusters within the input data. That is, in instances in which the input data includes multiple objects, documents, or other entities, the machine-learned model can sort the multiple entities included in the input data into a number of clusters. In some implementations in which the machine-leamed model performs clustering, the machine-learned model can be trained using unsupervised learning techniques.

The machine-learned model can perform anomaly detection or outlier detection. For example, the machine-leamed model can identify input data that does not conform to an expected pattern or other characteristics (e.g., as previously observed from previous input data). As examples, anomaly detection can be used for fraud detection or system failure detection.

The machine-leamed model can provide output data in the form of one or more recommendations. For example, the machine-leamed model can be included in a recommendation system or engine. As an example, given input data that describes previous outcomes for certain entities (e.g., a score, ranking, or rating indicative of an amount of success or enjoyment), the machine-learned model can output a suggestion or recommendation of one or more additional entities that, based on the previous outcomes, are expected to have a desired outcome (e.g., elicit a score, ranking, or rating indicative of success or enjoyment). As one example, given input data descriptive of a number of products purchased or rated highly by a user, a recommendation system can output a suggestion or recommendation of an additional product that the user might enjoy or wish to purchase.

The machine-leamed model can act as an agent within an environment. For example, the machine-learned model can be trained using reinforcement learning, which will be discussed in further detail below.

The machine-leamed model can be a parametric model, while, in other implementations, the machine-learned model can be a non-parametric model. In some implementations, the machine-learned model can be a linear model, while, in other implementations, the machine-learned model can be a non-linear model.

As described above, the machine-leamed model can be or include one or more of various different types of machine-leamed models. Examples of such different types of machine-leamed models are provided below for illustration. One or more of the example models described below can be used (e.g., combined) to provide the output data in response to the input data. Additional models beyond the example models provided below can be used as well.

The machine-learned model can be or include one or more classifier models such as, for example, linear classification models, quadratic classification models, etc..

The machine-leamed model can be or include one or more regression models such as, for example, simple linear regression models, multiple linear regression models, logistic regression models, stepwise regression models, multivariate adaptive regression splines; locally estimated scatterplot smoothing models; etc..

The machine-leamed model can be or include one or more decision tree-based models such as, for example, classification and/or regression trees; ID3 (Iterative Dichotomiser <NUM>) decision trees; C4. <NUM> decision trees; chi-squared automatic interaction detection decision trees; decision stumps; conditional decision trees; etc..

The machine-leamed model can be or include one or more kernel machines. In some implementations, the machine-leamed model can be or include one or more support vector machines.

The machine-learned model can be or include one or more instance-based learning models such as, for example, leaming vector quantization models, self-organizing map models, locally weighted leaming models, etc..

The machine-leamed model can be or include one or more nearest neighbor models such as, for example, k-nearest neighbor classifications models; k-nearest neighbors regression models; etc..

The machine-learned model can be or include one or more Bayesian models such as, for example, naive Bayes models, Gaussian naive Bayes models, multinomial naive Bayes models; averaged one-dependence estimators; Bayesian networks; Bayesian belief networks; hidden Markov models; etc..

The machine-learned model can be or include one or more artificial neural networks (also referred to simply as neural networks). A neural network can include a group of connected nodes, which also can be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be referred to as "deep" networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer. The nodes of the neural network can be connected or non-fully connected.

The machine-learned model can be or include one or more feed-forward neural networks. In feed-forward networks, the connections between nodes do not form a cycle. For example, each connection can connect a node from an earlier layer to a node from a later layer.

The machine-learned model can be or include one or more recurrent neural networks. In some instances, at least some of the nodes of a recurrent neural network can form a cycle. Recurrent neural networks can be especially useful for processing input data that is sequential in nature. In particular, in some instances, a recurrent neural network can pass or retain information from a previous portion of the input data sequence to a subsequent portion of the input data sequence through the use of recurrent or directed cyclical node connections.

As one example, sequential input data can include time-series data (e.g., sensor data versus time or imagery captured at different times). For example, a recurrent neural network can analyze sensor data versus time to detect or predict a swipe direction, to perform handwriting recognition, etc. As another example, sequential input data can include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.); notes in a musical composition; sequential actions that were taken by a user (e.g., to detect or predict sequential application usage); sequential object states; etc..

Example recurrent neural networks include long short-term (LSTM) recurrent neural networks; gated recurrent units; bi-direction recurrent neural networks; continuous-time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recursive neural networks; Hopfield networks; fully recurrent networks; sequence-to-sequence configurations; etc..

The machine-learned model can be or include one or more convolutional neural networks. In some instances, a convolutional neural network can include one or more convolutional layers that perform convolutions over input data using learned filters. Filters can also be referred to as kernels. Convolutional neural networks can be especially useful for vision problems such as when the input data includes imagery such as still images or video. However, convolutional neural networks can also be applied for natural language processing.

The machine-learned model can be or include one or more generative networks such as, for example, generative adversarial networks. Generative networks can be used to generate new data, such as new images or other content.

The machine-learned model can be or include an autoencoder. In some instances, the aim of an autoencoder is to learn a representation (e.g., a lower-dimensional encoding) for a set of data, typically for the purpose of dimensionality reduction. For example, in some instances, an autoencoder can seek to encode the input data and then provide output data that reconstructs the input data from the encoding. Recently, the autoencoder concept has become more widely used for leaming generative models of data. In some instances, the autoencoder can include additional losses beyond reconstructing the input data.

The machine-learned model can be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

One or more neural networks can be used to provide an embedding based on the input data. For example, the embedding can be a representation of knowledge abstracted from the input data into one or more learned dimensions. In some instances, embeddings can be a useful source for identifying related entities. In some instances, embeddings can be extracted from the output of the network, while in other instances, embeddings can be extracted from any hidden node or layer of the network (e.g., a close to final but not the final layer of the network). Embeddings can be useful for performing auto-suggest next video, product suggestion, entity, or object recognition, etc. In some instances, embeddings are useful inputs for downstream models. For example, embeddings can be useful to generalize input data (e.g., search queries) for a downstream model or processing system.

The machine-learned model can include one or more clustering models such as, for example, k-means clustering models, k-medians clustering models, expectation-maximization models, hierarchical clustering models; etc..

The machine-learned model can perform one or more dimensionality reduction techniques such as, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc..

The machine-learned model can perform or be subjected to one or more reinforcement learning techniques such as Markov decision processes, dynamic programming; Q functions or Q-learning; value function approaches; deep Q-networks; differentiable neural computers; asynchronous advantage actor-critics; deterministic policy gradient; etc..

The machine-learned model can be an autoregressive model. In some instances, an autoregressive model can specify that the output data depends linearly on its own previous values and on a stochastic term. In some instances, an autoregressive model can take the form of a stochastic difference equation. One example autoregressive model is WaveNet, which is a generative model for raw audio.

The machine-leamed model can include or form part of a multiple model ensemble. As one example, bootstrap aggregating can be performed, which can also be referred to as "bagging. " In bootstrap aggregating, a training dataset is split into a number of subsets (e.g., through random sampling with replacement) and a plurality of models are respectively trained on the number of subsets. At inference time, respective outputs of the plurality of models can be combined (e.g., through averaging, voting, or other techniques) and used as the output of the ensemble.

One example model ensemble is a random forest, which can also be referred to as a random decision forest. Random forests are an ensemble leaming method for classification, regression, and other tasks. Random forests are generated by producing a plurality of decision trees at training time. In some instances, at inference time, the class that is the mode of the classes (classification) or the mean prediction (regression) of the individual trees can be used as the output of the forest. Random decision forests can correct for decision trees' tendency to overfit their training set.

Another example ensemble technique is stacking, which can, in some instances, be referred to as stacked generalization. Stacking includes training a combiner model to blend or otherwise combine the predictions of several other machine-learned models. Thus, a plurality of machine-learned models (e.g., of the same or of a different type) can be trained based on training data. In addition, a combiner model can be trained to take the predictions from the other machine-leamed models as inputs and, in response, produce a final inference or prediction. In some instances, a single-layer logistic regression model can be used as the combiner model.

Another example ensemble technique is boosting. Boosting can include incrementally building an ensemble by iteratively training weak models and then adding to a final strong model. For example, in some instances, each new model can be trained to emphasize the training examples that previous models misinterpreted (e.g., misclassified). For example, a weight associated with each of such misinterpreted examples can be increased. One common implementation of boosting is AdaBoost, which can also be referred to as Adaptive Boosting. Other example boosting techniques include Linear Programming Boosting (LPBoost); TotalBoost; BrownBoost; XGBoost; MadaBoost, LogitBoost, gradient boosting; etc..

Furthermore, any of the models described above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. As an example, an ensemble can include a top-level machine-learned model or a heuristic function to combine and/or weight the outputs of the models that form the ensemble.

Multiple machine-leamed models (e.g., that form an ensemble) can be linked and trained jointly (e.g., through backpropagation of errors sequentially through the model ensemble). However, in some implementations, only a subset (e.g., one) of the jointly trained models is used for inference.

The machine-learned model can be used to preprocess the input data for subsequent input into another model. For example, the machine-learned model can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal components analysis, singular value decomposition, Word2vec/GloVe, and/or related approaches); clustering; and even classification and regression for downstream consumption. Many of these techniques have been discussed above and will be further discussed below.

Referring again to <FIG>, and as discussed above, the machine-leamed model can be trained or otherwise configured to receive the input data and, in response, provide the output data. The input data can include different types, forms, or variations of input data. As examples, in various implementations, the input data can include determined image features and/or image features provided by a user.

The machine-learned model can receive and use the input data in its raw form. In some implementations, the raw input data can be preprocessed. Thus, in addition, or alternatively to the raw input data, the machine-leamed model can receive and use the preprocessed input data.

Preprocessing the input data can include extracting one or more additional features from the raw input data. For example, feature extraction techniques can be applied to the input data to generate one or more new, additional features. Example feature extraction techniques include edge detection; corner detection; blob detection; ridge detection; scale-invariant feature transform; motion detection; optical flow; Hough transform; etc..

The extracted features can include or be derived from transformations of the input data into other domains and/or dimensions. As an example, the extracted features can include or be derived from the transformation of the input data into the frequency domain. For example, wavelet transformations and/or fast Fourier transforms can be performed on the input data to generate additional features.

The extracted features can include statistics calculated from the input data or certain portions or dimensions of the input data. Example statistics include the mode, mean, maximum, minimum, or other metrics of the input data or portions thereof.

As described above, the input data can be sequential in nature. In some instances, the sequential input data can be generated by sampling or otherwise segmenting a stream of input data. As one example, frames can be extracted from a video. In some implementations, sequential data can be made non-sequential through summarization.

As another example preprocessing technique, portions of the input data can be imputed. For example, additional synthetic input data can be generated through interpolation and/or extrapolation.

As another example preprocessing technique, some or all of the input data can be scaled, standardized, normalized, generalized, and/or regularized. Example regularization techniques include ridge regression; least absolute shrinkage and selection operator (LASSO); elastic net; least-angle regression; cross-validation; L1 regularization; L2 regularization; etc. As one example, some or all of the input data can be normalized by subtracting the mean across a given dimension's feature values from each individual feature value and then dividing by the standard deviation or another metric.

As another example preprocessing technique, some or all of the input data can be quantized or discretized. As yet another example, qualitative features or variables included in the input data can be converted to quantitative features or variables. For example, one hot encoding can be performed.

Dimensionality reduction techniques can be applied to the input data prior to input into the machine-learned model. Several examples of dimensionality reduction techniques are provided above, including, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc..

During training, the input data can be intentionally deformed in any number of ways to increase model robustness, generalization, or other qualities. Example techniques to deform the input data include adding noise; changing color, shade, or hue; magnification; segmentation; amplification; etc..

Referring again to <FIG>, in response to receipt of the input data, the machine-learned model <NUM> can provide the output data. The output data can include different types, forms, or variations of output data. As examples, in various implementations, the output data can include values that represent features of an image (e.g., an input image, an output image, a received image), values that represent image features embedded in the received image, and/or a prediction (e.g., an estimate) of the authenticity of the an image.

As discussed above, the output data can include various types of classification data (e.g., binary classification, multiclass classification, single label, multi-label, discrete classification, regressive classification, probabilistic classification, etc.) or can include various types of regressive data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multiple regression, etc.). In other instances, the output data can include clustering data, anomaly detection data, recommendation data, or any of the other forms of output data discussed above.

The output data can influence downstream processes or decision making. As one example, in some implementations, the output data can be interpreted and/or acted upon by a rules-based regulator.

In aspects, when a machine-learned model is stored on a computing system (e.g., user computing system <NUM>), software encryption rules (e.g., a secure hashing algorithm) can be utilized to protect the integrity of the model and prevent a third-party from tampering with a model (e.g., replacing part of a machine-learned model with another model). In aspects, an alert signal may be generated responsive to the detection of attempted tampering.

Thus, the present disclosure provides systems and methods that include or otherwise leverage one or more machine-learned models to generate a prediction (e.g., an estimate) of the authenticity of an image based on determined features and/or recovered image features embedded in the image. Any of the different types or forms of input data described above can be combined with any of the different types or forms of machine-learned models described above to provide any of the different types or forms of output data described above.

The machine-leamed model <NUM> can be stored at and/or implemented locally by a computing system (e.g., computing system <NUM> of <FIG>). For example, the machine-leamed model <NUM> can be stored at and/or implemented locally by a user computing device or an embedded computing device. Output data obtained through local implementation of the machine-leamed model at the computing system can be used to improve the performance of the computing system (e.g., an application implemented by the computing system). As one example, <FIG> illustrates a block diagram of a user computing system <NUM> (e.g., a mobile computing device) that stores and implements a machine-leamed model <NUM> locally.

The machine-learned model can be stored at and/or implemented by a server computing device (such as server computing system <NUM> of <FIG>). In some instances, output data obtained through the implementation of the machine-learned model at the server computing device can be used to improve other server tasks or can be used by other non-user devices to improve services performed by or for such other non-user devices. For example, the output data can improve other downstream processes performed by the server computing device for a user computing device or embedded computing device. In other instances, output data obtained through the implementation of the machine-learned model at the server computing device can be sent to and used by a user computing device, an embedded computing device, or some other client device. For example, the server computing device can be said to perform machine leaming as a service. As one example, <FIG> illustrates a block diagram of an example user computing system <NUM> that can communicate over a network <NUM> with an example server computing system <NUM> that includes a machine-learned model <NUM>.

Different respective portions of the machine-learned model can be stored at and/or implemented by some combination of a user computing device, an embedded computing device, a server computing device, etc..

Computing devices can perform graph processing techniques or other machine learning techniques using one or more machine learning platforms, frameworks, and/or libraries, such as, for example, TensorFlow, Caffe/Caffe2, Theano, Torch/PyTorch, MXNet, Cognitive Toolkit (CNTK), etc..

Computing devices can be distributed at different physical locations and connected via one or more networks. Distributed computing devices can operate according to sequential computing architectures, parallel computing architectures, or combinations thereof. In one example, distributed computing devices can be controlled or guided through the use of a parameter server.

Multiple instances of the machine-leamed model can be parallelized to provide increased processing throughput. For example, the multiple instances of the machine-leamed model can be parallelized on a single processing device or computing device or parallelized across multiple processing devices or computing devices.

The machine-learned models described herein can be trained at a training computing system and then provided for storage and/or implementation at one or more computing devices, as described above. For example, a model trainer <NUM> can be located at the training computing system <NUM>, as illustrated in <FIG>. The training computing system <NUM> can be included in or separate from the one or more computing devices that implement the machine-leamed model. As one example, <FIG> illustrates a block diagram of an example user computing system <NUM> in communication with an example training computing system <NUM> that includes a model trainer <NUM>.

The machine-leamed model can be trained in an offline fashion or an online fashion. In offline training (also known as batch learning), a model is trained on the entirety of a static set of training data. In online learning, the model is continuously trained (or retrained) as new training data becomes available (e.g., while the model is used to perform inference).

The model trainer can perform centralized training of the machine-leamed models (e.g., based on a centrally stored dataset). In other implementations, decentralized training techniques such as distributed training, federated learning, or the like can be used to train, update, or personalize the machine-learned models.

The machine-learned models described herein can be trained according to one or more of various different training types or techniques. For example, in some implementations, the machine-learned models can be trained using supervised learning, in which the machine-learned model is trained on a training dataset that includes instances or examples that have labels. The labels can be manually applied by experts, generated through crowd-sourcing, or provided by other techniques (e.g., by physics-based or complex mathematical models). In some implementations, if the user has provided consent, the training examples can be provided by the user computing device. In some implementations, this process can be referred to as personalizing the model.

The machine-learned model can be trained by optimizing an objective function. For example, in some implementations, the objective function can be or include a loss function that compares (e.g., determines a difference between) output data generated by the model from the training data and labels (e.g., ground-truth labels) associated with the training data. For example, the loss function can evaluate a sum or mean of squared differences between the output data and the labels. As another example, the objective function can be or include a cost function that describes the cost of a certain outcome or output data. Other objective functions can include margin-based techniques such as, for example, triplet loss or maximum-margin training.

One or more of various optimization techniques can be performed to optimize the objective function. For example, the optimization technique(s) can minimize or maximize the objective function. Example optimization techniques include Hessian-based techniques and gradient-based techniques, such as, for example, coordinate descent, gradient descent (e.g., stochastic gradient descent), subgradient methods, etc. Other optimization techniques include black-box optimization techniques and heuristics.

Backward propagation of errors can be used in conjunction with an optimization technique (e.g., gradient-based techniques) to train a model (e.g., a multi-layer model such as an artificial neural network). For example, an iterative cycle of propagation and model parameter (e.g., weights) update can be performed to train the model. Example backpropagation techniques include truncated backpropagation through time, Levenberg-Marquardt backpropagation, etc..

The machine-learned models described herein can be trained using unsupervised leaming techniques. Unsupervised leaming can include inferring a function to describe a hidden structure from unlabeled data. For example, classification or categorization may not be included in the data. Unsupervised leaming techniques can be used to produce machine-leamed models capable of performing clustering, anomaly detection, leaming latent variable models, or other tasks.

The machine-leamed models described herein can be trained using semi-supervised techniques that combine aspects of supervised leaming and unsupervised learning.

The machine-leamed models described herein can be trained or otherwise generated through evolutionary techniques or genetic algorithms.

The machine-learned models described herein can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., model) can take actions in an environment and learn to maximize rewards and/or minimize penalties that result from such actions. Reinforcement leaming can differ from the supervised learning problem in that correct input/output pairs are not presented, nor sub-optimal actions explicitly corrected.

One or more generalization techniques can be performed during training to improve the generalization of the machine-learned model. Generalization techniques can help reduce the overfitting of the machine-learned model to the training data. Example generalization techniques include dropout techniques, weight decay techniques, batch normalization, early stopping, subset selection, stepwise selection, etc..

The machine-leamed models described herein can include or otherwise be impacted by a number of hyperparameters, such as, for example, learning rate, number of layers, number of nodes in each layer, number of leaves in a tree, number of clusters; etc. Hyperparameters can affect model performance. Hyperparameters can be hand-selected or can be automatically selected through the application of techniques such as, for example, grid search; black-box optimization techniques (e.g., Bayesian optimization, random search, etc.); gradient-based optimization; etc. Example techniques and/or tools for performing automatic hyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE); etc..

Various techniques can be used to optimize and/or adapt the learning rate when the model is trained. Example techniques and/or tools for performing leaming rate optimization or adaptation include AdaGrad, Adaptive Moment Estimation (ADAM), ADADELTA, RMSprop, etc..

Transfer learning techniques can be used to provide an initial model from which to begin training of the machine-learned models described herein.

The machine-leamed models described herein can be included in different portions of computer-readable code on a computing device. In one example, the machine-learned model can be included in a particular application or program and used (e.g., exclusively) by such a particular application or program. Thus, in one example, a computing device can include a number of applications, and one or more of such applications can contain its own respective machine leaming library and machine-learned model(s).

The machine-learned models described herein can be included in an operating system of a computing device (e.g., in a central intelligence layer of an operating system) and can be called or otherwise used by one or more applications that interact with the operating system. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an application programming interface (API) (e.g., a common, public API across all applications).

The central device data layer can be a centralized repository of data for the computing device. The central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components.

The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken, and information sent to and from such systems.

In addition, the machine leaming techniques described herein are readily interchangeable and combinable. Although certain example techniques have been described, many others exist and can be used in conjunction with aspects of the present disclosure.

Thus, while the present subject matter has been described in detail with respect to various specific example implementations, each example is provided by way of explanation, not a limitation of the disclosure. One of ordinary skill in the art can readily make alterations to, variations of, and equivalents to such implementations. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one implementation can be used with another implementation to yield a still further implementation.

Claim 1:
A method comprising:
receiving, by a decoder system (<NUM>), an image (<NUM>) to be verified;
performing feature recognition on the received image (<NUM>) to determine a plurality of determined features (<NUM>) of the received image (<NUM>);
generating a first output (<NUM>) defining values representing the determined features (<NUM>) of the received image (<NUM>);
decoding the received image (<NUM>), by a message decoding neural network (<NUM>) of the decoder system (<NUM>), to extract a signature (<NUM>) embedded in the received image (<NUM>), the signature (<NUM>) representing recovered features (<NUM>) of the received image (<NUM>);
generating a second output (<NUM>) defining values representing the recovered features (<NUM>) of the received image (<NUM>);
providing the first output (<NUM>) and the second output (<NUM>) to a manipulation detection neural network (<NUM>) of the decoder system (<NUM>); and
generating, by the manipulation detection neural network (<NUM>), an estimation of an authenticity of the received image (<NUM>) utilizing at least the first output (<NUM>) and the second output (<NUM>).