Patent Description:
Image and video files are commonly encoded using a variety of encoding schemes for a number of different applications. Large data entities require significant resource investment and allocation of storage infrastructure to store these images and/or video files. As a result, these files often must be encoded using a lossy compression scheme to enable more efficient storage, retrieval and transmission.

However, encoding images and/or video files using a lossy compression scheme can cause a significant loss of image fidelity, and lossless compression alternatives rarely offer the space reduction necessary to enable efficient storage. As a result, when storing and transmitting image and/or video files, a choice must be made between loss of fidelity and significant resource investment for additional storage infrastructure.

<CIT> discloses methods for using digital watermarking to authenticate digital media signals, such as images, audio and video signals. It also describes techniques for using embedded watermarks to repair altered parts of a media signal when alteration is detected. Alteration is detected using hashes, digital watermarks, and a combination of hashes and digital watermarks.

One example aspect of the present disclosure is directed to a computer-implemented method to perform watermark-based image reconstruction to compensate for lossy encoding schemes. The method includes obtaining, by one or more computing devices, an input image. The method includes generating, by the one or more computing devices, a first output image by encoding and decoding the input image according to an encoding scheme. The method includes determining, by the one or more computing devices, a difference image that describes a difference between the input image and the first output image. The method includes generating, by the one or more computing devices and using a machine-learned message embedding model, a second output image that comprises an embedded message that is based at least in part on the difference image.

Example embodiments of the present disclosure are directed to systems and methods for watermark-based image reconstruction using machine-learned models. In particular, systems and methods described herein are directed to using a machine-learned message embedding model to embed a message into an image, where the message represents data lost from the image due to an encoding (and decoding) process. The embedded message can later be extracted from the image by a machine-learned message extraction model and the extracted message can be used in reconstructing the original image. Thus, as one example, an image can be compressed using a lossy compression technique (e.g., JPEG compression) to generate a first output image. A message that represents the data lost from the image due to such compression can be embedded (e.g., watermarked as image noise) within the original image to generate a second output image. The embedded message can be extracted from the second output image and can be used to reconstruct the original image from the second output image, thereby at least partially reversing the loss of image fidelity caused by compression. The proposed techniques represent a significant advancement in reconstructing images that have suffered data loss from image encoding processes. In particular, by capturing and embedding the data lost from the image during compression as a message, the proposed systems provide a method for image reconstruction that can produce reconstructed images more accurately than conventional techniques.

As one example, computing devices (e.g., a distributed network of computing devices) can obtain an input image (e.g., a RAW image). The computing devices can generate a first output image by encoding and decoding the image according to an encoding scheme. As one example, the computing devices may encode and decode the input image using a lossy JPEG compression scheme, the first output image being a decoded JPEG representation of the input image. A difference image can be determined that describes a difference between the first image and the first output image. For example, the difference image can describe the data lost as a pixel-by-pixel difference between the input image and the first output image. A machine-learned message embedding model can embed a message based on the difference image into the first image (e.g., as an image noise watermark) to produce a second output image. As one example, the data lost from JPEG compression can be represented as a latent space message vector. A watermark can be generated based on the message vector. The message can be embedded into the JPEG (e.g., by applying the watermark as image noise) to produce a second output image. The second output image can be encoded and then stored or transmitted.

The encoded second output image can be decoded, and using a machine-learned message extraction model, the message vector can be extracted from the second output image and used in reconstructing the difference image. For example, a machine-learned watermark extraction model can extract the message vector and a machine-learned difference reconstruction model can use the extracted message vector to generate a reconstruction of the difference image, which in turn can be used to reconstruct the input image. As one example, the input image can be reconstructed by adding the reconstructed difference image to the second output image. Thus, although the input image of the above examples was degraded by a lossy compression scheme, an identical or near-identical version of the input image was able to be reconstructed using the embedded message watermarked in the image.

The present disclosure provides a number of technical effects and benefits. As one example technical effect and benefit, the systems and methods of the present disclosure enable a significant advancement in reconstructed image quality compared to other approaches. Most other methods known in the art attempt to directly recover an input image from an encoded image. As an example, methods such as GIF2Video remove GIF artifacts by combining neural networks and the Lucas-Kanade method (See <NPL>)). However, under this approach, the reconstruction accuracy of a machine-learned model is necessarily limited because it cannot utilize the data that was initially lost in encoding. The present method differs from these previous approaches by embedding (e.g., watermarking) a message describing data lost during the encoding step into the image. Later, the message can be extracted and used to reconstruct the input image. In this fashion, a machine-learned model(s) can utilize the initial data lost during encoding and therefore produce a more accurate reconstructed image.

As another example technical effect and benefit, the systems and methods of the present disclosure enable a number of image encoding schemes (e.g., graphics interchange format (GIF) encoding) to be used in situations where they previously would not be chosen. As one example, a lossy GIF encoding scheme may not have previously been chosen in certain situations due to the data loss inherent to some lossy GIF encoding schemes. Using the method of the present disclosure, GIF encoded images may be reconstructed to a degree of accuracy sufficient to enable GIF encoding in situations requiring minimal image data loss. By enabling these additional encoding schemes, the present disclosure allows for more compression of images, necessarily saving storage space used to store images. Stated differently, aspects of the present disclosure represent an improvement in the curve of compression gains versus quality reduction. Thus, versus past compression techniques, additional compression gains can be achieved by the present disclosure while still retaining the same ultimate quality. These compression gains result in savings of resources such as memory usage, network bandwidth usage, etc..

Throughout the present disclosure, embodiments will be described with reference to JPEG and GIF compression, though it will be appreciated that systems and methods disclosed herein may additionally utilize other image compression techniques.

<FIG> depicts a block diagram of an example computing system <NUM> that performs embedding and extraction of messages using machine-learned models that are trained according to the present disclosure. The system <NUM> includes a first computing device <NUM> and a second computing device <NUM> that are communicatively coupled over a network <NUM>.

The first computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a personal assistant computing device, or any other type of computing device.

The first computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the first computing device <NUM> to perform operations.

According to an aspect of the present disclosure, the first computing device <NUM> can store or include one or more machine-learned models. The machine-learned models can be or can otherwise include one or more neural networks (e.g., deep neural networks) or the like. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks.

More particularly, machine-learned models can be implemented to provide embedding and extraction of messages within an input image. As one example, the machine-learned models can include a machine-learned message embedding model <NUM> and a machine-learned message extraction model <NUM>. In particular, the machine-learned message embedding model <NUM> can receive a difference image describing a difference between an input image and an encoded first output image and generate a message vector (e.g., a latent space vector) representing the difference image. The machine-learned message embedding model <NUM> can receive the message and generate a watermark representing the message vector. The watermark can be applied to the input image or the first output image to generate a second output image. The machine-learned message extraction model <NUM> can obtain the second output image as an input and extract the message vector from the second output image to obtain an extracted message vector. The machine-learned message extraction model <NUM> can receive the extracted message as an input and provide as output a reconstruction of the difference image. The reconstructed difference image can be added to the input image to generate a reconstructed input image.

The first computing device <NUM> can also include model trainer(s) <NUM>. The model trainer <NUM> can use training data <NUM> to simultaneously train or re-train machine-learned models, such as the machine-learned message embedding model <NUM> and the machine-learned message extraction model <NUM>, stored at the first computing device <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors (e.g., truncated backpropagation through time). In particular, the model trainer <NUM> can use training data <NUM> to simultaneously train or re-train the machine-learned message embedding model <NUM> and machine-learned message extraction model <NUM>. The specific training signal(s) used to train or retrain the machine-learned models will be discussed in-depth in the following figures.

The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained. Thereafter, the machine-learned message embedding model <NUM> and machine-learned message extraction model <NUM> can be used immediately to embed and extract messages in images.

Additionally, in some implementations, the machine-learned message embedding model <NUM> can include a machine-learned watermark generation model and machine-learned message generation model. Similarly, in some implementations, the machine-learned message extraction model <NUM> can include a machine-learned watermark extraction model and a machine-learned difference reconstruction model.

The first computing device <NUM> can also include one or more input/output interface(s) <NUM>. One or more input/output interface(s) <NUM> can include, for example, devices for receiving information from or providing information to a user, such as a display device, touch screen, touch pad, mouse, data entry keys, an audio output device such as one or more speakers, a microphone, haptic feedback device, etc. An input/output interface(s) <NUM> can be used, for example, by a user to control operation of the first computing device <NUM>.

The first computing device <NUM> can also include one or more communication/network interface(s) <NUM> used to communicate with one or more systems or devices, including systems or devices that are remotely located from the first computing device <NUM>. The communication/network interface(s) <NUM> can include any circuits, components, software, etc. for communicating with one or more networks (e.g., network <NUM>). In some implementations, the communication/network interface(s) <NUM> can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software, and/or hardware for communicating data.

The second computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the second computing device <NUM> to perform operations.

As described above, the second computing device <NUM> can store or otherwise include one or more machine-learned models. The machine-learned models can be or can otherwise include one or more neural networks (e.g., deep neural networks) and the neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks.

More particularly, the second computing device <NUM> can receive and store a trained machine-learned model, for example, from the first computing device <NUM> via the network <NUM>. For example, the computing device <NUM> can receive the machine learned message extraction model <NUM> (e.g., the machine-learned watermark extraction model and the machine-learned difference reconstruction model) to provide reconstruction of input images from images with embedded messages that are transmitted to computing device <NUM>. The second computing device <NUM> can use the machine-learned model(s) for the same or similar purposes as described above.

As an example, a second output image can be generated by the machine-learned message embedding model <NUM> and transmitted to the computing device <NUM> alongside and the machine-learned message extraction model <NUM> via network <NUM>. The computing device <NUM> can use the transmitted machine-learned message extraction model <NUM> to extract a message from the transmitted second output image and generate a reconstructed input image corresponding to the second output image.

The second computing device <NUM> can also include one or more input/output interface(s) <NUM>. The one or more input/output interface(s) <NUM> can include, for example, devices for receiving information from or providing information to a user, such as a display device, touch screen, touch pad, mouse, data entry keys, an audio output device such as one or more speakers, a microphone, haptic feedback device, etc. An input/output interface(s) <NUM> can be used, for example, by a user to control operation of the second computing device <NUM>.

The second computing device <NUM> can also include one or more communication/network interface(s) <NUM> used to communicate with one or more systems or devices, including systems or devices that are remotely located from the second computing device <NUM>. The communication/network interface(s) <NUM> can include any circuits, components, software, etc. for communicating with one or more networks (e.g., network <NUM>). In some implementations, the communication/network interface(s) <NUM> can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software, and/or hardware for communicating data.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well.

<FIG> depicts a flow diagram of an example method for generating a second output image according to example embodiments of the present disclosure. The method can be performed by one or more computing devices (e.g., computing device <NUM>, computing device <NUM>, etc.), and includes, for example, obtaining an input image <NUM>. The input image <NUM> can be a digital image file formatted in a RAW format, a raster format (e.g., bitmap image file (BMP), tagged image file format (TIFF), etc.), a vector format (e.g., computer graphics metafile (CGM), scalable vector graphics (SVG), etc.), or any other known image file format. The input image <NUM> can be obtained by one or more computing devices. In some implementations, the one or more computing devices can include one or more sensors (e.g., a digital camera) configured to capture an input image. In other implementations, the one or more computing devices can obtain the input image from another computing device.

In some implementations, the input image <NUM> that is obtained may include a plurality of frames. As one example, the input image <NUM> can be formatted in a format that allows for a plurality of image frames to be included in the image (e.g., graphics interchange format (GIF), WEBM, WEBP, etc.). As another example, the input image <NUM> can be formatted as a video format that allows for a plurality of image frames to be included in the image (e.g., MP4, VID, MPEG, AVI, etc.). It will be apparent to those skilled in the art that the following methods and processes can be applied, sequentially or non-sequentially, to each frame of a plurality of frames included in the input image <NUM>.

The input image can go through encoding/decoding scheme <NUM> to produce a first output image <NUM>. In some implementations, the encoding scheme applied to the input image <NUM> can be a differentiable lossy compression scheme. As one example, an input image <NUM> may be encoded and decoded using a differentiable JPEG compression scheme <NUM> to produce a first output image <NUM>, the first output image <NUM> being formatted as a JPEG. The resulting first output image <NUM>, encoded as a JPEG, may have lost data due to the lossy nature of the compression scheme used. As another example, the input image <NUM> including a plurality of frames may be encoded and decoded using a differentiable GIF compression scheme <NUM>. Each frame of the resulting first output image <NUM>, formatted as a GIF, can lose data due to the lossy nature of the compression scheme used.

A difference image <NUM> can be determined that describes a difference between the input image <NUM> and the first output image <NUM>. More particularly, the difference image <NUM> can describe the data lost from encoding/decoding the input image <NUM> to the first output image <NUM> in the encoding/decoding scheme <NUM>. In some implementations, the difference image <NUM> can represent the change in pixel values from encoding the first image <NUM>.

A message vector <NUM> (e.g., a latent space vector) can be generated by a machine-learned message embedding model based at least in part on the difference image <NUM>. In some implementations, the message vector <NUM> representation of the difference image <NUM> can be generated using a machine-learned message generation model <NUM> of the machine-learned message embedding model. In some implementations, the message vector <NUM> can be generated using an autoencoder (e.g., the machine-learned message generation model <NUM>). However, it should be noted that the message vector <NUM> can be represented in a format other than a latent space vector. The format of the message, depending on the machine-learned model that is used, can be any type of encoded representation of the difference image <NUM>. By representing the difference image <NUM> as a message vector that is reduced to its latent space vector representation, the difference image <NUM> can generally be reduced in size. In this fashion, the message can be more easily embedded in an encoded image without substantially increasing the space required to store the encoded image.

A second output image <NUM> can be generated by the machine-learned embedding model that includes the message vector <NUM>. More particularly, a watermark (e.g., image noise) representing the message vector <NUM> can be generated and added to the input image <NUM> to obtain the second output image <NUM>. Alternatively, in some implementations, the watermark can be added to the first output image <NUM> to obtain the second output image <NUM>. In some implementations, the watermark can be generated by a machine-learned watermark generation model <NUM> of the message embedding model. In some implementations, the watermark can be generated by an autoencoder (e.g., the machine-learned watermark generation model <NUM>).

In some implementations, the watermark can be applied to the image (e.g., input image <NUM> or first output image <NUM>) by modifying pixel values associated with the image (e.g., RGB channel values, intensity values, etc.). As one example, the message can be watermarked in the image as image noise (e.g., random variations of brightness and/or color information). As another example, the message can be watermarked as image blur (e.g., blurring of one or more portions of pixels of the image). Although image noise and image blurring are given as examples, any other form of pixel value modification can be used to watermark the message in the second output image <NUM>. Further, it should be noted that the second output image <NUM> is encoded in the same manner as the first output image <NUM>.

In some implementations, the machine-learned message embedding model (e.g., the machine-learned watermark generation model <NUM>) can take both the message vector <NUM> and the input image <NUM> as inputs to generate the second output image <NUM> embedded with the message. The inclusion of the input image <NUM> before it is encoded and decoded using encoding/decoding scheme <NUM> can, in some instances, improve the performance of the machine-learned message embedding model by applying the encoding scheme to the input image <NUM> and adding the message vector <NUM> (e.g., adding a watermark representing the message vector <NUM>) to the input image <NUM> simultaneously. As one example, the input image <NUM> can be encoded as a first output image <NUM> using an encoding scheme, and a message can be generated based at least in part on the difference (e.g., difference image <NUM>) between the input image <NUM> and the encoded first output image <NUM>. The machine-learned message embedding model can embed this message in the input image <NUM> while also applying the encoding scheme to the input image to produce an encoded second output image <NUM>. Including the input image <NUM> in such a fashion can, in some instances, result in a higher quality second output image <NUM> (e.g., less visual distortion from watermarking, higher message fidelity, etc.).

In some implementations, the encoded second output image <NUM> can be stored and/or transmitted to another computing device. As one example, the one or more computing devices may store the encoded second output image <NUM> for long term storage. In this fashion, the encoded second output image <NUM> requires much less storage space than the input image <NUM> while still capable of being reconstructed to a level of quality identical or substantially similar to the input image <NUM>. As another example, the one or more computing devices may transmit the encoded second output image <NUM> to another computing device. In this fashion, transmission of the encoded second output image requires less bandwidth than transmission of the input image while enabling image reconstruction to a level of quality identical or substantially similar to the input image <NUM>.

<FIG> depicts a flow diagram of an example method for reconstructing an input image from the second output image according to example embodiments of the present disclosure. The method can be performed by the one or more computing devices, and includes, for example, obtaining the second output image <NUM>. If the second output image <NUM> is encoded, the encoded second output image can be decoded to produce a decoded second output image. A machine-learned message extraction model extracts the embedded message from the second output image <NUM> to reproduce an extracted message vector <NUM>. The message is extracted by a machine-learned watermark extraction model <NUM> of the machine-learned message extraction model. The machine-learned watermark extraction model <NUM> can be or can otherwise include one or more neural networks (e.g., deep neural networks) or the like. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks. In some circumstances, the extracted message vector <NUM> reproduced by the machine-learned watermark extraction model <NUM> can lose at least some data throughout the embedding and extracting processes (e.g., encoding/decoding scheme <NUM>, embedding message vector <NUM>, etc.). However, such data loss does not necessarily render the extracted message vector <NUM> inoperable. In the case that the extracted message <NUM> is rendered inoperable, the usage of repetitive watermarking techniques can provide redundant messages that may survive to be extracted.

In some implementations, the second output image <NUM> can be evaluated by a discriminator <NUM> to determine if the second output image contains an embedded message vector <NUM>. The discriminator <NUM> can be a machine-learned discriminative model (e.g., general adversarial network, linear classifier, support vector machine (SVM), etc.). The discriminator <NUM> can be trained to determine if the second output image <NUM> contains an embedded message vector <NUM>. As one example, the discriminator <NUM> can be trained in a supervised fashion using training data including images known to contain embedded message vectors <NUM>. If the discriminator <NUM> determines that the second output image <NUM> contains an embedded message, the machine-learned message extraction model (e.g., the machine-learned message extraction model <NUM>) can take the second output image <NUM> as an input.

In some implementations, a reconstructed difference image <NUM> can be generated based at least in part on the embedded message vector <NUM> extracted by the machine-learned message extraction model. More particularly, a machine-learned difference reconstruction model <NUM> of the machine-learned message extraction model <NUM> can reconstruct the difference image based on the extracted message vector <NUM> to generate a reconstructed difference image <NUM>. As one example, an extracted message vector <NUM> representing a difference image <NUM> as a latent space vector can serve as an input to an autoencoder (e.g., the machine-learned difference reconstruction model <NUM>) to generate the reconstructed difference image <NUM>. As another example, an extracted embedded message vector <NUM> representing a difference image <NUM> as a latent space vector can serve as an input some other type of neural network architecture (e.g., feed-forward neural network, convolutional neural network, etc.) to generate the reconstructed difference image <NUM>.

A reconstructed input image <NUM> can be generated based at least in part on the decoded second output image <NUM> and the reconstructed difference image <NUM>. More particularly, the reconstructed difference image <NUM> can be added to the decoded second output image <NUM> to generate a reconstructed input image <NUM>. As one example, each pixel of the decoded second output image <NUM> can possess a difference in pixel value from the pixels of the input image <NUM>. The reconstructed difference image <NUM>, including difference values for each pixel of the decoded output image <NUM>, can be added pixel-by-pixel to the decoded output image <NUM> to reconstruct the input image (e.g., generate the reconstructed input image <NUM>).

The aforementioned machine-learned models (e.g., machine-learned message embedding and extraction models, machine-learned watermark generation and extraction models, machine-learned message generation and difference reconstruction models, discriminators, etc.) can be trained either together or simultaneously (e.g., in a joint fashion in which one or more gradients are passed from one network to another). The specifics of training the aforementioned machine-learned models with be discussed in greater detail in the subsequent figures.

<FIG> is a flowchart depicting an example method of generating a second output image in accordance with example embodiments. The method <NUM> can be implemented, for instance, using computing device(s) of <FIG>. <FIG> depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that various steps of any of the methods described herein can be omitted, rearranged, performed simultaneously, expanded, and/or modified in various ways without deviating from the scope of the present disclosure.

At <NUM>, the method can include obtaining an input image. The input image can be a digital image file formatted in a RAW format, a raster format (e.g., bitmap image file (BMP), tagged image file format (TIFF), etc.), a vector format (e.g., computer graphics metafile (CGM), scalable vector graphics (SVG), etc.), or any other known image file format.

In some implementations, the input image may include a plurality of frames. As one example, the input image can be formatted in a format that allows for a plurality of image frames to be included in the image (e.g., graphics interchange format (GIF), WEBM, WEBP, etc.). As another example, the input image can be formatted as a video format that allows for a plurality of image frames to be included in the image (e.g., MP4, VID, MPEG, AVI, etc.).

In some implementations, obtaining the input image can include receiving the input image from a computing device. The input image can be transmitted from one computing device to another through a network or through storage medium (e.g., flash storage media, a portable hard drive, etc.) In some implementations, the input image can be captured by a computing device configured to capture image data using one or more sensors (e.g., digital camera, webcam, etc.).

At <NUM>, the method can include generating a first output image by encoding and decoding the input image according to an encoding scheme. The encoding scheme can be any differentiable or approximately-differentiable encoding scheme. Although image compression schemes are primarily referenced, the encoding scheme can also be a non-compressing image and/or video encoding scheme. As one example, an input image may be encoded and decoded using a differentiable JPEG compression scheme to produce a first output image, the first output image being formatted as a JPEG. As another example, an input image including a plurality of frames may be encoded and decoded using a differentiable GIF compression scheme.

Differentiable JPEG encoding schemes have been previously explored in the art. As an example, a differentiable approximation of JPEG encoding has been described in "JPEG-resistant adversarial images" (<NPL>)). Similarly, differentiable approximation of other non-differentiable compression schemes (e.g., GIF and other image compression schemes) has been described in "Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations" (See <NPL>)). Accordingly, although differentiable JPEG encoding is used in some examples to illustrate the function of current embodiments, any differentiable or approximately differentiable image encoding scheme may be used. Further, any encoding scheme that is not currently differentiable can be used if a differentiable or approximately differentiable version or method of the encoding scheme is later developed.

At <NUM>, the method can include determining a difference image that describes a difference between the input image and the first output image. The difference image can describe the data lost from encoding/decoding the input image to the first output image in the encoding/decoding scheme. For instance, applying a differentiable lossy JPEG encoding scheme to an input image will necessarily cause the loss of at least some data on a pixel-by-pixel basis. The difference image can describe the pixel-by-pixel loss of data so that the input image can later be reconstructed. As one example, if a first pixel of the input image had a value of <NUM> before encoding and a value of <NUM> after encoding, the difference image can represent the difference between pixel values as <NUM>. This change in pixel values can be represented similarly for each pixel of the input image. Although the above example can be used to represent the difference image, it should be understood by those skilled in the art that any representation (e.g., integer, etc.) calculated in any manner (e.g., added, subtracted, etc.) can be used to represent the pixel-by-pixel difference in pixel values between the input image and the first output image.

At <NUM>, the method can include generating, using a machine-learned message embedding model, a second output image that comprises an embedded message based at least in part on the difference image. The second output image can, in some implementations, comprise the first output image with an embedded message representing the difference image. The embedded message representing the difference image can be a latent space vector representation of the difference image. This message vector (e.g., the latent space vector representation) can be generated by a machine-learned message embedding model. More specifically, a machine-learned message generation model of the machine-learned message embedding model can generate the message vector. As one example, the message vector representing the difference image can be generated using an autoencoder (e.g., the machine-learned message generation model). As another example, the message vector representing the difference image can be generated using some other type of neural network architecture (e.g., feed-forward neural network, convolutional neural network, etc.).

The machine-learned message embedding model can generate a watermark based on the message vector. More particularly, a machine-learned watermark generation model of the machine-learned message embedding model can take the message vector as an input to generate a watermark representing the message vector. In some implementations, the machine-learned watermark generation model can additionally take the input image as an input. In such fashion, the machine-learned watermark generation model can generate the watermark and apply the watermark to the input image. The machine-learned watermark generation model can be or can otherwise include one or more neural networks (e.g., deep neural networks) or the like. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks. As an example, the machine-learned watermark generation model can be an autoencoder. As another example, the machine learned watermark generation model can be a convolutional neural network.

The watermark can be added to the input image and then encoded using the encoding scheme to generate the second output image. Alternatively, in some embodiments, the watermark can be added to the first output image to generate the second output image. In some implementations, the watermark can be added to the image at multiple locations in the image. More particularly, the same watermark representing the message can be repetitively applied to different locations in the image. As one example, four corners of an image can be watermarked with the same watermark. As another example, the watermark can be applied to three randomized locations in the image. Thus, in such a fashion, repetitive watermarking can provide redundancies to ensure that the message is delivered in the case that one (or more) of the watermarks is rendered inoperable.

Although the machine-learned message generation model and machine-learned watermark generation model are discussed as being of the machine-learned message embedding model, in some implementations the machine-learned embedding model can perform both functions of the aforementioned models. As an example, the machine-learned message embedding model can be trained to perform the functions of both the machine-learned message generation model and the machine-learned watermark generation model, allowing the machine-learned message embedding model to generate both a message vector and a watermark representing the message vector.

<FIG> is a flowchart depicting an example method of reconstructing an input image from the second output image in accordance with example embodiments. The method <NUM> can be implemented, for instance, using computing device(s) of <FIG>. <FIG> depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that various steps of any of the methods described herein can be omitted, rearranged, performed simultaneously, expanded, and/or modified in various ways without deviating from the scope of the present disclosure.

At <NUM>, the method includes obtaining an encoded second output image. The encoded second output image can be an encoded (e.g., compressed) representation of the input image containing (e.g., watermarked with) an embedded message representing data lost from the input image due to encoding. The encoded second output image can be encoded using any differentiable encoding scheme (e.g., differentiable JPEG compression, differentiable GIF compression, etc.). The encoded second output image can be obtained in the same or similar fashion as the input image, as discussed in <FIG>. In some implementations, the encoded second output image can be transmitted to a computing device alongside trained machine-learned extraction model. In such fashion, the embedded message can be extracted from the second output image by the receiving computing device using the trained machine-learned extraction model.

At <NUM>, the method includes decoding the encoded version of the second output image to obtain a decoded version of the second output image. The second output image can be decoded as specified by the encoding scheme used to encode the first output image. In implementations where the encoding scheme used was a compression scheme, the decoded second output image will generally utilize less memory than the input image even with inclusion of the embedded message. However, in some implementations, the encoding scheme used is non-compressing.

At <NUM>, the method can include using a machine-learned watermark extraction model of the machine-learned message embedding model to extract the embedded message from the decoded version of the second output image to obtain an extracted embedded message. More particularly, the machine-learned watermark extraction model can take the decoded second output image as an input and, by extracting the watermark from the second output image, obtain an extracted embedded message. The machine-learned watermark extraction model can be or can otherwise include one or more neural networks (e.g., deep neural networks) or the like. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks. In some implementations, the embedded message can be extracted by an autoencoder (e.g., the machine-learned watermark extraction model). The extracted embedded message (e.g., the extracted message vector) can be the latent space vector representation of the difference image.

At <NUM>, the method can include using a machine-learned difference reconstruction model of the machine-learned message embedding model to reconstruct the difference image from the embedded message. More particularly, the machine-learned difference reconstruction model can take the extracted embedded message (e.g., the extracted message vector) as an input and then output the reconstructed difference image. The machine-learned difference reconstruction model can be or can otherwise include one or more neural networks (e.g., deep neural networks) or the like. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, and/or various other types of neural networks. In some implementations, the difference image can be reconstructed by an autoencoder (e.g., the machine-learned difference reconstruction model). The reconstructed difference image, in some circumstances, may have suffered data loss from the encoding and decoding processes. Further, the initial difference image's representation as a latent space vector (e.g., a message vector) can additionally lead to a loss of data. However, such data loss does not necessarily render the reconstructed difference image inoperable.

Although the machine-learned watermark extraction model and machine-learned difference reconstruction model are discussed as being of the machine-learned message extraction model, in some implementations the machine-learned extraction model can perform both functions of the aforementioned models. As an example, the machine-learned message extraction model can be trained to perform the functions of both the machine-learned watermark extraction model and the machine-learned difference reconstruction model, allowing the machine-learned message extraction model to both extract a message vector and reconstruct the difference image from the message vector.

At <NUM>, the method can include generating a reconstruction of the input image based at least in part on the decoded version of the second output image and the reconstruction of the difference image. More particularly, the reconstructed difference image can be added to the decoded second output image in a pixel-by-pixel fashion to reconstruct the input image. For example, the input image can have a first pixel value of <NUM> and the second output image can have a corresponding pixel value of <NUM>. The reconstructed difference image can have a respective pixel value of <NUM>, and adding the reconstructed difference image to the second output image can produce a respective pixel value of <NUM> for a reconstructed input image. In such fashion, the input image can be reconstructed pixel-by-pixel. Although the above example can be used to reconstruct the input image, it should be understood by those skilled in the art that that the reconstructed difference image can be utilized in any manner (e.g., added, subtracted, etc.) with the second output image to reconstruct the input image. It should be noted that in some circumstances, the corresponding pixel value of the reconstructed difference image may not identically match the difference from the initial difference image. As such, the reconstructed difference image can be identical or substantially similar to the initial difference image.

<FIG> is a flowchart depicting an example method for training one or more machine-learned models in accordance with example embodiments. The method <NUM> can be implemented, for instance, using computing device(s) of <FIG>. <FIG> depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that various steps of any of the methods described herein can be omitted, rearranged, performed simultaneously, expanded, and/or modified in various ways without deviating from the scope of the present disclosure.

At <NUM>, the method can include using a machine-learned message extraction model to extract the embedded message from the decoded version of the second output image to obtain a reconstruction of the difference image. In some implementations, a machine-learned watermark extraction model of the machine-learned message extraction model can extract the embedded message and a machine-learned difference reconstruction model of the machine-learned message extraction model can reconstruct the difference image. In some alternative implementations, the machine-learned message extraction model can perform both of the aforementioned functions. The aforementioned machine-learned models (e.g., machine-learned message extraction model, machine-learned extraction model, machine-learned difference reconstruction model, etc.) can be trained either together or simultaneously (e.g., in a joint fashion in which one or more gradients are passed from one network to another). For example, the aforementioned networks can be trained simultaneously as a large machine-learned model ensemble by providing a first training signal including a loss function to the machine learned message extraction model (e.g., the machine-learned watermark extraction and difference reconstruction model).

At <NUM>, the method can include evaluating a loss function that evaluates a difference between the input image and the reconstruction of the input image. In such fashion, the machine-learned model(s) can be trained to increase the quality of the reconstructed image. In some implementations, the machine-learned message extraction model can be trained at least in part by this training signal. In some other implementations, the machine-learned watermark extraction model and the machine-learned difference reconstruction model can be trained at least in part using this training signal.

At <NUM>, the method can include further evaluating the loss function to evaluate a difference between the first output image and the second output image. In such fashion, the machine-learned model(s) can be trained to minimize the perceptual effect of watermarking on the second output image. In some implementations, the machine-learned message extraction model (e.g., the machine-learned watermark extraction model and the machine-learned difference reconstruction model) can be trained at least in part using this training signal. In some other implementations, the machine-learned message generation model (e.g., the machine-learned message generation model and the machine-learned watermark generation model) can be trained at least in part using this training signal.

At <NUM>, the method can include modifying values for one or more parameters of at least the machine-learned message extraction model (e.g., the machine-learned watermark extraction model and the machine-learned difference reconstruction model) based on the loss function. In some implementations, the values for one or more parameters of at least the machine-learned extraction model are based only on the loss function evaluation of the difference between the input image and the reconstructed input image. The difference can be backpropagated through the machine-learned message extraction model (e.g., the machine-learned watermark extraction and difference reconstruction models) to determine values associated with one or more parameters of the model(s) to be updated. The one or more parameters can be updated to reduce the difference evaluated by the loss function (e.g., using an optimization procedure, such as a gradient descent algorithm).

Additionally, in some implementations, the aforementioned loss function can be further backpropagated through the machine-learned message generation model (e.g., the machine-learned message generation model and the machine-learned watermark generation model) to determine values associated with one or more parameters of the model(s) to be updated. The one or more parameters can be updated to reduce the difference evaluated by the loss function (e.g., using a gradient descent algorithm). In such fashion, both the machine-learned message embedding and extraction models (and their respective associated models) can be trained to generate a more accurate reconstructed image. As an example, the machine-learned message generation model can be trained by the loss function to generate a message vector that enables more accurate reconstruction of the difference image.

In some implementations, the method can include modifying values for one or more parameters of the machine-learned message generation model (e.g., the machine-learned message generation model and the machine-learned watermark generation model). More particularly, the model(s) can be trained at least in part using the loss function evaluation of a perceptual difference (e.g., a perceptual loss) between the first output image and the second output image. The difference can be backpropagated through the machine-learned model(s) to determine values associated with one or more parameters of the model(s) to be updated. The one or more parameters can be updated to reduce the difference evaluated by the loss function (e.g., using a gradient descent algorithm). In such fashion, the model(s) can be trained to reduce the perceptual difference associated with adding a watermark to the second output image.

Alternatively, in some implementations, different models of the machine-learned message embedding and extraction models can be trained separately. More particularly, the machine-learned watermark generation and extraction models can be trained simultaneously and separately from the machine-learned message generation and difference reconstruction models. The machine-learned message embedding model and the machine-learned message extraction models can be trained simultaneously using a training signal including a loss function. More particularly, a training signal including a loss function can be backpropagated through the models to train both models simultaneously. As one example, the loss function can evaluate a first difference between the input image and the reconstructed input image. Further, the loss function can, in some implementations, further evaluate a second difference between the fidelity of the first output image and the fidelity of the second output image. These difference(s) can be backpropagated through the machine-learned watermark generation and extraction models to determine values associated with one or more parameters of the model(s) to be updated. The one or more parameters can be updated to reduce the difference evaluated by the loss function (e.g., using a gradient descent algorithm).

Similarly, the machine-learned message generation and machine-learned difference reconstruction models can be trained simultaneously and separately. In some implementations, the embedded message vector is generated and reconstructed using an artificial neural network architecture (e.g., an autoencoder). The neural network architecture can be trained to both generate the embedded message vector for embedding in the output image and reconstruct the difference image from the extracted embedded message (e.g., the message extracted by the machine-learned message extraction model). The machine-learned message generation and difference reconstruction models can be trained simultaneously using a training signal. More particularly, a training signal including a loss function can be backpropagated through the model(s) to train both models simultaneously. In some implementations, the loss function can evaluate a difference between the difference image and the reconstructed difference image. The difference can be backpropagated through the neural network architecture to determine values associated with one or more parameters of the model(s) to be updated. The one or more parameters can be updated to reduce the difference evaluated by the loss function (e.g., using a gradient descent algorithm). Thus, in such fashion, the machine-learned message generation and machine-learned difference reconstruction models can be trained to generate a message representing the difference image and to reconstruct the difference image from the extracted embedded message in a manner that maximizes the quality of the reconstructed difference image.

In some implementations, a discriminator can be utilized to determine the perceptual loss between the first output image and the second output image. The discriminator can be a machine-learned discriminative model (e.g., general adversarial network, linear classifier, support vector machine (SVM), etc.). The discriminator can be trained to determine if the second output image contains an embedded message. As one example, the discriminator can be trained in a supervised fashion using training data including images known to contain embedded messages. The output of the discriminator can, in some implementations, be used as a training signal to the machine-learned watermark generation model and/or the machine-learned message generation model. In such a fashion, the perceptual difference caused by embedding the message in the second output image can be reduced.

Thus, the systems and methods of the present disclosure substantially reduce the data loss associated with encoding images, specifically encoding using lossy compression schemes. As such, the systems and methods of the present disclosure can drastically increase the quality of encoded images, therefore allowing compression to be used in quality-sensitive image storage scenarios in which compression would previously be inoperable.

Claim 1:
A computer-implemented method of encoding an input image (<NUM>) using watermark-based image reconstruction to compensate for lossy encoding schemes, the method comprising:
obtaining, by one or more computing devices, the input image;
generating, by the one or more computing devices, a first output image (<NUM>) by encoding and decoding the input image according to a lossy encoding scheme (<NUM>);
determining, by the one or more computing devices, a difference image (<NUM>) that describes a difference between the input image and the first output image;
generating, by the one or more computing devices and using a machine-learned message embedding model, a second output image (<NUM>) that comprises an embedded message that is based at least in part on the difference image, comprising:
generating, by the one or more computing devices and as an output of a machine-learned watermark generation model of the machine-learned message embedding model, the watermark data based at least in part on the difference image; and
adding, by the one or more computing devices, the watermark data to the input image to obtain the second output image; and
encoding, by the one or more computing devices and using the encoding scheme, the second output image to obtain an encoded second output image.