Pragmatic Image Compression for Human-in-the-Loop Decision-Making

1 Introduction Modern web platforms serve billions of images every day , and typically rely on lossy compression algorithms to store and transmit this data efficiently . Recent work on machine learning methods for lossy image compression [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] improves upon standard methods like JPEG [ 12 ] by training neural networks to minimize the number of bits needed to generate high-fidelity reconstructions . In this paper , we explore the idea of compressing images to even smaller sizes by intentionally allowing reconstructions to deviate drastically from the visual appearance of their originals , and instead optimizing reconstructions for the specific , downstream tasks that the user wants to perform with them , such as video conferencing , online gaming , or remotely operating space robots [ 13 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main observation in this work is that , instead of optimizing the compression model for a taskagnostic perceptual similarity objective function , we can instead optimize the compression model for functional similarity : producing compressed images that , when shown to the user , induce the user to take the same actions that they would have taken had they observed the original , uncompressed images . We call this PragmatIc COmpression ( PICO ) , inspired by prior work on pragmatics [ 14 , 15 , 16 ] that characterizes the meaning of a message through the behavior it induces in a listener . PICO adapts compression to user behavior , enabling the user to perform their individually-desired tasks with compressed images instead of the original images . For example , consider two users with distinct tasks : one flying a quadcopter , and the other driving a ground robot . On a network with an extremely low bitrate , we would like the compressed video feed of the ground robot to preserve ground-level obstacles and terrain while discarding details about power lines and tree canopies , and the quadcopter feed to do the opposite . To this end , we formulate compression as a human-in-the-loop learning problem , in which the compression model is represented as an encoder-decoder neural network that takes the original image as an input and outputs the compressed image . The user sees the compressed image , and takes an action to perform their desired task ( see Figure 2 ) . The main challenge in this work is designing a loss function for the compression model that evaluates the quality of the compressed image in the context of the original image and the user ’ s action . We do not assume knowledge of the user ’ s desired task , so we can not directly evaluate the quality of the compressed image by evaluating the fitness of the user ’ s action upon seeing the compressed image . We also do not assume access to ground-truth action labels for the original images in the streaming setting , so we can not compare the user ’ s action upon seeing the compressed image to some ground-truth action . Instead , we define the loss function through adversarial learning . For example , consider a user browsing an online shopping catalogue , observing photos and clicking on appealing items . To collect positive and negative examples of user behavior , we simply randomize whether a user sees the original or compressed version of an image while they are shopping , and record their actions . We then train a discriminator to predict the likelihood that a user ’ s action was taken in response to the original rather than a compressed image , and train the compression model to maximize this predicted likelihood . Our primary contribution is the PICO algorithm for human-in-the-loop learning of data compression models . We validate PICO through three user studies on Amazon Mechanical Turk , in which we train and evaluate our compression models on data from human participants . In the first study , we asked participants to read handwritten digits and identify the numbers – PICO learned to preserve the number and discard handwriting style ( Figure 3 ) . In the second study , we asked users to browse a car catalogue and select cars based on perceived price – PICO learned to preserve overall shape and sportiness while randomizing paint jobs and backgrounds ( Figure 4 ) . In the third study , we asked participants to verify photos of faces by checking if heads or eyes were covered – PICO learned to preserve hats and eyeglasses while randomizing faces ( Figure 5 ) . In all three studies , PICO obtained up to 2-4x lower bitrates than non-adaptive baseline methods . To show that PICO can be used in sequential decision-making problems , we also ran a user study with 12 participants who played a car racing video game – at a fixed bitrate , PICO learned to preserve bends in the road substantially better than a non-adaptive baseline method , enabling users to drive more safely ( Figure 6 ) . 2 Related Work . Prior work on learned lossy image compression focuses on overcoming various challenges in training neural networks on images [ 17 ] , including amortized variable-rate compression [ 1 , 4 ] , end-to-end training with quantization [ 2 , 5 , 6 ] , optimizing the rate-distortion trade-off [ 7 , 8 ] , optimizing perceptual quality [ 9 , 10 , 11 ] , training hierarchical latent variable models [ 3 ] , and sequential compression of videos [ 18 , 19 ] . While these methods aim to generate visually-pleasing reconstructions that are perceptually similar to their originals , PICO focuses on preserving functional similarity . Hence , PICO can achieve substantially lower bitrates for specific downstream tasks ( e.g. , see Figure 3 ) . Prior work has studied human-in-the-loop learning in related contexts , including reinforcement learning of text summarization policies from user feedback [ 20 ] and automatic data visualization for decision support systems [ 21 ] . In the context of imitation learning , the idea of fitting a model of human behavior using generative adversarial networks [ 22 ] has also been explored [ 23 ] . PICO differs from [ 21 , 23 ] in that it tackles image compression – an entirely different problem from decision support and imitation learning . In contrast to [ 20 ] , which elicits user comparisons between different summaries of the same text , PICO can be used for sequential tasks like video games ( see Section 5.3 ) where the user can not be repeatedly queried with different compressed versions of the same image . 3 Pragmatic Compression . Generative models are typically used for sampling and representation learning , but they can also be used for compression [ 24 , 25 , 26 , 27 ] . For example , variational autoencoders [ 28 ] are trained with a variational information bottleneck [ 29 ] that explicitly constrains the amount of information carried by their latent variables – hence , we can use a trained encoder to compress an image , and a trained decoder to reconstruct it from latent features [ 5 , 30 ] . In contrast to compression methods that train such generative models to maximize the visual fidelity of the reconstruction , we formulate compression as a problem of control , including the downstream behavior of the user in the problem statement . First , the environment generates an image x 2 Rw⇥h⇥c . Given the original image x , the compression system generates a compressed image x̂ 2 Rw⇥h⇥c that can be represented using no more than n bits , where n is a hyperparameter . The user then observes the compressed image x̂ and samples an action a ⇠ ⇡ ( a|x̂ ) from their unknown policy ⇡ . We do not assume access to the user ’ s utility function U ( x , a ) or a specification of their desired task . Our goal is to generate a compressed image x̂ that induces an action a that maximizes the unknown utility U ( x , a ) . We approach this problem by generating a compressed image x̂ that induces the user to take the same action a that they would have taken had they seen the original image x instead . Let f✓ ( x̂|x ) denote a parametric model of our compression function , where ✓ are the model parameters ( e.g. , neural network weights ) . To train f✓ , we need a loss function that evaluates the difference between an original image x and the output of the compression model x̂ ⇠ f✓ ( x̂|x ) . One approach is to use conditional generative adversarial networks [ 31 ] to train a discriminator D ( x̂ , x ) that tries to distinguish between original and compressed images , and train the compression model to generate compressed images x̂ that fool this discriminator , analogous to prior work on adversarial image compression [ 11 ] . However , this approach seeks to maximize the perceptual similarity of the original and compressed image , whereas we would like to maximize their functional similarity . The key challenge for our method then is to train the discriminator D ( x̂ , x ) to detect differences between x and x̂ that influence the user ’ s downstream action , while ignoring superficial differences between the images that do not affect the user ’ s action . We address this challenge by first training an action discriminator D ( a , x ) to predict whether the user saw the original or a compressed image before taking the action a . This action discriminator D captures differences in user behavior caused by compression , while ignoring visual differences between the original and compressed images . To construct a loss function that links the compressed images to these behavioral differences , we distill the action discriminator D ( a , x ) into an image discriminator D ( x̂ , x ) . 3.1 Maximizing Functional Similarity of Images through Adversarial Learning . We formalize the idea of maximizing the functional similarity of the original x and compressed image x̂ as follows . Let T 2 { 0 , 1 } denote whether the user sees the original or a compressed image before taking an action : if T = 1 , then x̂ x ; else if T = 0 , sample x̂ ⇠ f✓ ( x̂|x ) . We would like to train Algorithm 1 Pragmatic Compression ( PICO ) Initialize compression model f✓ while true do x ⇠ penv ( x ) . environment generates original image T ⇠ Bernoulli ( 0.5 ) . randomly decide whether user sees compressed image or original if T = 1 then x̂ x else x̂ ⇠ f✓ ( x̂|x ) a ⇠ ⇡ ( a|x̂ ) . user takes action using unknown policy D D [ { ( T , x , x̂ , a ) } +r P ( T , x , a ) 2D T · logD ( a , x ) + ( 1 T ) · log ( 1 D ( a , x ) ) . update action discrim . r P ( x , x̂ , a ) 2D DKL ( D ( a , x ) k D ( x̂ , x ) ) . update image discriminator ✓ ✓ +r✓ P x2D logD ( f✓ ( x ) , x ) . update compression model the compression model to minimize the divergence of the user ’ s policy evaluated on the compressed image ⇡ ( a|x̂ ) from the policy evaluated on the original ⇡ ( a|x ) , L ( ✓ ) = Ex [ D ( ⇡ ( a|x ) k Ex̂⇠f✓ ( x̂|x ) [ ⇡ ( a|x̂ ) |x ] ) ] = Ex [ D ( p ( a|x , T = 1 ) k p ( a|x , T = 0 ; ✓ ) ) ] , ( 1 ) where D is a divergence ( e.g. , the Jensen-Shannon divergence ) – note that we are overloading D to denote a divergence in Equation 1 , and to denote a discriminator elsewhere . Since the user ’ s policy ⇡ is unknown , we approximately minimize the loss in Equation 1 using conditional generative adversarial networks ( GAN ) [ 31 ] , where the side information is the original image x , the generator is the compression model f✓ ( x̂|x ) , and the discriminator D ( a , x ) tries to discriminate the action a that the user takes after seeing the generated image x̂ . To train the action discriminator , we need positive and negative examples of user behavior ; in our case , examples of user behavior with and without compression . To collect these examples , we randomize whether the user sees the compressed image or the original before taking an action . Let T ⇠ Bernoulli ( 0.5 ) represent this random assignment . When T = 1 , the user sees the original x and takes action a , and we record ( x , x̂ , a ) as a positive example of user behavior . When T = 0 , the user sees the compressed image x̂ and takes action a , and we record ( x , x̂ , a ) as a negative example . Let D denote the dataset of all recorded tuples ( T , x , x̂ , a ) . We train an action discriminator D ( a , x ) to predict the likelihood p ( T = 1|a , x ) , using the standard binary cross-entropy loss and the training data D. Note that this action discriminator is conditioned on the original image x and the user action a , but not the compressed image x̂ – this follows from our problem formulation in Equation 1 , and ensures that the action discriminator captures differences in user behavior caused by compression , while ignoring differences between the original and compressed images that do not affect user behavior .

Summary: The foundational motivation for this work is the quite reasonable assumption that when a learned image compression codec is to be used as a component in a larger system, it might be desirable to optimize the codec for its functional utility for the downstream task instead of general perceptual quality. Forgoing the need for good all-around perceptual reconstruction can therefore lead to large gains in terms of compression ratio.  The main challenge is to quantify the functional similarity of a compressed image compared to the original image, without making any assumptions about what the users' downstream task is, or how the users select their actions upon seeing an image. To this end, the authors propose an adversarial framework. In a manner similar to A/B testing, they present users with either the original images $x$ or their compressed versions $\hat{x}$, and record what actions $a$ the users took. Then, they train a discriminator $D(a, x)$ to predict if a user's action was taken after seeing the original or the compressed image. This first discriminator is distilled to a second discriminator $D(x, \hat{x})$, which predicts which image was seen based on the prediction of the first discriminator. Now, the second discriminator $D(x, \hat{x})$ can be readily used as a measure of functional similarity between $x$ and $\hat{x}$ for the unknown downstream task of the user.  If this system was to be trained end-to-end, it would require access to a large amount of data harvested from the users. To circumvent this, the authors use pre-trained generative models in their experiments, and optimize the discriminators on quantities derived from the generative models' latent spaces, making the optimization process converge much faster.   The authors perform extensive experiments to verify that 1) their proposed adaptive method performs better than general-purpose, non-adaptive methods, 2) their method adapts to the specific needs of the downstream task and 3) their method can learn in a sequential decision-making setting. They obtain good results for all 3 settings.