Pragmatic Image Compression for Human-in-the-Loop Decision-Making

1 Introduction Modern web platforms serve billions of images every day , and typically rely on lossy compression algorithms to store and transmit this data efficiently . Recent work on machine learning methods for lossy image compression [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] improves upon standard methods like JPEG [ 12 ] by training neural networks to minimize the number of bits needed to generate high-fidelity reconstructions . In this paper , we explore the idea of compressing images to even smaller sizes by intentionally allowing reconstructions to deviate drastically from the visual appearance of their originals , and instead optimizing reconstructions for the specific , downstream tasks that the user wants to perform with them , such as video conferencing , online gaming , or remotely operating space robots [ 13 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main observation in this work is that , instead of optimizing the compression model for a taskagnostic perceptual similarity objective function , we can instead optimize the compression model for functional similarity : producing compressed images that , when shown to the user , induce the user to take the same actions that they would have taken had they observed the original , uncompressed images . We call this PragmatIc COmpression ( PICO ) , inspired by prior work on pragmatics [ 14 , 15 , 16 ] that characterizes the meaning of a message through the behavior it induces in a listener . PICO adapts compression to user behavior , enabling the user to perform their individually-desired tasks with compressed images instead of the original images . For example , consider two users with distinct tasks : one flying a quadcopter , and the other driving a ground robot . On a network with an extremely low bitrate , we would like the compressed video feed of the ground robot to preserve ground-level obstacles and terrain while discarding details about power lines and tree canopies , and the quadcopter feed to do the opposite . To this end , we formulate compression as a human-in-the-loop learning problem , in which the compression model is represented as an encoder-decoder neural network that takes the original image as an input and outputs the compressed image . The user sees the compressed image , and takes an action to perform their desired task ( see Figure 2 ) . The main challenge in this work is designing a loss function for the compression model that evaluates the quality of the compressed image in the context of the original image and the user ’ s action . We do not assume knowledge of the user ’ s desired task , so we can not directly evaluate the quality of the compressed image by evaluating the fitness of the user ’ s action upon seeing the compressed image . We also do not assume access to ground-truth action labels for the original images in the streaming setting , so we can not compare the user ’ s action upon seeing the compressed image to some ground-truth action . Instead , we define the loss function through adversarial learning . For example , consider a user browsing an online shopping catalogue , observing photos and clicking on appealing items . To collect positive and negative examples of user behavior , we simply randomize whether a user sees the original or compressed version of an image while they are shopping , and record their actions . We then train a discriminator to predict the likelihood that a user ’ s action was taken in response to the original rather than a compressed image , and train the compression model to maximize this predicted likelihood . Our primary contribution is the PICO algorithm for human-in-the-loop learning of data compression models . We validate PICO through three user studies on Amazon Mechanical Turk , in which we train and evaluate our compression models on data from human participants . In the first study , we asked participants to read handwritten digits and identify the numbers – PICO learned to preserve the number and discard handwriting style ( Figure 3 ) . In the second study , we asked users to browse a car catalogue and select cars based on perceived price – PICO learned to preserve overall shape and sportiness while randomizing paint jobs and backgrounds ( Figure 4 ) . In the third study , we asked participants to verify photos of faces by checking if heads or eyes were covered – PICO learned to preserve hats and eyeglasses while randomizing faces ( Figure 5 ) . In all three studies , PICO obtained up to 2-4x lower bitrates than non-adaptive baseline methods . To show that PICO can be used in sequential decision-making problems , we also ran a user study with 12 participants who played a car racing video game – at a fixed bitrate , PICO learned to preserve bends in the road substantially better than a non-adaptive baseline method , enabling users to drive more safely ( Figure 6 ) . 2 Related Work . Prior work on learned lossy image compression focuses on overcoming various challenges in training neural networks on images [ 17 ] , including amortized variable-rate compression [ 1 , 4 ] , end-to-end training with quantization [ 2 , 5 , 6 ] , optimizing the rate-distortion trade-off [ 7 , 8 ] , optimizing perceptual quality [ 9 , 10 , 11 ] , training hierarchical latent variable models [ 3 ] , and sequential compression of videos [ 18 , 19 ] . While these methods aim to generate visually-pleasing reconstructions that are perceptually similar to their originals , PICO focuses on preserving functional similarity . Hence , PICO can achieve substantially lower bitrates for specific downstream tasks ( e.g. , see Figure 3 ) . Prior work has studied human-in-the-loop learning in related contexts , including reinforcement learning of text summarization policies from user feedback [ 20 ] and automatic data visualization for decision support systems [ 21 ] . In the context of imitation learning , the idea of fitting a model of human behavior using generative adversarial networks [ 22 ] has also been explored [ 23 ] . PICO differs from [ 21 , 23 ] in that it tackles image compression – an entirely different problem from decision support and imitation learning . In contrast to [ 20 ] , which elicits user comparisons between different summaries of the same text , PICO can be used for sequential tasks like video games ( see Section 5.3 ) where the user can not be repeatedly queried with different compressed versions of the same image . 3 Pragmatic Compression . Generative models are typically used for sampling and representation learning , but they can also be used for compression [ 24 , 25 , 26 , 27 ] . For example , variational autoencoders [ 28 ] are trained with a variational information bottleneck [ 29 ] that explicitly constrains the amount of information carried by their latent variables – hence , we can use a trained encoder to compress an image , and a trained decoder to reconstruct it from latent features [ 5 , 30 ] . In contrast to compression methods that train such generative models to maximize the visual fidelity of the reconstruction , we formulate compression as a problem of control , including the downstream behavior of the user in the problem statement . First , the environment generates an image x 2 Rw⇥h⇥c . Given the original image x , the compression system generates a compressed image x̂ 2 Rw⇥h⇥c that can be represented using no more than n bits , where n is a hyperparameter . The user then observes the compressed image x̂ and samples an action a ⇠ ⇡ ( a|x̂ ) from their unknown policy ⇡ . We do not assume access to the user ’ s utility function U ( x , a ) or a specification of their desired task . Our goal is to generate a compressed image x̂ that induces an action a that maximizes the unknown utility U ( x , a ) . We approach this problem by generating a compressed image x̂ that induces the user to take the same action a that they would have taken had they seen the original image x instead . Let f✓ ( x̂|x ) denote a parametric model of our compression function , where ✓ are the model parameters ( e.g. , neural network weights ) . To train f✓ , we need a loss function that evaluates the difference between an original image x and the output of the compression model x̂ ⇠ f✓ ( x̂|x ) . One approach is to use conditional generative adversarial networks [ 31 ] to train a discriminator D ( x̂ , x ) that tries to distinguish between original and compressed images , and train the compression model to generate compressed images x̂ that fool this discriminator , analogous to prior work on adversarial image compression [ 11 ] . However , this approach seeks to maximize the perceptual similarity of the original and compressed image , whereas we would like to maximize their functional similarity . The key challenge for our method then is to train the discriminator D ( x̂ , x ) to detect differences between x and x̂ that influence the user ’ s downstream action , while ignoring superficial differences between the images that do not affect the user ’ s action . We address this challenge by first training an action discriminator D ( a , x ) to predict whether the user saw the original or a compressed image before taking the action a . This action discriminator D captures differences in user behavior caused by compression , while ignoring visual differences between the original and compressed images . To construct a loss function that links the compressed images to these behavioral differences , we distill the action discriminator D ( a , x ) into an image discriminator D ( x̂ , x ) . 3.1 Maximizing Functional Similarity of Images through Adversarial Learning . We formalize the idea of maximizing the functional similarity of the original x and compressed image x̂ as follows . Let T 2 { 0 , 1 } denote whether the user sees the original or a compressed image before taking an action : if T = 1 , then x̂ x ; else if T = 0 , sample x̂ ⇠ f✓ ( x̂|x ) . We would like to train Algorithm 1 Pragmatic Compression ( PICO ) Initialize compression model f✓ while true do x ⇠ penv ( x ) . environment generates original image T ⇠ Bernoulli ( 0.5 ) . randomly decide whether user sees compressed image or original if T = 1 then x̂ x else x̂ ⇠ f✓ ( x̂|x ) a ⇠ ⇡ ( a|x̂ ) . user takes action using unknown policy D D [ { ( T , x , x̂ , a ) } +r P ( T , x , a ) 2D T · logD ( a , x ) + ( 1 T ) · log ( 1 D ( a , x ) ) . update action discrim . r P ( x , x̂ , a ) 2D DKL ( D ( a , x ) k D ( x̂ , x ) ) . update image discriminator ✓ ✓ +r✓ P x2D logD ( f✓ ( x ) , x ) . update compression model the compression model to minimize the divergence of the user ’ s policy evaluated on the compressed image ⇡ ( a|x̂ ) from the policy evaluated on the original ⇡ ( a|x ) , L ( ✓ ) = Ex [ D ( ⇡ ( a|x ) k Ex̂⇠f✓ ( x̂|x ) [ ⇡ ( a|x̂ ) |x ] ) ] = Ex [ D ( p ( a|x , T = 1 ) k p ( a|x , T = 0 ; ✓ ) ) ] , ( 1 ) where D is a divergence ( e.g. , the Jensen-Shannon divergence ) – note that we are overloading D to denote a divergence in Equation 1 , and to denote a discriminator elsewhere . Since the user ’ s policy ⇡ is unknown , we approximately minimize the loss in Equation 1 using conditional generative adversarial networks ( GAN ) [ 31 ] , where the side information is the original image x , the generator is the compression model f✓ ( x̂|x ) , and the discriminator D ( a , x ) tries to discriminate the action a that the user takes after seeing the generated image x̂ . To train the action discriminator , we need positive and negative examples of user behavior ; in our case , examples of user behavior with and without compression . To collect these examples , we randomize whether the user sees the compressed image or the original before taking an action . Let T ⇠ Bernoulli ( 0.5 ) represent this random assignment . When T = 1 , the user sees the original x and takes action a , and we record ( x , x̂ , a ) as a positive example of user behavior . When T = 0 , the user sees the compressed image x̂ and takes action a , and we record ( x , x̂ , a ) as a negative example . Let D denote the dataset of all recorded tuples ( T , x , x̂ , a ) . We train an action discriminator D ( a , x ) to predict the likelihood p ( T = 1|a , x ) , using the standard binary cross-entropy loss and the training data D. Note that this action discriminator is conditioned on the original image x and the user action a , but not the compressed image x̂ – this follows from our problem formulation in Equation 1 , and ensures that the action discriminator captures differences in user behavior caused by compression , while ignoring differences between the original and compressed images that do not affect user behavior .

Summary: This paper propose a deep image compression method that train a compression model through human-in-the-loop. The method train discriminator that tries to distinguish whether a user’s action was taken in response to the compressed image or the original. They evaluate their method through tasks that do not rely solely on perceptual similarity.