Pragmatic Image Compression for Human-in-the-Loop Decision-Making
Authors: Siddharth Reddy
Anca Dragan
Sergey Levine
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: image compression
human-in-the-loop learning
generative adversarial networks
sequential decision-making
pragmatics
generative models

Abstract: Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by the user for downstream tasks -- e.g., deciding which product to click on in a shopping website -- is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's decisions. We approach this problem by training a compression model through human-in-the-loop learning as the user performs tasks with the compressed images. The key insight is to train the model to produce a compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, we train a discriminator that tries to distinguish whether a user's action was taken in response to the compressed image or the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing a car racing video game. The results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compression model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.

Pragmatic Image Compression for Human-in-the-Loop Decision-Making
Siddharth Reddy, Anca D. Dragan, Sergey Levine University of California, Berkeley {sgr, anca, wwwplleinneebeereeeeeeee edu
Abstract
Standard lossy image compression algorithms aim to preserve image's 's appear- ance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by user for downstream tasks e.g., deciding which product to click on shopping website - is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's 's decisions. We approach this problem by training a compression model through human-in-the-le learning as the user performs tasks with the compressed images. The key insight is to train the model to produce compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, train discriminator that tries to distinguish whether a user's 's action was taken in response to the compressed image the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compres- sion model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.
racing video game. The
Introduction
Compressed (for users with four different tecks)
Modern web platforms serve billions of images every day, and typically rely on lossy compression algorithms store and transmit this data efficiently. Recent work on machine learning methods for lossy image compression [| 222 4
Original
(b)
6, 7,888 10 [11 improves upon standard methods like JPEG 12 by training neural networks to minimize the number of bits needed to generate high-fidelity reconstructions. In this paper, we explore the idea of compressing images to even smaller sizes by intentionally allowing reconstructions to deviate drastically from the visual appearance of their originals, and instead optimizing reconstructions for the specific, downstream tasks that the user wants to perform with them, such as video conferencing, online gaming, remotely operating space robots [3]
(d)
Figure I: Images compressed 2-4x smaller than JPEG retain information for tasks like shopping for cars in perceived price range (a), surveying car colors (b), and checking photos for eyeglasses (c) or hats (d).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Distrr nodd docarded
User action
ll (x, x)
Encoder
Decoder
X
Z
a Steer left
Compression loss
Do(a, x) Discriminator:
fo (xx) x)
Original image
Compression model
Compressed image
Was user shouns
original compressed image?
Steer left
Figure 2:
Given the original image x, we would like to generate a compressed image & such that the user's 's action a upon seeing the compressed image is similar to what it would have been had the user seen the original image instead. In a 2D top-down car racing video game, compression model learns that, in order match the user's steering with and without compression, must preserve bends, but can discard road farther ahead.
Our main observation in this work is that, instead of optimizing the compression model for a task- agnostic perceptual similarity objective function, we can instead optimize the compression model for functional similarity: producing compressed images that, when shown to the user, induce the user to take the same actions that they would have taken had they observed the original, uncompressed images. We call this Pragmatlc COmpression (PICO), inspired by prior work on pragmatics [14 IS 16 that characterizes the meaning of a message through the behavior it induces in a listener. PICO adapts tasks with
compression to user behavior, enabling the user to perform their individually-desiee compressed images instead of the original images. For example, consider two users with distinct tasks: one flying a quadcopter, and the other driving a ground robot. On a network with an extremely low bitrate, we would like the compressed video feed of the ground robot to preserve ground-level obstacles and terrain while discarding details about power lines and tree canopies, and the quadcopter feed to do the opposite.
To this end, we formulate compression as a human- in-em-ii-iiii the- learning problem, in which the compression model is represented as an encoder-decoder neural network that takes the original image as an input and outputs the compressed image. The user sees the compressed image, and takes an action perform their desired task (see Figure D). The main challenge in this work is designing a loss function for the compression model that evaluates the quality of the compressed image in the context of the original image and the user's action. We do not assume knowledge of the user's 's desired task, so we cannot directly evaluate the quality of the compressed image by evaluating the fitness of the user's action upon seeing the compressed image. We also do not assume access to ground-truth action labels for the original images in the streaming setting, so we cannot compare the user's s action upon seeing the compressed image to some ground-truth action.
Instead, we define the loss function through adversarial learning. For example, consider a user browsing an online shopping catalogue, observing photos and clicking on appealing items. To collect positive and negative examples of user behavior, we simply randomize whether a user sees the original or compressed version of an image while they are shopping, and record their actions. We then train a discriminator to predict the likelihood that user's 's action was taken in response to the original rather than compressed image, and train the compression model maximize this predicted likelihood.
Our primary contribution is the PICO algorithm for human-in-the-logg learning of data compression models. We validate PICO through three user studies Amazon Mechanical Turk, in which train and evaluate our compression models on data from human participants. In the first study, we asked participants to read handwritten digits and identify the numbers PICO learned to preserve the number and discard handwriting style (Figure 3). In the second study, we asked users to browse a car catalogue and select cars based on perceived price PICO learned preserve overall shape and sportiness while randomizing paint jobs and backgrounds (Figure 4. In the third study, we asked participants to verify photos of faces by checking if heads or eyes were covered PICO learned to preserve hats and eyeglasses while randomizing faces (Figure 5). In all three studies, PICO obtained up to 2-4x lower bitrates than non-adaptive baseline methods. To show that PICO can used in sequential decision-making problems, we also ran user study with 12 participants who played a car racing video game- at a fixed bitrate, PICO learned to preserve bends in the road substantially better than non-adaptive baseline method, enabling users to drive more safely (Figure 5

---

2
Related Work
Prior work on learned lossy image compression focuses on overcoming various challenges training neural networks on images 17 including amortized variable-rate compression [1 4l end-to-end training with quantization
6 optimizing the rate-distortion trade-off αα, optimizing percep- tual quality [2. TO MTI training hierarchical latent variable models [3], and sequential compression
2
of videos T8 19 While these methods aim to generate visually-pleasing reconstructions that are
perceptually similar to their originals, PICO focuses on preserving functional similarity. Hence, PICO can achieve substantially lower bitrates for specific downstream tasks (c.g., see Figure S)
Prior work has studied simaa-in-the-loog learning in related contexts, including reinforcement learning of text summarization policies from user feedback [ and automatic data visualization for decision support systems [21]. In the context of imitation learning, the idea of fitting model of human behavior using generative adversarial networks [22 has also been explored (% PICO differs
from [21 23 in that it tackles image compression an entirely different problem from decision support and imitation learning. In contrast to 200 which elicits user comparisons between different summaries of the same text, PICO can be used for sequential tasks like video games (see Section 5.3 where the user cannot be repeatedly queried with different compressed versions of the same image.
3 Pragmatic Compression
Generative models are typically used for sampling and representation learning, but they can also
For example, variational autoencoders [2]] are trained with a variational information bottleneck [2 that explicitly constrains the amount of information
be used for compression [2 25 26
27
carried by their latent variables - hence, we can use a trained encoder to compress an image, and a trained decoder to reconstruct from latent features [5] 30 In contrast to compression methods that train such generative models to maximize the visual fidelity of the reconstruction, we formulate compression as a problem of control, including the downstream behavior of the user in the problem statement. First, the environment generates an image x E Ruxbxe Given the original image x, the compression system generates a compressed image E Ruxxxx that can be represented using no more than n bits, where is a hyperparameter. The user then observes the compressed image x and samples an action a ~ ra i)) from their unknown policy и. We do not assume access to the user's utility function (xx a) or specification of their desired task. Our goal is to generate a compressed image & that induces an action a that maximizes the unknown utility U (xx,a) a).
We approach this problem by generating a compressed image x that induces the user to take the same action a that they would have taken had they seen the original image x instead. Let fo (Xx xx|x) denote a parametric model of our compression function, where 0 are the model parameters (e.g.. neural network weights). To train f8. we need loss function that evaluates the difference between an original image x and the output of the compression model & ~ fx(x)x). One approach is to use conditional generative adversarial networks
K1 to train a discriminator D(x, x) that tries to distinguish between original and compressed images, and train the compression model to generate compressed images & that fool this discriminator, analogous to prior work on adversarial image compression [Α]] However, this approach seeks to maximize the perceptual similarity of the original and compressed image, whereas we would like to maximize their functional similarity.
The key challenge for our method then is to train the discriminator D( x) to detect differences between x and x that influence the user's downstream action, while ignoring superficial differences between the images that do not affect the user's action. We address this challenge by first training an action discriminator (a x) to predict whether the user saw the original or compressed image before taking the action a. This action discriminator Da captures di differences in user behavior caused by compression, while ignoring visual differences between the original and compressed images. To construct a loss function that links the compressed images to these behavioral differences, we distill the action discriminator D2 (a, x) into an image discriminator Do (x, x).
3.1 Maximizing Functional Similarity of Images through Adversarial I Learning
We formalize the idea of maximizing the functional similarity of the original x and compressed image as follows. Let T E (0,1) 1) denote whether the user sees the original or compressed image before taking an action: if T = 1, then x <- else if T = 0, sample x ~ fo (x)x). We would like to train
3

---

Algorithm Pragmatic Compression (PICO)
Initialize compression model fo
while true do
X Panv (x)
environment generates original image > randomly decide whether user sees compressed image or original T = then & + x else x ~ (x
T ~ Bernoull(
a (a|x)
user takes action using unknown policy
U (T, x, (,,)
á f- Φ + Vol L (Txx))ED T log (a, (1 (1-11)7 T) log (1 Da(a, x)) update action discrim.
& - + V ax DEL (D,((a((,x)) (x,x)))
& update image discriminator update compression model
0 - 0 Vo EXED log D& (fo(xx),
the compression model minimize the divergence of the user's policy evaluated on the compressed image T[a(8) from the policy evaluated on the original π ralxx)
()) = Ex (P(T(a)))) fo (T (aax))(x)) Ex [Dp((xxx 1) p(a)x, T 0)),
(1)
where D is divergence (e.g., the Jensen-Shannon divergence) note that are overloading D to denote a divergence in Equation ! and to denote a discriminator elsewhere. Since the user's policy π is unknown, we approximately minimize the loss in Equation [ Jusing conditional generative adversarial networks (GAN) GI. where the side information is the original image x, the generator is the compression model fx(x,,, and the discriminator D(a, x) tries to discriminate the action a that the user takes after seeing the generated image &
To train the action discriminator, we need positive and negative examples of user behavior; in our case, examples of user behavior with and without compression. To collect these examples, we randomize whether the user sees the compressed image the original before taking an action. Let ~ Bernoulli(055) represent this random assignment. When T 1, the user sees the original x and takes action a, and we record (x, X, a) positive example of user behavior. When 7 = O, the user sees the compressed image x and takes action a, and we record (x, X, a) as a negative example. Let D denote the dataset of all recorded tuples xT, X, a). We train action discriminator (a x) to predict the likelihood p(T = 1 Ja, x), using the standard binary cross- cross-entropy loss and the training data D. Note that this action discriminator is conditioned the original image and the user action a, but not the compressed image & - this follows from our problem formulation in Equation and ensures that the action discriminator captures differences in user behavior caused by compression, while ignoring differences between the original and compressed images that do not affect user behavior.
3.2 Distilling the Discrininator and Training the Compression Model
The action discriminator (a x) gives us a way to approximately evaluate the loss function in Equation| However, we cannot train the compression model fo (x(x) to optimize this loss directly, since D& does not take the compressed image & as input. To address this issue, we distill the trained action discriminator (a(a, x), which captures differences in behavior caused by compression, into an image discriminator ($, x) that links the compressed images to these behavioral differences. In partieular, we train D& approximate D& by optimizing the loss,
{(bb=
Σ (x.D
DKI (D,(ax)) (X))).
Then, given the trained image discriminator De, we train the compression model using the standard GAN generator loss 22 311.
{()) = Σ - log (fo(x)),,
(3)
XED
where fo(x) denotes Ex~fo(& x) [x|x]. Our complete pragmatic compression method is summarized in Algorithm [ We randomly initialize the compression model fo. The environment samples an original image from an unknown distribution Penv. To decide whether the user sees the original or

---

compressed image, we sample a Bernoulli random variable T. After seeing the chosen image, the user samples an action a from their unknown policy π. To update the action discriminator Dz. we take gradient step on the binary cross-entropy loss. To update the image discriminator Dz, we take a gradient step on the KI--divergence loss in Equation 2 take gradient step on the GAN generator loss in Equation S
To update the compression model fo. we See Appendix AA.3 for details.
Structured Compression using Generative Models
One approach to representing the compression model fe could be structure it a variational autoencoder (VAE) 28.. and train the VAE end to end on the adversarial loss function in Equation g instead of the standard reconstruction error loss. This approach fully general, but requires training separate model for each desired bitrate (which is determined by the 3 coefficient in the VAE training objective), and can require extensive exploration of the pixel output space before discovers effective compression model. To simplify variable-rate compression and exploration in our experiments, we forgo end-to-end training, and first train a generative model on a batch of images without the human in the loop by optimizing task-agnostic perceptual loss, yielding encoder and decoder such that z = enc(x) and x = doc(z). where z E IR is the latent embedding. Analogous to prior work conditional image generation [22 we then train compression model fo (2/z) to compress the latent embedding, instead of compressing the original pixels. We use a variety of different generative models in our experiments, including a 3-VAE
33 for the handwritten digit identification experiments in Figure 3. a StyleGAN2 model 34 for the car shopping and survey experiments in Figure 2 NVAE model [35] for the photo verification experiments in Figure and
a VAF for the car racing experiments in Figure 6 See Appendix A.4 for details.
Generative models like the VAE and StyleGAN2 tend to learn disentangled features - hence, instead of training fe to map directly to the latent space Rd, we structure fe to output a vector of probabilities that determines which latent features are transmitted exactly between and 2, and which other features are masked out and then reconstructed from the prior distribution. In particular, we structure fo Rdd - [0, 1]dd to output vector of mask probabilities p [0, 1] given the latent embedding 2 E TRdd Then, given hyperparameter À E [0, 1] that controls the compression rate, we transmit the [dd] latent features with the lowest mask probabilities p;, and mask out the remaining d - [[dd features. We reconstruct the masked features by assuming that 2 follows multivariate normal distribution, and sampling the masked feature values from the conditional prior distribution given the transmitted feature values. See Appendix A.4 for details.
This design of the compression model fo simplifies variable-rate compression: at test time, we simply choose a value of À that obtains the desired bitrate, without retraining the model. It also simplifies exploration: instead of exploring pixel output space, we explore in the space of masks over latent features, which leverages the decoder to generate more realistic compressed images during the early stages of training. We can now also reduce the dimensionality inputs: instead of training De (x, x). we train (p, x). In our experiments, we also leverage the low-dimensional mask output space to perform batch learning instead of online learning, which greatly simplifies our implementation of PICO with real users. See Appendix A. for additional discussion.
the image discriminator
While these simplifications enable us to provide proof of concept for pragmatic compression in this paper, we acknowledge that they do require both server and client to have copy of a domain-specitn (but task-agnostic) generative model. End-to-end training of the compression model would be more general approach that does not involve learning and storing separate generative model this is a promising direction for future work, which we discuss in Section
User Studies
In our experiments, we evaluate to what extent PICO can minimize the number of bits needed to transmit an image, while still preserving the image's usefulness to users performing downstream tasks. We conduct user studies on Amazon Mechanical Turk, which we ask human participants to complete three tasks at varying bitrates: reading handwritten digits from the MNIST dataset [36]. verifying attributes of faces from the CelebA dataset 37 and browsing shopping catalogue of cars from the LSUN Car dataset
3
To study PICO's performance sequential decision-
5

---

problems, we also run an experiment with 12 participants who play the Car Racing video game from OpenAl Gym (9 under constraint on the bitrate of the video feed. In all experiments, we train our discriminators and compression model on 1000 negative examples and varying numbers of positive examples, and split PICO into two rounds of batch learning and evaluation (see Appendices A. and
A.5
Appendix A discusses the implementation details.
5.1
Minimizing Bitrate by Maximizing User Action Agreement
We claim that PICO can learn to transmit only the features that users need to perform their tasks. Our first set of user studies seeks to answer Q1: does maximizing user action agreement enable PICO to obtain lower bitrates than baseline methods that do not take into account downstream user behavior? We would like to study this question in domains where we can measure the performance of various compression methods by computing the agreement between the user's 's actions with and without compression- - i.e., collecting action labels for the original images, and comparing the user's actions upon seeing compressed versions of those images to the labels. As such, we run experiments on Amazon Mechanical Turk that focus on single-step decision-maie making settings where we can collect action labels for fixed dataset of images: (a) identifying handwritten digit, (b) clicking
item in a shopping catalogue, and (c) verifying photos of faces. In (a), we instruct users to identify the number in the image within the range 0-9. In (b), to simulate the experience of browsing a catalogue on a budget, we instructed users to click on images of cars that they perceive to be worth less than $20,000. In (c), we instruct users to check if the person's 's eyes are covered (e.g.. by eyeglasses) and click on one of two buttons labeled "covered" and "not covered".
In all domains, we evaluate PICO by varying the bitrate and, at each bitrate, measuring the agreement of user actions upon seeing compressed image with user actions upon seeing the original version of that image (see Appendix A.7 for details). As discussed in Section 4 PICO learns a compression model Je that, given a separate generative model, selects which latent features to transmit. Since the purpose of this experiment is to test the effect of user-adaptive compression in PICO, we compare to non-adaptive baseline method that selects uniform-random random subset of features to transmit, but otherwise uses the same generative model as PICO this enables us to conduct an apples-to-apples comparison that isolates the effect of training fo on user behavior data. We also compare to a baseline method that maximizes perceptual similarity by replacing the adversarial loss in Equation of PICO with the mean absolute pixel difference |x - x|. In simulation experiments, we found that this perceptual similarity baseline performed better than the non-adaptive baseline in the MNIST domain, but did not perform better in the other domains (see Appendix O so we only test it in the MNIST user study. To provide point of comparison to widely-used compression methods, also compare to JPEG (12). where the quality parameter is set to the lowest value (1) in order to bring the bitrate close as possible to the range obtained by PICO and the non-adaptive baseline. Though JPEG is no longer the state of the art, it enables us
Transcrihe handwaileen digils
Original
PICO
Poreppiual Similarity
MNIST Digits
6
8 8
6 6
6
to roughly calibrate the results
1
1
/
/
/
/
achieved by PICO as well as
User's Agtion Agroamennt
0.8
2 2 2
2
2
2
2
6
2
2
2
the non-adaptive and perceptual similarity baselines.
355 8
3
8
8
3
8
6
3
4:1 4 / 4
7
4
4
4
7
Figures 3 and 5) show that, at low bitrates, PICO achieves substantially higher user ac- tion agreement than the non- adaptive baseline (orange vs. gray) and perceptual similar- ity bascline (orange vs. red). PICO also obtains much lower bitrates than the JPEG baseline (orange vs. teal), while main- taining higher agreement on CelebA, comparable agreement on MNIST, and lower agree- ment on LSU Car. The samples in Figure 3 show that PICO learns to preserve digit numbers more often than the adn-aive and perceptual similarity baselines, while randomizing handwriting
0.4-
5
8 8 5
5
8
5
3
8
8
PCO (uut))
6.6
6
6 7
6
8
6
6
3
Non-kdaptive (laonina)
0.2
Decceptual Simiellly (Dase ins)
7.77
7
7
7
4
9
9
9
pst (Sascino
8 8
8
5
8
8
3
8
8
8
0.000 0.005 0.010 0.015 0.020
Birraa (Fraction of Lassless PNG Sloe)
9:9 9
9
9
9
7
9
7
9
Figure 3: MNIST digit identification experiments that address Q1. When
users are instructed to identify the digit number, PICO learns to preserve the digit number while randomizing handwriting style. The plots show user action agreement evaluated on 100 held-out images, with error bars representing standard error. The average lossless PNG file size is 0.3kB, and each image has dimensions 28x28x1. Each of the five columns in the two groups of compressed images represents different sample from the stochastic compression model ff (x)x) at bitrate 0.01

---

Shop for within budget
Original
PICO (Shopping)
PICO (Survey)
LSUN Cars Shopping
User's Acion Agreementt
0.7-
0.8-
HCO (Ouns) Ain-Aclative (Bassina)
0.5
0.000
0.005
0,010
0.015
O. 0.020
Bitrate (Fraction Lossless PNG Size)
Look cats particular color
LSUN Cars Survey
Agreementt
0.8
PEO JOurS)
Non-kktagtioe (Baselne) PCG (Suas
0.5-
0.000
0.005
0.010
0.015 0.020
Bitrate (Fraction Lousless PNG Size)
Figure 4: LSUN Car shopping experiment that addresses Q1, and survey experiment that addresses Q2. The plots show action agrecment evaluated on 100 held-out images, with error bars representing standard error. The average lossless PNG file size is 247kB, and each image has dimensions 512x512x3. The shopping samples show that, when users are instructed to click on cars they perceive to be worth less than $20,000, PICO learns preserve the overall shape and sportiness of the car, while randomizing paint jobs, backgrounds, and other details that are irrelevant to the users' perception of price. contrast, when users are instead instructed to determine whether the car is "dlok-colored" or "light-colored" for a survey task, PICO learns to preserve the car's 's color while randomizing its pose. We intentionally show compressed image samples for a low bitrate (0.011) 1) highlight differences between the compression models learned for the two tasks.
style in order to satisfy the bitrate constraint. The samples in Figure 4 show that, for users performing the shopping task, PIĆO learns to preserve the overall shape and sportiness of the car, while randomiz- ing paint jobs, backgrounds, and other details that are irrelevant to the user's 's perception of the price of the car. The samples in Figure 5 show that, for users checking whether eyes are covered, PICO learns to preserve the presence of eyeglasses while randomizing hair color, faces, and other irrelevant details (see top row of samples). The dip in the orange curve in the car shopping plot may be due to the fact that increasing the bitrate preserves more of the encoded latent features, which, when combined with features sampled from the prior, can be auttof-distribution inputs to the StyleGAN2 decoder 40 41] potentially leading to degraded image quality (see Appendix A.4 for details). Figures and 10 in the appendix include more examples.
5.2 Adapting Compression to Different Downstream Tasks
The experiments in the previous section show that PICO can outperform a non-adaptive baseline method by transmitting only the features that users need to perform their tasks. Our second set of user studies investigates this mechanism further, by asking Q2: can PICO adapt the compression model to the specific needs of different downstream tasks in the same domain? To answer this question, we run an additional experiment in the CelebA domain from the previous section, in which users are instructed to check if the person's 's head is covered (c.g., by a hat). We also run an additional experiment in the LSUN Car domain from the previous section, in which we simulate survey task that asks users to 'help car dealership conduct market research' by determining whether an observed has a "dark-colored" or "light-colored" paint job.
Figure 5 shows that PICO adapted the compression model to the user's particular task. In the experiment from the previous section, when users checked eyes, PICO learned to preserve the presence of eyeglasses while randomizing hair color, faces, and other irrelevant details (see top row of samples). On the other hand, when users checked for head coverings like hats and helmets, PICO
7

---

Original
PICO
decressing
Photo veriication:
check eyes covered
venfication:
chock head covered.
CeleiA Faces Eycglasses
Colib Faces
User 44tton
110 Soodind
0.04
Eirase ifraction Loethos PNG Stre)
Strate Stration Leeelis PNG Fbxt
Samples floed bitrale
Figure 5: CelebA photo attribute verification experiments that address Q1 and Q2. Depending on the instructions given to the user, PICO learns to either preserve hats or eyeglasses, while randomizing faces and other task- irrelevant details. The plots show action agreement evaluated on 100 held-out images, with error bars representing standard error. The average lossless PNG file size is 7.7kB, and each image has dimensions 64x64x3.
learned to preserve the presence of hats while randomizing eyes and other details (see second row of samples). The third and fourth rows of samples illustrate the fact that PICO learns a stochastic compression model fo(x(x) from which we can draw multiple compressed samples for a given original x. The fact that all the samples in the third row have eyeglasses but differ in other attributes like pose angle, and those in the fourth row all have hats while some are smiling and some are not, shows that even though the compression model is stochastic, it produces stable attributes when they are needed for the downstream task. Figure D in the appendix includes more qualitative examples. In addition to these photo verification results, the samples in Figure illustrate substantial differences in the compression models learned for the car shopping and survey tasks. For users performing the shopping task, PICO learned to preserve perceived price while randomizing color. In contrast, for users performing the survey, PICO learned to preserve color while randomizing perceived price.
5.3 Compressing Observations for Sequential Decision-Making
Our third user study seeks to answer Q3: can PICO learn to compress image observations in the sequential decision-making setting? To answer this question, we run an experiment with 12 participants in which we ask users play 2D top-down car racing video game, while constraining the number of bits that can be used to transmit the image observation to the user at each timestep. We would like to measure the performance of PICO and the non-adaptive baseline by computing user action agreement, as in the previous sections. However, since images rarely re-occur in this video game, it is unlikely that we will have an action label for the exact pixels in any given observation. Instead, we measure the user's progress along the road in the game specifically, the fraction of new road patches visited during an episode. these experiments, we fix the bitrate to 85 bits per step, which is well below the 170 bits per step required to transmit the full set of features for the 64x64x3 images. To simplify our experiments and ensure that they could be completed within the allotted 30 minutes per participant, we trained the PICO compression model on data from a pilot user, then evaluated the compression model's performance with each of the 12 participants. Appendix A.. describes the experimental setup in further detail.
Figure 6 shows that, at fixed bitrate, PICO enables the user to perform substantially better on the driving task than the non-adaptive compression baseline (orange vs. gray), and comparably to a positive control in which we do not compress the image observations all (orange vs. teal). The first and second film strips show that, when we use the non-adaptive compression baseline, there is a substantial difference between the originals and the compressed images. For example, even at the first timestep, the compressed image shows the road to be less tilted than it actually is, so in the next frame we see that the user has mistakenly driven forward and ended up in the grass instead of turning right to stay on the road. In contrast, the third and fourth film strips show that PICO has learned to preserve the angle of the road, while discarding the details of the road much farther ahead order to satisfy the bitrate constraint. We ran a one-way repeated measures ANOVA on the road progress
8

---

Car Racing
Car Racing
0.350
0.300
0.2-
Drive the
on the road
0.250
0.3
0.225
0.25
0.30
0.35
No Comoncssion
PICO
Nen Adsetie (Sasaline)
Read Adturtie
Standardi Methodi
Original
Non-Adaptive (Baseline)
Original
PICO (Ours)
Figure 6: Car Racing game experiments that address Q3. The scatter plot shows that, each the users (orange), road progress with PICO was substantially higher than with the non-adaptive compression baseline. The bar chart shows road progress averaged over all users, with error bars representing standard error.
metrics from the non-adaptive baseline and PICO conditions with the presence PICO as a factor, and found that f(1, 11) 176.32, < .0001. The subjective evaluations in Table D in the appendix corroborate these results: users reported feeling higher situational awareness and ability to control the with PICO compared the non-adaptive baseline. After evaluating PICO, one "This environment was a lot easier. It felt more consistent. felt like we had a mutual understanding of when I would and what it would show in more detail, and videos are available on the project website
commented,
make me turn. Appendix B discusses the results
6 Discussion
We presented a proof of concept that, through human-in-the-loww learning, we can train models to communicate relevant information to users under network bandwidth constraints, without prior knowledge of the users' desired tasks. Our experiments show that, for variety of tasks with different kinds of images, pragmatic compression can reduce bitrates 2-4x compared to non-adaptive and perceptual similarity baseline methods, by optimizing reconstructions for functional similarity. Since we needed to carry out user studies with real human participants, we decided to limit the number of parameters trained during these experiments for the sake of efficiency, by using pre- traitreine generative model as a starting point and only optimizing over the latent space this model. This can be problematic when the generative model does not include task-relevant features in its latent space e.g., the yellow sports car in rows 7-8 of Figure 10 in the appendix gets distorted when encoded into the StyleGAN2 latent space, even without any additional compression. An end-to-end version of PICO should in principle also be possible, but would likely require longer numan-in-the-lope training sessions. This may, however, be practical for real-world web services and other applications, where users already continually interact with the system and A/B testing is standard practice. End-to-end training could also enable PICO be applied problems other than compression, such image captioning for risually-impaire users, or audio visualization for hearing impaired users 42 such applications could also be enabled through continued improvements generative models for video 43 44]. audio 45). and text 4o 47 Another exciting area for future work is to apply pragmatic compression wider range of realistic applications, including video compression for robotic space

---

exploration 13 audio compression for hearing aids 48 49 and spatial compression for virtual reality 500
7
Acknowledgements
Thanks members of the InterACT and RAIL labs at UC Berkeley for feedback on this project. This work was supported in part by AFOSR 1-9550-17-1-0000 NSF NRI 1734633, GPU donations from NVIDIA, and the Berkeley Existential Risk Initiative.
References
[1] George Toderici, Sean M O'Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar, Variable rate image compression with recurrent neural networks. arXiv preprint arXiv: /51 11551...00.5.. 2015.
[2] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compres- sion. arXiv preprint 161iv:1611. 1.017444 2016.
[3] Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. arXiv preprint arXiv:1604.08777 2016.
[4] George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[5] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autocncoders. arXiv preprint (rXXiv1703.00395 2017.
[6] Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Lue Van Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. arXiv preprint arXiv: 001.1704444044 2017.
[7] Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Conditional probability models for deep image compression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[8] Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao, and David Zhang. Learning convolutional networks for weighttd image compression. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[9] Michael Tschannen, Eirikur Agustsson, and Mario Lucic. Deep generative models for lossy compression. In Neural Information Processing Systems, 2018.
[10] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[11] Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. Neural Information Processing Systems, 2020.
[12] Gregory K Wallace. The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics, 1992.
[13] Terrence Fong, Jennifer Rochlis Zumbado, Nancy Currie, Andrew Mishkin, and David I. Akin. Space telerobotics: unique challenges human-robot collaboration in space. Reviews of Human Factors and Ergonomics, 2013.
[14] Herbert P Grice. Logic and conversation. In Speech acts. 1975.
[15] Dan Sperber and Deirdre Wilson. Relevance: Communication and cognition. 1986.
10

---

[16] Michael Frank, Noah Goodman, Peter Lai. and Joshua Tenenbaum. Informative communication word production and word learning. In Proceedings of the anmual meeting of the cognitive science society, 2009.
[17] J Jiang. Image compression with neural networks a survey. Signal Processing: Image Communication, 1999.
[18] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. DVC: An end-to-end deep video compression framework. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2019.
[19] Salvator Lombardo, Jun Han, Christopher Schroers, and Stephan Mandt. Deep generative video compression. In Neural Information Processing Systems, 2019.
[20] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint 2tth:2000901325.. 2020.
[21] Sophie Hilgard, Nir Rosenfeld, Mahzarin R Banaji, Jack Cao, and David C Parkes. Learning representations by humans, for humans. arXiv preprint
2019.
[22] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014.
[23] Jonathan Ho and Stefano Ermon, Generative adversarial imitation learning. In Neural Informa- fion Processing Systems, 2016.
[24] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 1948.
[25] Brendan Frey and Geoffrey E Hinton. Efficient stochastic source coding and an application to a Bayesian network source model. Computer Journal, 1997.
[26] Guy E Blelloch. Introduction to data compression. 2001.
[27] David JC MacKay. Information theory, inference and learning algorithms. 2003.
[28] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[29] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00411 2016.
[30] James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. arXiv preprint arXiv:1901.04866 2019.
[31] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:141..1784, 2014.
[32] Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. arXiv preprint arXiv:1711.05772 2017.
[33] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement variational autoencoders. Neural Information Processing Systems, 2018.
[34] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[35] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898 2020.
[36] Yann LeCun. The MNIST database of handwritten digits, 1998.

---

[37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), December 2015.
[38] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.033: 550.03335, 2015.
[39] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv:1666.00 60.660000050, 2016.
[40] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: StyleGAN encoder for image-to-image translation. arXiv preprint arXiv:2008.00955/ 2020.
[41] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for StyleGAN image manipulation. arXiv preprint arXiv:2 2021.
[42] Yuchi Zhang, Willis Peng, Bastian Wandt, and Helge Rhodin. Audio Learning to visualize sound. arXiv preprint arXiv:2012.13341, 2020.
[43] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with VQVAE. arXiv preprint arXiv:2103. 07950, 2021.
[44] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.10157, 2021.
[45] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv: 034999, 2016.
[46] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[47] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[48] Alex Armstrong. Chi Chung Lam, Shievanie Sabesan, and Nicholas A Lesica. The hearing aid dilemma: amplification, compression, and distortion of the neural code. bioRxiv, 2020.
[49] Nasim Alamdari, Edward Lobarinas, and Nasser Kehtarnavaz. Personalization of hearing aid compression by numan-in-the-loop deep reinforcement learning. IEEE Access, 2020.
[50] Niels Christian Nilsson, Tabitha Peck, Gerd Bruder, Eri Hodgson, Stefania Serafin, Mary Whitton, Frank Steinicke, and Evan Suma Rosenberg. 15 years of research redirected walking in immersive virtual environments. IEEE Computer Graphics and Applications, 2018.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[52] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution, arXiv preprint arXiv:1809.019901 2018
12


Reviews:
[{"review_id": "ttWq0LM5zCU", "reviewer": "Reviewer_Maz5", "summary": "The foundational motivation for this work is the quite reasonable assumption that when a learned image compression codec is to be used as a component in a larger system, it might be desirable to optimize the codec for its functional utility for the downstream task instead of general perceptual quality. Forgoing the need for good all-around perceptual reconstruction can therefore lead to large gains in terms of compression ratio.\n\nThe main challenge is to quantify the functional similarity of a compressed image compared to the original image, without making any assumptions about what the users' downstream task is, or how the users select their actions upon seeing an image. To this end, the authors propose an adversarial framework. In a manner similar to A/B testing, they present users with either the original images $x$ or their compressed versions $\\hat{x}$, and record what actions $a$ the users took. Then, they train a discriminator $D(a, x)$ to predict if a user's action was taken after seeing the original or the compressed image. This first discriminator is distilled to a second discriminator $D(x, \\hat{x})$, which predicts which image was seen based on the prediction of the first discriminator. Now, the second discriminator $D(x, \\hat{x})$ can be readily used as a measure of functional similarity between $x$ and $\\hat{x}$ for the unknown downstream task of the user.\n\nIf this system was to be trained end-to-end, it would require access to a large amount of data harvested from the users. To circumvent this, the authors use pre-trained generative models in their experiments, and optimize the discriminators on quantities derived from the generative models' latent spaces, making the optimization process converge much faster. \n\nThe authors perform extensive experiments to verify that 1) their proposed adaptive method performs better than general-purpose, non-adaptive methods, 2) their method adapts to the specific needs of the downstream task and 3) their method can learn in a sequential decision-making setting. They obtain good results for all 3 settings.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThe work is well-motivated, the problem and the framework are interesting, and the utility of a solution is clear. The use of the adversarial framework to obtain a proxy metric for the functional similarity between the original and compressed images is a simple and elegant solution.\n\nThe reduction of the problem to the latent space of generative models is also elegant and greatly extends the applicability of the proposed method. \n\nThe experiments are well-motivated, well-set-up and well-executed. The illustration and analysis of which features PICO retains or allows to vary is very convincing.\n\nThe paper is well-written with good structure. Very nice submission overall.\n\n# Weaknesses\nI am slightly confused about the \"bitrate-user action agreement\" graphs. Concretely, I don't understand how the methods can start at 0 bitrate. If the bitrate is actually a positive number close to 0, then this should be cleared up as the graphs are confusing/ misleading in their current state.\n\n# Questions\n- Have the authors attempted end-to-end training their method on toy tasks instead of reducing it to the latent space of a generative model? It seems to me that one reason the proposed method works so well is because the general-purpose compression codec provided by the generative model is already quite good.\n- In particular, one reason for such a clear behaviour of the system to keep relevant features and randomize irrelevant ones is due to the fact that the authors used pre-trained generative models that are known to be biassed towards learning independent latent features. Do the authors think we would see a similar disentanglement if they trained the system end-to-end?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-6"}, "extra_text": {}}, {"review_id": "S27tEia8gVn", "reviewer": "Reviewer_99WX", "summary": "The authors claim to account not just for optimizing a lossy image compression algorithm to preserve an image's appearance but additionally to minimizes bits while maintaining information necessary for a user to complete downstream tasks correctly.  They achieve this by training a discriminator with human in the loop training and demonstrate the effectiveness of their methodology.  They claim that they show that the method learns to match the user's actions with and without compression at lower bitrates than baseline methods and adapts the compression to user behavior.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a clever paper that uses a novel combination of methodologies to achieve its results. These authors are not the first to train a compression algorithm for downstream tasks as an objective, but as far as I can tell, they are the first to have those downstream tasks completed by real users in the training loop, such that the compression algorithm is learning to optimize for real users and not some modeled approximation to them, or to some CV algorithm.   They call this PICO (pragmatic compression).  \n\nThe results themselves (though meeting the authors claims) don't suggest to me that they are extremely useful as an application as of yet (ie you wouldn't surface these images via a web tool for anyone to actually complete the task if you were relying on that task being done perfectly).  However, the key application (not fleshed out by the authors) seems to me to be in the cognitive sciences, as this serves as a powerful engine for pulling out of images the key elements that people are relying on to do the downstream tasks.\n\nOverall I think this is a novel and cleve contribution that will be interesting to many people at NeurIPS.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Could be used to illicit what parts of images or other inputs human users are utilizing to make their decisions without the user realizing that is happening, not just in toy tasks but in other everyday interactions with things like phones and other devices. Though the method would have to progress quite significantly before that would become a real worry. The authors did not discuss this though they may not have realized the impact for implicit information extraction that is available via a method like this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 "}, "extra_text": {}}, {"review_id": "MOMRO8kgqOU", "reviewer": "Reviewer_cV9i", "summary": "This paper propose a deep image compression method that train a compression model through human-in-the-loop.\nThe method train discriminator that tries to distinguish whether a user’s action was taken in response to the compressed image or the original. They evaluate their method through tasks that do not rely solely on perceptual similarity.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is interesting challenge to the limit of bit rate by taking advantage of the characteristics of deep image compression. \nUsing action discriminator and distilled image discriminator for training the image compression model may be a new approach.\nAlso, I think it would be useful for readers to show how much learning time is required for human-in-the-loop learning compared to baseline.\nOne minor concern is that the bitrate is expressed in comparison to PNG and not in absolute bitrate (e.g. bit-per-pixel), which makes it difficult to get a sense of the bitrate level.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I thought it would be good to discuss whether having a person in the learning loop could potentially introduce bias into the compression results.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "4JUO43Yr3ZJ", "reviewer": "Reviewer_SBfJ", "summary": "The paper proposes a downstream task specific compression method by training a compression model through human-in-the-loop leaning, which adaptively maintain core information based on whether human can make the same decisions when they see compressed or original images. The user study show the proposed method outperforms JPEG codec.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Novel but not good. The human-in-the-loop learning is firstly used in compression to the best of my knowledge. The proposed method directly evaluates the quality of images based on the consistency of human decisions seeing compressed/original images. And the distillation design, using another discriminator which judge images with action to train the wanted discriminator which judge images directly, is brilliant. \nHowever, this design hugely relies on participants, which limits its generalization and evaluation. In fact, it replaces a lot of concurrent perceptual loss functions with “human-in-the-loop” in my opinion.\n\nQuality: Bad. The drawback of this work is intuitive, that is, it requires a lot of users to infer which actions to take, and it is time-consuming because with the model trained, pictures provided to users change as well. The paper discusses this in Sec. 6 but I do not totally agree that using web services is a good solution, which means the model will be hugely affected by users during a period, which can be biased in my opinion. \nOn the other hand, whether the “human” in the loop can be replaced with a neural network, which just requires ground truth label and it can be collected at a time, is not discussed. The excellent performance of neural network to simulate certain functions should be considered, and I think adding some module like attention map will improve the performance.\nFurthermore, there is a lot of advanced compression methods besides relatively old JPEG. More experiments for comparison should be provided. And user study data of JPEG is not enough at all. I want to see its performance with low bit-rate.\nTo summarize, I doubt that the proposed hugely participant-oriented method is a good solution to so-called Pragmatic Image Compression. And if we just consider the performance of some specific task, why not directly training a network to tell users the label they wanted, without image at all (both ways need different models for different tasks and label needs slight bit-rate).\n\nClarity: Could be better. The human-in-the-loop strategy can be explained earlier and more clearly, which is a key problem to discuss. The References should be arranged in a uniform style.\n\nSignificance: Not good. The paper does not provide enough data of JPEG with low bit-rate. As we can notice in Fig. 3, number 0, 3, 5 are not reconstructed enough and I doubt that some autoencoder-based methods can surpass it.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussed but not adequately addressed. Some constructive suggestions can be found above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
- Novel and elegant approach
- Clever training procedure that incorporates humans in-the-loop
- Can potentially give insight into human decision-making process (from visual stimuli)
- Well-executed experiments involving real human subject feedback


Weaknesses:
- No clear application or real-world use case
- Requires a more careful discussion on algorithmic decision support and agency 


Summary:

Reviewers were mostly unanimous in their opinion that this is a solid contribution to the growing literature on learning to support human decision-making. One reviewer initially raised several concerns regarding practicality, but the authors response (which included additional experimentation) were helpful in increasing his score.

Learning with humans in-the-loop is challenging, and this paper does a commendable job at executing experiments in this setting. As one reviewer comments, the authors should clarify the amount of human resources (time, queries, etc.) required in their approach. But more importantly, I would encourage the authors to discuss in detail the potential impacts of deploying their algorithm in the real world. Issues such as the role of algorithms for decision-support and their relation to human agency should be acknowledged – especially given that the paper does not highlight any specific use-case. The authors should also carefully consider and justify what they measure and why. All in all, paper presents a fresh approach and is likely to be the focus of interesting follow-up work.

Decision: Accept (Spotlight)