Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The foundational motivation for this work is the quite reasonable assumption that when a learned image compression codec is to be used as a component in a larger system, it might be desirable to optimize the codec for its functional utility for the downstream task instead of general perceptual quality. Forgoing the need for good all-around perceptual reconstruction can therefore lead to large gains in terms of compression ratio.

The main challenge is to quantify the functional similarity of a compressed image compared to the original image, without making any assumptions about what the users' downstream task is, or how the users select their actions upon seeing an image. To this end, the authors propose an adversarial framework. In a manner similar to A/B testing, they present users with either the original images $x$ or their compressed versions $\hat{x}$, and record what actions $a$ the users took. Then, they train a discriminator $D(a, x)$ to predict if a user's action was taken after seeing the original or the compressed image. This first discriminator is distilled to a second discriminator $D(x, \hat{x})$, which predicts which image was seen based on the prediction of the first discriminator. Now, the second discriminator $D(x, \hat{x})$ can be readily used as a measure of functional similarity between $x$ and $\hat{x}$ for the unknown downstream task of the user.

If this system was to be trained end-to-end, it would require access to a large amount of data harvested from the users. To circumvent this, the authors use pre-trained generative models in their experiments, and optimize the discriminators on quantities derived from the generative models' latent spaces, making the optimization process converge much faster. 

The authors perform extensive experiments to verify that 1) their proposed adaptive method performs better than general-purpose, non-adaptive methods, 2) their method adapts to the specific needs of the downstream task and 3) their method can learn in a sequential decision-making setting. They obtain good results for all 3 settings.

main_review: # Strengths
The work is well-motivated, the problem and the framework are interesting, and the utility of a solution is clear. The use of the adversarial framework to obtain a proxy metric for the functional similarity between the original and compressed images is a simple and elegant solution.

The reduction of the problem to the latent space of generative models is also elegant and greatly extends the applicability of the proposed method. 

The experiments are well-motivated, well-set-up and well-executed. The illustration and analysis of which features PICO retains or allows to vary is very convincing.

The paper is well-written with good structure. Very nice submission overall.

# Weaknesses
I am slightly confused about the "bitrate-user action agreement" graphs. Concretely, I don't understand how the methods can start at 0 bitrate. If the bitrate is actually a positive number close to 0, then this should be cleared up as the graphs are confusing/ misleading in their current state.

# Questions
- Have the authors attempted end-to-end training their method on toy tasks instead of reducing it to the latent space of a generative model? It seems to me that one reason the proposed method works so well is because the general-purpose compression codec provided by the generative model is already quite good.
- In particular, one reason for such a clear behaviour of the system to keep relevant features and randomize irrelevant ones is due to the fact that the authors used pre-trained generative models that are known to be biassed towards learning independent latent features. Do the authors think we would see a similar disentanglement if they trained the system end-to-end?

limitations_and_societal_impact: They have.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors claim to account not just for optimizing a lossy image compression algorithm to preserve an image's appearance but additionally to minimizes bits while maintaining information necessary for a user to complete downstream tasks correctly.  They achieve this by training a discriminator with human in the loop training and demonstrate the effectiveness of their methodology.  They claim that they show that the method learns to match the user's actions with and without compression at lower bitrates than baseline methods and adapts the compression to user behavior.
main_review: This is a clever paper that uses a novel combination of methodologies to achieve its results. These authors are not the first to train a compression algorithm for downstream tasks as an objective, but as far as I can tell, they are the first to have those downstream tasks completed by real users in the training loop, such that the compression algorithm is learning to optimize for real users and not some modeled approximation to them, or to some CV algorithm.   They call this PICO (pragmatic compression).  

The results themselves (though meeting the authors claims) don't suggest to me that they are extremely useful as an application as of yet (ie you wouldn't surface these images via a web tool for anyone to actually complete the task if you were relying on that task being done perfectly).  However, the key application (not fleshed out by the authors) seems to me to be in the cognitive sciences, as this serves as a powerful engine for pulling out of images the key elements that people are relying on to do the downstream tasks.

Overall I think this is a novel and cleve contribution that will be interesting to many people at NeurIPS.

limitations_and_societal_impact: Could be used to illicit what parts of images or other inputs human users are utilizing to make their decisions without the user realizing that is happening, not just in toy tasks but in other everyday interactions with things like phones and other devices. Though the method would have to progress quite significantly before that would become a real worry. The authors did not discuss this though they may not have realized the impact for implicit information extraction that is available via a method like this.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a downstream task specific compression method by training a compression model through human-in-the-loop leaning, which adaptively maintain core information based on whether human can make the same decisions when they see compressed or original images. The user study show the proposed method outperforms JPEG codec.
main_review: Originality: Novel but not good. The human-in-the-loop learning is firstly used in compression to the best of my knowledge. The proposed method directly evaluates the quality of images based on the consistency of human decisions seeing compressed/original images. And the distillation design, using another discriminator which judge images with action to train the wanted discriminator which judge images directly, is brilliant. 
However, this design hugely relies on participants, which limits its generalization and evaluation. In fact, it replaces a lot of concurrent perceptual loss functions with “human-in-the-loop” in my opinion.

Quality: Bad. The drawback of this work is intuitive, that is, it requires a lot of users to infer which actions to take, and it is time-consuming because with the model trained, pictures provided to users change as well. The paper discusses this in Sec. 6 but I do not totally agree that using web services is a good solution, which means the model will be hugely affected by users during a period, which can be biased in my opinion. 
On the other hand, whether the “human” in the loop can be replaced with a neural network, which just requires ground truth label and it can be collected at a time, is not discussed. The excellent performance of neural network to simulate certain functions should be considered, and I think adding some module like attention map will improve the performance.
Furthermore, there is a lot of advanced compression methods besides relatively old JPEG. More experiments for comparison should be provided. And user study data of JPEG is not enough at all. I want to see its performance with low bit-rate.
To summarize, I doubt that the proposed hugely participant-oriented method is a good solution to so-called Pragmatic Image Compression. And if we just consider the performance of some specific task, why not directly training a network to tell users the label they wanted, without image at all (both ways need different models for different tasks and label needs slight bit-rate).

Clarity: Could be better. The human-in-the-loop strategy can be explained earlier and more clearly, which is a key problem to discuss. The References should be arranged in a uniform style.

Significance: Not good. The paper does not provide enough data of JPEG with low bit-rate. As we can notice in Fig. 3, number 0, 3, 5 are not reconstructed enough and I doubt that some autoencoder-based methods can surpass it.

limitations_and_societal_impact: Discussed but not adequately addressed. Some constructive suggestions can be found above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper propose a deep image compression method that train a compression model through human-in-the-loop.
The method train discriminator that tries to distinguish whether a user’s action was taken in response to the compressed image or the original. They evaluate their method through tasks that do not rely solely on perceptual similarity.
main_review: I think this paper is interesting challenge to the limit of bit rate by taking advantage of the characteristics of deep image compression. 
Using action discriminator and distilled image discriminator for training the image compression model may be a new approach.
Also, I think it would be useful for readers to show how much learning time is required for human-in-the-loop learning compared to baseline.
One minor concern is that the bitrate is expressed in comparison to PNG and not in absolute bitrate (e.g. bit-per-pixel), which makes it difficult to get a sense of the bitrate level.
limitations_and_societal_impact: I thought it would be good to discuss whether having a person in the learning loop could potentially introduce bias into the compression results.
review_title: null

Meta-review: Strengths:
- Novel and elegant approach
- Clever training procedure that incorporates humans in-the-loop
- Can potentially give insight into human decision-making process (from visual stimuli)
- Well-executed experiments involving real human subject feedback


Weaknesses:
- No clear application or real-world use case
- Requires a more careful discussion on algorithmic decision support and agency 


Summary:

Reviewers were mostly unanimous in their opinion that this is a solid contribution to the growing literature on learning to support human decision-making. One reviewer initially raised several concerns regarding practicality, but the authors response (which included additional experimentation) were helpful in increasing his score.

Learning with humans in-the-loop is challenging, and this paper does a commendable job at executing experiments in this setting. As one reviewer comments, the authors should clarify the amount of human resources (time, queries, etc.) required in their approach. But more importantly, I would encourage the authors to discuss in detail the potential impacts of deploying their algorithm in the real world. Issues such as the role of algorithms for decision-support and their relation to human agency should be acknowledged – especially given that the paper does not highlight any specific use-case. The authors should also carefully consider and justify what they measure and why. All in all, paper presents a fresh approach and is likely to be the focus of interesting follow-up work.

Decision: Accept (Spotlight)