Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The goal in this work is to improve machine interpretability of images.
The authors main claims are:
-	Their proposed approach improves image recognition accuracy even without knowing subsequent recognition tasks and recognition models used to perform them (transferable model to different recognition models/tasks).
-	For this they propose what they call “Recognition-Aware” processing that combines image processing loss and recognition loss.
-	The approach is evaluated on three image processing tasks with two downstream recognition tasks:
o	Image super-resolution, de-noising, and JPEG-de-blocking processing tasks, with 
o	Image classification and object detection recognition tasks.

The paper is well written and organized, experiments carried are extensive but the reuse of known neural networks, many simplifications (shortcuts), a not clear enough methodology (see below), limited processing & recognition tasks used to support it, do not justify in our opinion the main (over-arching) work’s claim:
-	In 3.2 optimizing recognition loss/Last paragraph:  “Interestingly, we find that image processing models trained with the loss of one recognition model R1, can also boost the performance when evaluated using recognition model R2, even if model R2 has a different architecture, recognizes a different set of categories or even is trained for a different task.”.

The paper would greatly benefit (to understand the context of the work or the explanations provided) from clarification of the many under-defined, not clearly introduced concepts it carries:
-	Meaning of “Network” is not clearly defined:
o	Abstract: “image processing network”.
o	Introduction: “the network maps an image to a semantic label”
o	Later in the paper only networks introduced are deep neural networks. That should be clear from beginning of the paper.
-	“Retraining/Adaptation”  in 1st paragraph page 2.
-	In 1. Introduction/Paragraph 1: You use “.. techniques .. have been proposed for making the output images look natural to human”:
o	Noise is part of nature. A de-noised (smoothed) image is not “more natural”.
o	Enhanced (processed) images are not necessarily “more” natural, rather they take advantage of the human visual perception characteristics to enhance recognition for example.
-	In 1. Introduction/Paragraph 3:
o	“.. of great importance that the processed images be recognizable”  Should explain the concept of image recognition! Because it could be related to contained objects, overall description (for captioning for example) etc. 
-	“Image processing” in the context of the paper is intended only as “image enhancement for recognition”. Pattern detection, segmentation, object extraction etc. are not included in this restrictive definition. Should specify for example: image enhancement and restoration. 
-	Figure 1: As an illustration, it’s completely counterproductive for your discourse as many simple image recognition algorithms would recognize the bird even in the noisy image.
-	In 3. Unsupervised optimization of recognition loss: The “unsupervised RA” process is not  clear enough to us especially the statement:
o	“.. only “unsupervised” for training model P, but the target pre-trained model R can still be trained in full supervision.”.


-	“We may not know what network architectures (e.g. ResNet or VGG) will be used for inference, what object categories the downstream model recognizes (e.g. animals or scenes), or even what task will be performed on the processed image (e.g. classification or detection)”. 
o	Is your goal a universal “recognition model” applicable to anything?
-	I also have some trouble with the terminology:  
o	In 1. Introduction/Paragraph 4: “It is also important that the enhanced machine semantics is not specific to any concrete recognition model”: “enhanced machine semantics”!
o	In 1. Introduction/Paragraph 4: “..transferable among different recognition architectures..”. Does “architectures” refer to deep neural networks (DNN)? If yes, is recognition performed only by DNN? What about the preceding bullet (“is not specific to any concrete recognition model”)?
-	In 1. Introduction/Paragraph 3: 
o	 “.. we argue that image processing systems should maintain/enhance machine semantics”. Do not see what’s to argue here?
o	“Recognition-Aware Image Processing” is it simply put Image Processing techniques for recognition enhancement (“Recognition” still needs to be defined)?
-	In 2 Related work :
o	 “ .. we assume we do not have the control on the recognition model, as it might be on the cloud or decided in the future, thus we advocate adapting the image processing model only. This also ensures the recognition model is not harmed on natural images.” Care to explain?
o	: “to achieve better recover the face identity from low-resolution images”, Typo?
-	In 1. Introduction/Paragraph 1:
o	“ .. might not look “natural” to machines”: Care to explain this concept? 
	Would advise to just keep the second part of the sentence.
-	In 1. Introduction/Paragraph 2: “One could specifically train a recognition model only on these output images produced by the de-noising model to achieve better performance on such images, but the performance on natural images can be harmed.”  Care to explain?. 
o	More complicated images (noisier, multiple obstructions etc.) are recognized nowadays and true to actual applications.

-	3.4 using an intermediate transformer/Last paragraph:
o	“ .. that there are two instances for each image (the output of model P and T), one is “for human” and the other is “for machines”.”:
	The “Transformer” characteristics are not clearly defined for the intended output (For machines?).
	Why is output of model T not represented in Figure 2 (Right)?
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper presents several models for visual recognition in the presence of image degradation (e.g., low-resolution, noise, compression artifacts). In the models, an image enhancement network is placed in front of a recognition model and trained together with the recognizer to improve the recognition accuracy as well as to enhance the image quality. The proposed approach is simple, straightforward, yet effective. It has been also shown that the image enhancement module is transferable between different recognition tasks and architectures.

Although the paper addresses a timely topic and the performance gain is substantial, my current decision is reject mainly because of its weakness in technical novelty and contribution. The proposed models are simple and straightforward combinations of two separate networks, one for image enhancement and the other for recognition. This approach also makes the entire networks overly heavy, and introduces hyper-parameters (e.g., lambda) that have to be carefully tuned. Overall, it was hard to find interesting ideas that future readers may learn from the paper. 

Other comments:

The 2nd model based on knowledge distillation (KD) is called "unsupervised", which however sounds weird. As already mentioned in the manuscript, the teacher network for KD is trained in a fully supervised manner for the target task, so it cannot be considered as an unsupervised model. Further, the advantage of the 2nd model is marginal in practice.

The advantage of the transformer in the 3rd model is not clearly discussed. It is unknown in the paper why the 3rd model with the transformer works best in the experiments. Also, regarding the main goal of the paper (i.e., image enhancement not for human but for recognition networks), the reason for adopting the transformer is hard to understand.

The degrees of image corruption (e.g., down-sampling, noise, compression) applied during testing are not mentioned at all, although they are important to understand the empirical advantage of the proposed models. 

The transferability is one of the most important benefit of the proposed model, but not convincing sufficiently. The proposed models are transferable between different object categories, but the plain models seem to be also transferable, sometime more transparently. Also, it is not clearly discussed what makes the proposed models attaining the transferability.

It would be nice to apply the proposed models to the ImageNet-C benchmark.

Missing references
- Studying Very Low Resolution Recognition Using Deep Networks, CVPR 2016
-  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Claims: 

The paper presents a concept of "recognition-aware (RA) image processing": when one enhances image in a some way, not only human judjement should be taken into account, but also performance of various computer vision application using that image.

As an example of processing tasks, authors take super-resolution, denoising and JPEG-artifacts removal. Downstream applications covered are image classification and object detection. 

Authors propose a several training schemas to solve this problem and discuss a limitations of each one: 
 - "simple" preprocessing, when the only image enhancement loss is optimized
 - "RA" joint optimization of recognition and enhancement loss (supervised and unsupervised)
 - a variant when two images are created: one for human and one for machine.
 
****

Recommendation: strong accept

****
Comments: 

 Experiments are vast and performed on a variety of CNN architectures: ResNets, DenseNet ant VGGNet.
 Because one cannot predict, which computer vision tasks will be needed in the future, the natural question arise: how the results got for one set of tasks, architectures and image enhancement types transfer to another. Paper carefully studies this aspect as well.
  
 Overall paper is well written and is pleasure to read. While reading, I made notes to ask in review - just to see the my questions answered in a next section.
Authors also provide source code for training. I haven`t run them though, but glanced through them.
 
 Weaknesses: I cannot really find a significant one. As a minor points:
  - I would recommend to cite not the last papers for image enhancement porblems themselves like super-resolution and denoising: these are old problems with rich history, e.g.
 
 L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms Physica D, 60 (1992), pp. 259–268.
 
 - "Transformer" is probably bad name for deep learning component, as it is already widely used for a specific seq2seq architecture 


****
After rebuttal: I am now even more convinced that paper should be accepted.
review_title: Official Blind Review #3

Meta-review: This paper presents several models for recognition-aware image enhancement. The authors propose to enhance the image quality in the presence of image degradation (e.g., low-resolution, noise, compression artifacts) as well as to improve the recognition accuracy in a joint model. While acknowledging that the paper is addressing an interesting direction, the reviewers and AC note the following potential weaknesses: presentation clarity, limited technical contributions, insufficient empirical evidence. AC can confirm all the reviewers have read the rebuttal and have contributed to the discussion. All the reviewers and AC agree that the rebuttal was informative, and the authors have partially addressed some of the concerns (e.g. additional experiments). R2 has raised the score from reject to weak reject. However, at this stage AC suggest the manuscript is below the acceptance bar and needs a major revision before submitting for another round of reviews. We hope the reviews are useful for improving and revising the paper.
Decision: Reject