Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents an approach to discrete input selection for NNs, using the Gumbel-Softmax trick at its core. It motivates this problem in the context of communicating data over a network with limited bandwidth budget. It proposes constructing different kinds of masks that can be applied over channels or pixels in the input, grounding the discussion in the image domain. This can be seen as a special case of Feature Selection, with image specific substructures motivating the choice of mask types. 

There is very little novelty in this work over that presented in Abubakar Abid et. al. [1], where the idea of using Gumbel-Softmax as a differentiable Feature Selection algorithm has already been expounded at depth, both in unsupervised as well as supervised settings. The current work draws directly from the supervised form in [1]. The only incremental contribution in this work is the specific mask types and mask-specific losses.

Pros
•	Interesting approach to extend the framework in [1] to CNNs, with use of masks and mask-specific loss
•	Clear motivation for the network bandwidth limited use case

Cons
•	Hardly any technical novelty because the core ideas are already presented as well as applied to the same task in [1]
•	It is very surprising that the authors do not even cite [1] in their paper, despite their work being extremely closely related to it
•	Most of the discussion in the Related Work section is unrelated to the specific task they tackled in the paper (i.e., input/feature selection). The second paragraph in this section talks about ‘gradient-driven search’ for discrete selection, which has been recently explored not only in [1] but also related G-S applications like [2], [3], but the authors seem unaware of this line of works
•	The authors do not compare their approach against any existing baselines from literature for this task, again with the most apt being [1] and baselines therein. This makes it hard to understand the true value of their proposals such as mask types, schedule that adjusts both ‘tau’ and ‘lambda’ during training etc.

[1] Abubakar Abid et al., “Concrete Autoencoders for Differentiable Feature Selection and Reconstruction”, ICML 2019, (https://arxiv.org/abs/1901.09346)
[2] Hanxiao Liu et al., “DARTS: Differentiable Architecture Search”, ICLR 2019
[3] Bichen Wu et al., “FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search” CVPR 2019

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper addresses the problem of high-cost transfer between server and user for machine learning applications. The method proposes to augment inputs with channel/spatial masks that are trained via the Gumbel-softmax trick together with the model's weights trained/finetuned to account for the loss of information. These learned masks are then applied to the image before sending it to a server with where inference takes place to reduce file transfer costs. In the experimental study, the paper shows that on computer vision tasks inputs can be reduced with relatively little drop in accuracy and analyses how hyperparameters of the model affect its performance. The framework is also adapted to the downstream task-guided choice of compression techniques, e.g. which compression quality to choose for JPEG that will be an optimal trade-off in terms of downstream task quality vs data transfer cost.
Overall, this paper could be a valid algorithmic contribution, however, I have concerns about how this method fares with others, resolving the following issues in the author response will likely increase the score.
- A discussion on the connection/comparison with representation learning and dimensionality reduction (VAEs, quantization, etc) would help improve the exposition of the paper and help define how and when the suggested method is more appropriate to use, as well as citations to the other lines of work in reducing the model size (knowledge distillation [1], tensor decomposition approaches [2], adaptive computation time techniques [3], etc).
- I would like to see an experiment that compares other works mentioned as related work. [4] has been mentioned as one of the nontrivial compression methods for image data, how does the proposed method compare to it?
- Another experiment comparing a method from representation learning, e.g. a VAE trained with the embedding size corresponding to some optimal Q value in this work would be helpful.
- Please include reference accuracy values for the dataset/NN pairs used in Table 1.
- In Section 4.5, the paper states that large lambda values correspond to blue and red lines, however in the corresponding figure large lambda values correspond to blue and orange lines which exhibit different behaviors, could you please clarify this?

References
[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network."
[2] Novikov, Alexander, et al. "Tensorizing neural networks." Advances in neural information processing systems. 2015.
[3] Figurnov, Michael, et al. "Spatially adaptive computation time for residual networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
[4] Jiang, Feng, et al. "An end-to-end compression framework based on convolutional neural networks." IEEE Transactions on Circuits and Systems for Video Technology 28.10 (2017): 3007-3018.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors argue that data transfer costs for ML inference may be significant in some scenarios, such as for remote sensing. They propose to jointly train a model to maintain good performance while significantly reducing input size by applying a global mask. To allow training with discrete masks, the Gumbel-softmax trick is used. The experiments cover multiple datasets and multiple types of mask. Generally, performance remains reasonably high and degrades gracefully as the input is increasingly masked.

I lean slightly towards rejecting the paper. The problem is interesting and potentially important, but many experiments are  too simplistic and lack strong baselines.

I believe the problem is well motivated, but not very much explored yet. There has been much work on reduce ML system computation costs and memory storage requirements, but mostly by modifying the model instead of the data. The paper proposes a reasonable approach to reduce data transfer costs.

The authors propose 4 types of masks (channel/any, channel/xor, pixel/any, pixel/xor), which are applicable under different circumstances. Some of them may also be combined. To learn discrete masks, they apply the Gumbel-softmax trick. Results clearly show that learning discrete masks (reducing input size) while maintaining decent performance is feasible .

As the objective function is modified during training by adjusting \lambda, the performance/size trade-off is only loosely specified. All presented results are learning curves, but there are no clear final numbers.

The channel selection task (4.1) is potentially interesting, but lacks a baseline. How does random selection of channels perform?

The pixel selection task (4.2) is simplistic. Using a cloud of pixels near the center of the images appears sufficient, which could be inferred by looking at a few samples and doesn't necessarily necessitate learning. Could the approach be extended to more complex images, predicting one mask per image instead of a global mask?

Feature map selection (4.3, channel 'xor') could be likely solved with hyper-parameter search, especially if the number of channels is small. Section 4.4 combines the previous two subsections, and it is unclear how much we gain from learning the masks over using simple heuristics.

More minor points:

In the related work section, the author could additionally mention distillation.

The Gumbel noise is used on half of the inputs, while the current argmax is used otherwise. It is unclear whether this is necessary, and there are no related experiments.

Although this is not crucial, for the 'any' variants, the last mask dimension appears superfluous (2-class softmax). The binary variant of Gumbel-softmax (Maddison et al. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables, Appendix B) could be used.
review_title: Official Blind Review #1

Meta-review: This paper proposes to address the high bandwidth cost when transferring data between server and user for machine learning applications. The input data is augment with channel and spatial mask so that the file transfer cost is reduced. While the reviewers agree that this is a well motivated and interesting problem to study, a number of concerns are raised, including loosely specified performance/size trade-off, how this work is compared to related work, low novelty relative to a few key missing references. The authors respond to Reviewers’ concerns but did not change the rating. The ACs concur the concerns and the paper can not be accepted at its current state.
Decision: Reject