Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces  a novel DPS(Deep Probabilistic Subsampling) framework for the task-adaptive  subsampling case, which attempts to resolve the issue of end-to-end optimization of an optimal subset of signal with jointly learning a sub-Nyquist sampling scheme and a predictive model for downstream tasks. The parameterization is used to simplify the subsampling distribution and ensure an expressive yet tractable distribution.  The new approach contribution is applied to  both reconstruction and classification tasks and demonstrated with a suite of experiments in a toy dataset, MINIST, and COFAR10.


Overall, the paper requires significant improvement. 

1. The approach is not well justified either by theory or practice. There is no experiment clearly shows convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches (Xie & Ermon (2019); Kool et al. (2019); PlÂšotz & Roth (2018) ).

2. The paper never clearly demonstrates the problem they are trying to solve (nor well differentiates it from the compressed sensing problem  or sample selection problem)

   The method is difficult to understand, missing many details and essential explanation, and generally does not support a significant contribution. 

3. The paper is not nicely written or rather easy to follow. The model is not well motivated and the optimization algorithm is also not well described.

4. A theoretical analysis of the convergence of the optimization algorithm could be needed.

5. The paper is imprecise and unpolished and the presentation needs improvement.

**There are so many missing details or questions to answer**

1. What is the Gumbel-max trick? 
2. How to tune the parameters discussed in training details in the experiments?
3. Why to use experience replay for the linear experiments?
4. Are there evaluations on the utility of proposed compared to existing approaches?
5. Does the proposed approach work in real-world problems?
6. Was there any concrete theoretical guarantee to ensure the convergence of the algorithm.

[Post Review after discussion]: The uploaded version has significantly improved over the first submission. It is now acceptable. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose a new approach of deep probabilistic subsampling for compressed sensing, based on Gumbel-softmax, which is interesting.

A few points should be clarified:

- in compressed sensing one has e.g. the restricted isometry property (RIP) related to recovery. How does the new method relate to such theoretical results? Are the results and findings along similar lines as (classical) compressed sensing theory? 

- Methods in compressed sensing are typically convex, e.g. using l1-regularization. What are the drawbacks of using deep learning in this context, e.g. related to non-convexity? What is the role of initialization?

- Does the method both work for underdetermined and overdetermined problems (number of data versus number of unknowns)?

- What is the influence of the hyper-parameters mu and lambda in eq (14)? How should the model selection be done (currently lambda is set to 0.004 without further motivation)? 

- MNIST: 60,000 instead of 70,000?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a learning-based adaptive compressed sensing framework in which both the sampling and the task functions (e.g., classification) are learned jointly end-to-end. The main contribution includes using the Gumbel-softmax trick to relax categorical distributions and use back-propagation to estimate the gradient jointly with the tas neural network. The proposed solution has the flexibility of able to be used in several different tasks, such as inverse problems ( super-resolution or image completion) or classification tasks.  The paper is very well written. 

The paper locates itself well in current baselines and explains Experiments mostly well. However, there are significant limitations in demonstrating the effectiveness/impact of the proposed technique: 
1) The only comparison to another non-fixed sampling baseline is Kool et al. 2019. The visualization and a thorough comparison were missing in MNIST classification. This baseline was also missing in image reconstruction.  
2) Compressive Sensing incorporates vast literature of algorithms focusing on different aspects of improvements; algorithms focused on classification and inverse problems. Even if done disjointly, how does the proposed joint learning is compared to those algorithms in these domains? 
3) Top row of Figure 3 nicely explains how the learned sampling paradigm performs compared to other mechanisms (such as uniform, random, low-pass). But there is no comparision against other non-fixed techniques.  
review_title: Official Blind Review #3

Meta-review: This paper introduces a probabilistic data subsampling scheme that can be optimized end-to-end.  The experimental evaluation is a bit weak, focusing mostly on toy-scale problems, and I would have liked to see a discussion of bias in the Gumbel-max gradient estimator.  

It's also not clear how the free hyperparameters for this method were chosen, which makes me suspect they were tuned on the test set.

However, the overall idea is sensible, and the area seems under-explored.
Decision: Accept (Poster)