Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper develops methods to perform active subsampling. That is, given some downstream task like classification or image reconstruction, it sequentially selects which elements of an image or signal to sample so as to perform said task. It does so by extending the Deep Probabilistic Subsampling (DPS) method developed by Huijben et al. The proposed method is applied to two problems as well as a simplified, low-resolution MRI reconstruction problem.

### Strengths
Motivation: Active sampling is an interesting idea that has been around for some time, but was often computationally impractical. Thanks to GPUs and deep learning, active sampling is becoming more practical and its interesting to see new work in this direction.

### Weaknesses
Novelty: The method is a small extension to the DPS method where the network that selects which rows to samples is conditioned on the existing measurements.

Validation: The paper did not compare to any other active sampling strategies. The authors made no effort to replicate existing methods.

Clarity: 
The Markov chain example in section 4.1 was hard to follow and more distracting than informative. The phrase "the task model gets to sample only one position out of every three" reads as if the model is sampling one position out of every three in the sequence. It took some time before I realized this meant that at every position in the sequence it was probing one of the three states.

Impact: The results with active sampling were only marginally better than results with a fixed (learned) sampling strategy.

Limitations: The method is applicable only to true subsampling problems, not general sensing. That is, one isn't designing the rows of a measurement matrix on the fly but rather selecting which row from an existing matrix (identity in most of the examples) that one would like to sample from.


### Recommendation
The paper's presentation could be improved and it is sorely missing comparisons to other active sampling methods. I don't think the papers novelty is enough to overcome these issues and so I do not believe it is ready for publication.

### Comments
While the proposed method was computationally impractical, active sampling was discussed extensively in [A] from a information theoretical perspective.
[A] Ji, S., Xue, Y., & Carin, L. (2008). Bayesian compressive sensing. IEEE Transactions on signal processing, 56(6), 2346-2356.

Because of the nonlinearity in the forward model, equation (9) is not actually proximal gradient descent. I believe there's a sign(F^HD\circFX) term missing from the (sub) gradient.

### Update

I thank the authors for their comprehensive response. While its unfortunate they couldn't compare to any other active methods, the related work and overall clarity of the paper is significantly improved. The t-SNE plots were informative and interesting. While I have reservations about the paper's lack of comparisons, I think its publication still might be a net positive for the research community.

I have updated my score.

##### Other comments
Let A(X)=F^H D\circ F X. The expression A^H(Ax-Y\circ sign(A(x))) is a subgradient of 1/2|| Y - |A(X)|||^2 but A^H(|Ax|-Y) is not. I would avoid calling (9) projected gradient descent as the "gradient" isn't really a gradient.

"We have performed a statistical analysis on the performance gains made by A-DPS over DPS in the MRI reconstruction task, concluding that they are indeed statistically significant." It would be nice to see confidence intervals in Tables 1 and 2.

#### Questions/comments that do not effect the review:
Why use an LSTM/any network with memory? It seems the next sample depends on the previous samples, but not their order. The ablation study on pg 6 shows that memory helps (at low sampling rates), but I don't understand the intuition why. Could the LSTM just have more capacity?

Typos:
pg 2: "cells.During" space
pg 3: "However, This" capitalization
review_title: Limited novelty and validation

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors consider the problem of compressed sensing where the underlying signal of interest is captured and restored based only on sparse measurements: Specifically, this paper focuses on the scenario of Deep Probabilistic Subsampling (DPS) which finds sparse measurements in the way that the models designed to solve specific learning problems based on these measurements are jointly optimized. The authors extend DPS to a sequential framework that iteratively and actively selects the next measurement points: The proposed approach encodes the information accumulated until a time step into a context vector which is updated, and used in selecting the next point, in an LSTM-like framework (see minor comments below). In the experiments with two toy problems (including MNIST) and an MRI reconstruction problem, the authors demonstrated that the proposed Active DPS (ADPS) outperforms DPS (in toy problems) and three other compressed sensing algorithms (for MRI reconstruction).

I think this paper makes a borderline case: DPS provides a framework that combines the compressed sensing part (sparse data acquisition) and the subsequent learning part in an end-to-end manner. This paper contributes by extending DPS into an active/sequential learning framework achieving significant performance gains over DPS (mainly on toy problems. see minor comments below). On the other hand, the proposed approach appears to be incremental: ADPS adds a simple sequential update structure (of a context vector) to DPS, which can be described by only two equations (6 and 7). The simplicity of the changes proposed (over DPS) is not a limitation, but it could be accompanied by an in-depth theoretical analysis, a convincing qualitative discussion or _extensive_ experiments demonstrating the practical relevance of the proposed approach.

Minor comments
- Apart from the last one paragraph, the Introduction Section focuses on discussing the context and motivation of Deep Probabilistic Subsampling (DPS). Instead, the authors could use this space to describe and characterize the proposed Active DPS in detail. 
- I was not sure why the proposed architecture (Figure 1 and equations 6 and 7) is called LSTM, it has a recurrent network structure but I was not able to find any attention (gating) mechanism that characterizes LSTM. Please advise me if I missed anything. 
- Please test if the improvements gained by ADPS over DPS on MRI reconstruction are statistically significant.

Update:

Thank the authors for their responses, clarification, and additional experiments. I read through authors’ responses and the comments from the other reviewers. I still think this paper makes a borderline case for 1) its technical contribution on extending DPS and thereby achieving significant performance gain on a toy problem and MRI reconstruction tasks, still 2) with limited novelty and room for a more extensive experimental validation (perhaps, beyond MRI). My other concerns on clarity and significance of experiments have been addressed. I would raise my rating to marginally above acceptance threshold (borderline).

review_title: A borderline case?

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY:
The paper at hand deals with compressed sensing (CS) and introduces an extension to deep probabilistic subsampling (DPS) called active deep probabilistic subsampling (A-DPS): instead of learning a sampling pattern that is equal for each element of the dataset, A-DPS adaptively selects entries (of each element) based on the information acquired so far. It is shown that this active sampling increases performance for different tasks: a toy example that aims to demonstrate the benefits of active sampling, a classification task (from subsampled inputs) on the MNIST dataset and a reconstuction task on the NYU fastMRI database of knee MRI volumes. 

STRENGTHS:
1. The paper is very clearly written and very comprehensible. Furthermore, it is very detailed about the experimental setup. I also liked the description of the general framework which thoroughly defines the used notation.
2. The idea is well motivated and the approach of selecting samples depending on the previously selected ones makes intuitively sense.
3. The results of the experiments on MNIST and the NYU fastMRI data are promising. A plenthora of (non-active) subsampling schemes are benchmarked as well.

WEAKNESSES:
1. The greatest weakness of this paper is the missing comparison to other active sub-sampling schemes (Zhang et al., 2019; Jin et al., 2019). It would be nice to see wether the proposed method produces better results than the existing methods. 
2. I found the toy example very constructed. It is not really easy to understand and does in my opinion not improve the quality of the paper.

QUESTIONS:
- What happens when the MNIST sampling ratio in Figure 3a is further increased? Does A-DPS consistently outperform DPS in low sampling ratio regimes?

DECISION:
Overall, the paper presents an interesting and novel approach. However, it remains an open question wether the proposed A-DPS scheme performs better than already existing active subsampling schemes. Besides this, the experimental evaluation is solid. I lean towards acceptance.

UPDATE AFTER REBUTTAL:
I thank the authors for their responses and appreciate the inclusion of some of the requested changes in the paper. However, the paper still misses the comparison to other adaptive methods which is the paper's greatest weakness. Therefore, I decided to keep my score at 6.

MINOR REMARKS:
- Caption of Table 1 could use some more spacing

review_title: Interesting extension of an existing method with somewhat limited experiments

Meta-review: The review phase was very constructive, where reviewers raised several opportunities for improvements. The authors did a very good job in their rebuttal, which led some reviewers to change their opinion in a positive direction. Overall, reviewers agree that this is the borderline paper with remaining concerns about the weak experimentation.  The paper was again discussed by the Area Chair and Program chairs.  Due to the competitive nature of the conference and the high bar of experimental evaluations expected by empirical papers, the paper was finally rejected.  We hope authors will use the feedback from the reviews and make a stronger submission in near future. 
Decision: Reject