Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: The authors propose to solve the problem of learning sparse masks end-to-end during training. They do this by training an MLP model to optimize reconstruction loss measured against an oracle.
strengths_and_weaknesses: The problem statement is valid. The need for differentiabiity is a known aspect of generating sparse masks.

The work seems very behind on literature survey. Learning sparse masks has had some attention and there are a variety of techniques under investigation that employ this idea of learning the masks The authors have the right sense of the problem and should formulate it correctly in the empirical setting (see a few references below).

The experiment is quite preliminary and the evaluation is missing.  

Gonc ̧ alo M. Correia, Vlad Niculae, and Andr  ́e F. T. Martins. Adaptively sparse transformers. CoRR,
abs/1909.00015, 2019. URL http://arxiv.org/abs/1909.00015.

Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, and Xu Sun. Explicit sparse transformer: Concentrated attention through explicit selection, 2019.
Sung, Y.-L., Nair, V., and Raffel, C. Training neural networks with fixed sparse masks. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021
suggested_changes: Method 3 is where the paper shold begin even in its correct form. Methods 1 and 2 use terms that are not defined until later.

The evaluation cannot be limited to the visualization. The authors say future work is needed to determine how these methods scale to
different downstream tasks - that is where this work needs to go.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper focuses on the problem of creating end-to-end learnable masks by making the masking indices differentiable. The authors study three different methods and compare their performance on a simple validation task.
strengths_and_weaknesses: **Clarity and Correctness**

The paper is well-written and easy to follow. Method discussion is clear and concise. While the authors evaluate the three methods on a simple (single) validation task, there is no discussion of the results (for example, comparing the three methods among themselves; the relation between the final predicted mask in Fig. 1, and the method itself, etc). This makes it hard to derive key takeaways from the work. 


suggested_changes: Overall, I do like the work and strongly believe it's a relevant problem to tackle. I would love to increase the rating of the paper, if (at least some of) the following changes can be incorporated:
- I urge the authors to open-source the code after the review period. Also, I also strongly believe that the readers would significantly benefit from a pseudocode implementation of the three proposed methods (perhaps in the appendix). 
- At its current stage, the authors only present results on predicting a single target mask. While I appreciate the simplicity of the evaluation protocol, it's too little to derive any conclusions. For example, are certain masks easier to learn by certain methods; I also observe certain patterns in the final predicted mask, is there any relation to the method itself; etc.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: In this paper, the authors focus on the problem of differentiability of sparsity inducing methods. To this end, this paper proposes three methods for making sparsity-inducing operation fully differentiable. In turn, this allows the methods to be used as layers and submodules within neural networks.
strengths_and_weaknesses: **Strengths**
* S1: The three methods have been explained clearly and concisely
* S2: Experiments have been presented to evaluate the three methods.
* S3: clear directions have been identified for future work

**Weaknesses**
* W1: Hyperparameter values and other details used for experimenting are not shared which can be a challenge if someone id trying to reproduce the work.
suggested_changes: C1: Some graphics can be added in the appendix, explaining each method visually. This would allow the readers to understand the three methods in a better way.
review_title: null

Decision: Invite to archive