Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Structured dropout are methods that determine probability of deactivation based on certain inductive priors such as neighborhood or location, which are important in architectures like CNN.  
The author surveyed existing structure dropout method and developed improved version based on block dropout called PROBDROPBLOCK. which assign different deactivation probability based on activation levels. It is interesting to see these method work well in transformers. 
strength_and_weaknesses: strength: clear motivation ,simple method, wide impact 

weakness: this manuscript serve well as a systematic review article , but lack of novel in method as a original research article by ICLR standard. The motivation or theoretical analysis of why choosing to conduction stochastic dropout in PROBDROPBLOCK is unclear. 
clarity,_quality,_novelty_and_reproducibility: very  clear, reproducible, good writing quality but lack of novelty in method 
summary_of_the_review: Although lacking novelty in method, due to the importance and popularity of dropout in machine learning and the well designed experiments, I think this could bring values to the community
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ProbDropBlock, an adaptive version of DropBlock in which the drop probability of the mask is defined to be negatively correlated to the absolute value of the centroid entry. Experiments are conducted on 3 NLP datasets (MNLI, QNLI, RTE) and 3 CV datasets (Cifar-10, Cifar-100, Imagenet).
strength_and_weaknesses: Strength:

1. The paper is easy to follow

2. The idea of investigating both unstructured and structured dropout methods on SOTA models w.r.t. both NLP and CV tasks is well motivated and beneficial for advance of these two research tasks.

Weakness:

1. Experiment results of baseline method are not SOTA. For example, the (top-1) accuracy of Resnet50 on Imagenet reported in Table 5 is lower than that in Ghiasi et al. (74.22 vs 76.51). Other examples include the accuracy of WideResNet28 on CIFAR100 (which is lower than that reported in the original WRN paper). This weakens the claim that made in the introduction section about testing dropout methods on current SOTA models. Some SOTA results can be found in https://paperswithcode.com/sota/image-classification-on-cifar-100

2. Lack of explanation and analysis about the adaptive dropout ratio, which is negatively correlated to the absolute value of layer output activations. It is hard to understand why the proposed method performs better than existing methods. 

clarity,_quality,_novelty_and_reproducibility: This paper is easy to follow. The proposed method about adaptive dropout ratio needs further analysis (theoretically or experimentally). Experiment settings can be found in the paper.
summary_of_the_review: In general, this paper is well motivated and focuses on an important problem which will be beneficial for future studies in both NLP and CV communities. However, both theoretical and experimental analysis are not thorough and do not support the claims well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes application of structured dropout methods (e.g. drop block) to transformers.  It also proposes an enhancement to drop block which utilizes activation values to assign dropout probability to nodes in feature maps as opposed to random selection of blocks.  Empirical results on language and vision tasks demonstrate that structured dropout is advantageous for transformer models and the proposed modification to drop block methods leads to an improvement in accuracy for language and vision tasks.

strength_and_weaknesses: Pros:
* Exploration of structured dropout for transformers
* An enhancement to drop block that shows better empirical performance as compared to the baseline approach
* Combination with linear dropping schedule gives further gains, reconfirming the value of linear dropout schedule

Cons:
N/A
clarity,_quality,_novelty_and_reproducibility: Clarity: very clear
Novelty: sufficiently novel
Reproducibility: fairly straightforward, should be reproducible
summary_of_the_review: see comments above
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an adaptive structured dropout method, ProbDropBlock, which drops contiguous blocks from feature maps with a probability given by the normalized feature salience values. The authors evaluate ProbDropBlock on both vision and language tasks.
strength_and_weaknesses: Strengths:

1. The paper proposed an adaptive structured dropout method that is beneficial for both vision and language tasks. It indicates the potential of using structured dropout in Transformer-based models.
2. The paper provided an thorough investigation of existing structured dropout methods.

Weakness:

1. The experimental settings and the performance gains are insufficient to demonstrate the effectiveness of the method. In Table 2, I am not quite convinced by the RoBERTa results, as unstructured dropout can usually achieve some gains for Transformers/ViTs. The authors may present results on more models/tasks to show that structured dropout works better in Transformers/ViTs. For a fair comparison, the authors may compare the proposed adaptive structured dropout to adaptive unstructured dropout. In Table 3, the gains are marginal (typo: the gain on QNLI should be $0.15$). 

2. The motivation of assigning higher drop rate to activations with higher magnitude is unclear. More ablations and analysis are needed to understand its influence to the training. It seems to impose a strong regularization but may also prevent the model from seeing important features. 

3. Expanding blocks based on the sampled $A_{i,j}$s may be problematic as the surrounding activations may not have high magnitude on average.

clarity,_quality,_novelty_and_reproducibility: 1. the writing is generally clear and understandable.
2. The idea is simple and novel. The technical contribution maybe limited: it is a combination of structured and adaptive dropout. Using magnitude as saliency score is also adapted from model pruning. 
summary_of_the_review: The paper proposed an adaptive structured dropout method that is beneficial for both vision and language tasks. It indicates the potential of using structured dropout in Transformer-based models. However, the experiment results are not sufficient to demonstrate the effectiveness of structured dropout and the proposed adaptive idea, especially in Transformers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a version of structured dropout called ProbDropBlock. They drop contiguous blocks from feature maps with a probability given by normalized feature salience values. All reviewers agree that revisiting structured dropout is a good research direction to continue for the deep learning community. However, the consensus among reviewers is that the improvements on MNLI and ResNet50 are too marginal and are not state-of-the-art results. Another main concern is that the motivation for assigning a higher drop rate to activations with a higher magnitude is unclear. Overall, the reviewers have seen merits in this paper, but because of the lack of empirical evidence, the panel's consensus is that this paper is not ready for ICLR 2023. 
justification_for_why_not_higher_score: See the meta review and summary.
Decision: Reject