Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary of the paper:
The paper tackles the probabilistic classification problem which is interesting and important in deep learning. The paper proposes an approach called omnibus dropout. Omnibus dropout is a sequential execution of existing dropout techniques such as drop layer, drop channel, drop block and regular dropout. The experimental results suggest that omnibus dropout perform reasonably well in various datasets.

Pros:
1. The proposed approach is reasonable which I found no surprising it produces reasonably well results.

Cons:
1. The paper could be improved in writing, especially on justification why this kind of dropout combination gives better performance. 
2. The results are mixed and the improvements are not significant. So I am not convinced the proposed approach is always a better strategy.
3. The proposed approach is simply a combination of the existing dropout methods. The contribution of the paper is very limited.

Questions:
1. In table 1, different dropout rate is chosen for different method. I think it is also reasonable to compare different methods with the same dropout rate (basically this corresponds to the actual model capacity).
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: PAPER SUMMARY: The paper argues that (ensembles of models with different types of dropout applied to each model) perform better than (ensembles in which the same type of dropout is applied to each model). They attribute this to increasing model diversity in the former case, and experimentally validate their claims.

MAJOR COMMENTS: 
1. Motivation Unclear:
- Notational issues in (2): MSE(h_t | x) involves y, but y is not specified in the definition. As a result, later E_{x}[…] is evaluated disregarding the dependence on y, which is ambiguous. 
- Derivation of the second equality in (2) is not obvious, and needs a detailed proof. Notational issues exist in switching between H and h.
- The above is used to argue that “the more diverse the models, the better performance achieved”. It is unclear how this follows from (2).

2. Unclear / Imprecise Writing
- Before Sec. 3.4:  “This is because higher Dropout rates lead to smaller effective network capacities” Needs reference. 
- “Dropout uncertainty can be obtained sequentially”. Unclear what this means.

3. The entire proposed method is described in one line, “we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods”. It is very unclear as to what the authors mean by combination.

4. Experiments
- It seems that Omnibus dropout leads to a moderate “diversity”, and improves ensemble performance only for SVHN, out of the three datasets tested in Figure 4. For SVHN, the improvement is 0.1% accuracy, which is within the standard error (0.1). Hence, it seems that the proposed method provides no performance improvement.
- A Similar statement is true for the other metrics (NLL, Brier Score, ECE). 

Score [Scale of 1-10]:
3: Reject
The paper needs either more convincing experiments demonstrating the claims or theoretical analysis explaining the behavior for small networks. The paper needs to be rewritten to improve presentation, and state motivation, problem statement, and contributions clearly. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed to use multiple structured dropout techniques to improve the ensemble performance of convolutional neural networks as well as the calibration of their probabilistic predictions. 

The approach, which is termed omnibus dropout and combines multiple existing dropout methods, is reasonable and straightforward. 

The paper presents extensive experimental results and analyses by mainly comparing with the explicit ensemble of multiple neural networks. The experiments reveal interesting properties of the learned networks and compelling results. 

My main concern is about the technical novelty of the paper. It does not supply a new method in essence and instead provides careful experimental studies, from both accuracy and calibration perspectives, about a combination of existing dropout techniques. It reads like a very solid workshop paper in my opinion, but it is probably not a good fit to the main conference.

Questions:
1. The reasoning in the first two paragraphs of the introduction is confusing or misleading. The first paragraph is mainly about the poorly calibrated probabilistic outputs of neural networks, while the second paragraph suddenly shifts to the performance of ensembled networks in terms of accuracy. 

2. Some of the comparisons with the "deep ensemble" may be unfair. There are only five networks in "deep ensemble", but 30 are used in test time for the proposed method. 

3. Where is "deep ensemble" in the active learning experiments? I could not find it in Figure 5. 
review_title: Official Blind Review #1

Meta-review: The paper investigates how to improve the performance of dropout and proposes an omnibus dropout strategy to reduce the correlation between the individual models.

All the reviewers felt that the paper requires more work before it can be accepted. In particular, the reviewers raised several concerns about novelty of the method relative to existing methods, significance of performance improvements and clarity of the presentation. 

I encourage the authors to revise the draft based on the reviewers’ feedback and resubmit to a different venue.

Decision: Reject