Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a model collaboration scheme to tackle the insufficient model capacity against adversarial examples instead of model ensemble. Through replacing voting-based strategy with selecting the best-performing sub-model, each sub-model only fits its specific adversarial areas, which enables the models with limited capacity to achieve better adversarial robustness. An auxiliary head which outputs the confidence is introduced to identify the best-performing sub-models. Experiments on CIFAR-10 and ResNet-20 demonstrate the effectiveness of proposed algorithm.
strengths_and_weaknesses: Pros:

* This work is well-organized with clear motivation and generally easy to read.
* The authors provide the source code with instructions, which makes the algorithm easy to follow.
* The designing of collaboration scheme seems natural and reasonable with its corresponding theoretical analysis.
* The comparison with other defense baselines on CIFAR-10 demonstrate the superiority of proposed algorithm.

Cons:

* The motivation of this paper is that each sub-model can handle its corresponding adversarial subspace via collaboration scheme so that the adversarial robustness with limited model capacity can be improved. However, it is difficult to the empirical evidence of it in experimental section. For example, some visualization of adversarial examples with decision boundaries of collaborated models are recommended.
* The performance under different attack settings includes PGD and adaptive attacks is reported. However, some popular stronger attacks are not included for comparison, such as autoattack.

questions: Overall, I think this paper is interesting. I have several suggestions for the authors:

1. Please provide some empirical evidence that sub-model handles its corresponding adversarial subspace via collaboration scheme.
2. Please include some stronger attacks for comparison, such as autoattack.

limitations: The authors mentioned that they have discussed the limitations in the Appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method to improve the adversarial robustness of an ensemble. Instead of using the voting strategy, the proposed method aims to identify the model with the highest confidence on the ground-truth label. In this way, it can improve the model performance when the correct prediction remains with the minority. The authors conduct experiments on both wight-box and black-box attacks to demonstrate the effectiveness of their method.
strengths_and_weaknesses: [Strengths]
+ The authors rethink the classic voting-based strategy and note the problem with the case when correct predictions remain with the minority. Then, they propose a new “collaboration” strategy, which is very novel and interesting.
+ This paper is well-structured and easy to follow.

[Weaknesses]
- The formulation of the evaluator head in the SoE is questionable. In SoE, the evaluator head is designed to estimate the predicted probability $\hat p_y(x)$ through a BCE loss. However, if $g_{\phi}$ has successfully fitted the probability $\hat p_y(x)$, then we can directly identify which dimension in $\hat p(x)$ is the same as $g_{\phi}$, thereby obtaining the ground-truth label. Therefore, the ensemble seems useless.
- To learn the collaboration between sub-models, why not directly construct a one-hot label for the best-performing sub-model on each input, and use the cross-entropy to learn the probability of each sub-model being the best-performing model?
- The utility of the proposed method in minimizing the vulnerability overlap of all sub-models is not directly proven or verified. The main purpose of the proposed SoE is to minimize the vulnerability overlap of all sub-models. However, its effectiveness is not directly demonstrated. I suggest the authors provide some visual demonstrations or quantitative results to compare the vulnerability lap between SoE and other methods.
- In Algorithm 1, a surrogate loss is used to approximate the loss on the best-performing sub-model. Why not directly use the loss of the sub-model with the largest confidence $g_{\phi}(x)$?
- Algorithm 2 is somewhat confusing. The adversarial example is generated to worsen $\hat{p}(x)$, which is the output of the sub-model with the highest confidence. Therefore, this adversarial example is generated only based on the best-performing sub-model. Actually, such adversarial examples have been used in Algorithm 1, so it is confusing why we need Algorithm 2.
- The attacking methods used in experiments are not sufficient. First, the proposed method should be at least compared with the PGD/C&W attack on the best-performing sub-model. Second, the attack that simultaneously destroys the estimator output and the predictor head should be considered. Third, the adaptive attack used in the paper is confusing. There should be more discussions about $l_2$ and the selection of $j$. Besides, in $l_2$, $y$ is usually not known by the attacker, so it is unclear how to compute $l_2$.

questions: - Given that $g_{\phi}$ and $f_{\theta}$ are learned alternatively, how stable is the learning?
- Which loss function is used in Table 1 and Figure 4? Why does the robustness increase when the weight $\lambda$ increases? Why is the robustness higher when $\epsilon=0.07$ than $\epsilon=0.06$?
- Have you measured the robustness when the ensemble contained different numbers of sub-models? When using SoE, can you use fewer sub-models to achieve higher robustness than previous voting-based methods?

limitations: The authors do not address the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an adversarial defense method that can better utilize the ensemble of multiple experts. In contrast to previous ensembling methods that require more than half methods give the correct prediction, for each sample, the proposed method employs a router module that routes the sample to the best model for prediction. In this way, the proposed method can yield the correct result if the selected model can predict right. The proposed method demonstrates better performance compared to previous ensemble methods.
strengths_and_weaknesses: Strengths:
1. The writing is good and easy to follow
2. There are some theoretical proofs

Weaknesses:
My main concern lies in the fairness of evaluation. For white-box attacks, I don't think the non-adaptive SoE can be a fair comparison: all adversarial methods assume the visibility of the whole model, which should include the E head. However, in non-adaptive SoE, E head is not visible. 
The author also gives a setting with an adaptive adversarial attack that can attack the E head, which is fairer than the previous setting. However, I don't think the adaptive setting is fair enough. For example, besides attacking the best model to be less confident, all worse models should also be encouraged to predict higher confidence. I think this adversarial setting can be more challenging given the E head of worse models can be easier to be fooled for this image (since its backbone is easier to be fooled). 
This example is one evaluation method. If the author can give a more systematic evaluation setting besides this, I would be more convincing.

Also, what's the adversarial sample employed in transfer adversarial attack? Is it adaptive or non-adaptive?

Some minor typos:
1. Algorithm 2, "duel" heads should be "dual" heads
questions: A more careful and convincing white attack method should be employed to evaluate the proposed method.
limitations: No social negative impact is found.
review_title: null

Meta-review: metareview: This paper proposes an ensemble-type solution for improving the adversarial robustness of a model. The proposed idea is simple, yet novel with theoretical supports. The authors did a good job clarifying reviewers' concerns and all reviewers finally recommend acceptance. AC also thinks that this is a good paper in various aspects (novel idea, good write-up, solid theoretical supports) and has a potential to be a generic ensemble-type solution even in non-adversarial/standard setups, e.g., see [1]. Furthermore, the confidence prediction can be used for other purposes, e.g., out-of-distribution detection [2] and active learning [3]. It is useful for readers to discuss about these extensions.

[1] Confident Multiple Choice Learning, Kimin Lee et al., ICML 2017.

[2] Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers, Apoorv Vyas et al., ECCV 2018.

[3] Learning Loss for Active Learning, Donggeun Yoo and In So Kweon, CVPR 2019.
confidence: Certain
Decision: Accept