Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a two-stage defense method to improve the adversarial robustness over different perturbation types. Specifically, it first builds a hierarchical binary classifier to differentiable the perturbation types and then uses the result to guide to its corresponding defense models.  It first proves the different types of perturbations could be separable and the adversary could be weakened to fool the binary classifier. It shows their methods achieve a clear improvement in the experiments.

Pros:
1. The paper is good-written and easy to follow.
2. The proposed idea is interesting.
3. The experiment is detailed and comprehensive.

Cons:
1. There is a major problem in their method. In the last sentence of section 5.2, it says uses the soft relaxation only in generating the adversarial example, but not for inference. It clearly caused the gradient making problem in the adversarial attack later on to test the robustness.  The gradient is blocked before reaching into the binary classifier so that the adversarial attack fails, which I think it is not truly improving the model's robustness.
2. In my opinion, this method is just a dynamic voting based model ensemble. Just take the binary classifier as a voting procedure.  Therefore, the traditional adversarial attacks won't work in general. I would suggest using the soft relaxation in the inference as well for the adversarial attack. 
3. Also, the assumption that different norm adversarial examples could be clearly separable might be wrong. You could find the adversarial examples that satisfy both l1, l2 and l_inf constraint by just choosing the \epsilon for every norm differently.

review_title: The good results may be just brought by inadequate attack evaluation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an ensemble approach to deal with multiple perturbation types. The underlying idea is to train a robust classifier for each perturbation type (i.e., l1, l2, and l-inf) and choose a model to predict based on the decision of a perturbation classifier which is trained to distinguish perturbation types.
The idea is interesting, and the experiments seem solid; however, I am hesitating to give this paper an acceptance decision due to some unclear points as follows:
1.	I have not checked the proof of Theorem 1, but it seems a bit counter-intuitive to me. The reason is as follows. Let’s assume that we are attacking a clean image x. We start from x0 inside both l1 and l-inf balls for example. We further assume that during the process of updating x{t}, we never do any projection onto any ball (i.e., x{t} always lie inside the balls). If so, there is not any difference between x_adv w.r.t l1 and l_inf, meaning that there are possibly a certain number of adversarial examples shared across l1 and l-inf attacks.
2.	In your Theorem 1 (also your experiments), the radius of l1 is eps and that of l-inf is eps/sqrt(d), meaning that the ball of l-inf is a subset of the ball of l1. However, in Section 5.3, you claimed that “The dotted line shows the decision boundary for the perturbation classifier C_adv, which correctly classifies inputs subjected to large ‘1 perturbations δ00 as ‘1 attacks (green), but can misclassify samples with smaller perturbations”. It seems that you reckon l-inf adversarial examples are having more perturbation than l1, do not you?
3.	Theorem 2 is not understandable to me. What is worst-case adversary (this needs to be explained and defined because we can understand this notion in several different ways)? What is delta?
4.	I recommend testing Projector against the attack over a uniform average of Mp in addition to what is doing.
5.	It is encouraging to compare your proposed method against (Francesco Croce and Matthias Hein, 2020 a). You did mention this work, but not compare it in experiments.

review_title: The paper looks good but some points are unclear to me

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposed a pipeline method which first classify the attack type and then choose the proper predictor for that type. The authors provide both theoretical and experimental proof to support their pipeline method. 

Pros:

(1) The idea is good and reasonable. Also, the paper is well written and easy to read. The proof part is clear and natural. 
(2) The result of theorem 1 is interesting and it proved the different perturbation types (under this problem setting) is separable with a high probability. Although the setting is a little restrictive, it is still very impressive.  

Cons:
(1) The method and analysis only apply to the Lp attack type. It will be good if it can be extended. 
(2) Assume that we treat the two pipelines as the whole process (end-end deep network), e.g., the first layers determines the type and then that type will determine which part to activate for training the data points, that is, you have a network with special topology and structure. If the attacker only cares about this input and output of this network (no pipeline here), we should have an attack strategy. How to explain this view using your analysis ? Could we say this is equivalent  to changing the structure of the network  ? From this perspective, could you tell the difference between these two views ?



review_title: A good pipeline idea to improve robustness

final_score: rating: 4: Ok but not good enough - rejection
review_content: In theoretical analysis, the authors conclude with an Gaussian assumption on the data distribution. 

This assumption is very restrictive, and the corresponding conclusion does not provide a general view of what is happening for real datasets. Gaussian condition on dataset is very unrealistic, and its theoretical analysis should only be considered as dealing with a toy model. 

Authors should remove this Gaussian assumption, or at least give strong reasons in plain natural language why Gaussianity can represent natural datasets in the adversarial study.  
review_title: assumptions on distribution unrealistic

Meta-review: The paper proposes a model to defend against multiple lp norm attacks by classifying those attacks. The reviewers raised several concerns about the methodologies. Furthermore, it's not clear how the proposed algorithm can deal with an unseen attack (e.g., only trained on l1, l_infty attacks but encounter l2 attack in the testing phase). The assumption that the attack types are known beforehand is restricted.  
Decision: Reject