Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of building robust models by only enhancing a part of the whole model (subnetwork). The paper is mostly theoretical and propose a new concept of semi-robust. The authors also provide empirical evidences to support their claim.
strength_and_weaknesses: # Strength
By only enhancing a part of the whole network, the proposed approach can save the complexity of sota defenses such as adversarial training.

# Weakness
1. This paper does not illustrate their idea clear enough. Many expressions are informal and confusing. For instance, how do you define "high correlated layers" 
2. The theoretical findings seems not enough to support the main claim. Especially the bound of parameter gamma 1 to j is not tight enough to guarantee the meaning of robust.
3. The authors try to use empirical evidences to illustrate the correctness of their method. Yet, there are many issues in the experiments.  The setup is not very clear. What od *acc and ~acc represent? What is the perturbation radius. What kind of attacks have you chose? Usually the robust accuracy should be around 60%, why your score is so high?
4. The authors claim that if the subnetwork is robust, then the whole model is robust. Then please explain, if I add a network layer before a trained model such as AlexNet, and the layer do nothing but pass the value through to the network, this layer is apparently robust, how can you prove the model will not be attacked by the adversarial example.  
clarity,_quality,_novelty_and_reproducibility: This paper is not very clear. The quality and novelty is somewhat reasonable. The reproducibility is poor.
summary_of_the_review: As in Strength vs Weakness
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new concept, semirobusness, and a corresponding framework to study if the (semi-)robustness of a subnetwork can indicate the (semi-)robustness of the rest or the all network. The major contribution is a set of theorems to show that under assumptions related to the mutual information between representations output by one layer and the other, the proposed semirobsutness will be carried out to deeper layers (and maybe with a different level). 

strength_and_weaknesses: ### Strength
This paper studies an interesting question of whether the robustness of part of the network can imply the rest (or the whole). Towards this end, it proposes this notion of semi-robustness, which is essentially that when the bottom part of the network is fixed, if there exists a top network such that the combination of the bottom and the top produces robust prediction, measured with the correlation of prediction and the label. Although with the dense theorems in the main paper I am not able to examine the proof, I think the theoretical results intuitively make sense. 

### Weaknesses
My major concerns for this work are (1) if the contribution is over-claimed; and (2) experiment results may overfit to the test set and $\epsilon$ is missing from the paper. 

**Semi-robustness Guarantee**. The paper claims that “For the first time we provide a theoretical framework and prove that under some assumptions if the first part of the network is semirobust then the second part of the network’s robustness is guaranteed”. My understanding of this claim is that for a network $F = F^b(F^a(x))$, if $F^a$ satisfies some assumptions and is semi-robust, then we know something about the robustness of $F$ or $F^b$. However, my observation is that the activation of the first layer of $F^b$ is also part of the assumptions A1 and A2 (please correct me if this is not true). Therefore, the assumption is not only about the bottom network $F^a$ and the current contribution statement sounds like it only requires additional information about $F^a$. Therefore, the current contribution seems to be over-claimed to me. 

**Overfitting in the Algorithm 1.** This is my biggest concern in the experiment. The paper directly tries to learn a top network $F^b$ to match the **test robust accuracy** of an adversarially trained network. This is different from matching the robustness of a traditional adversarial training algorithm as you are leaking test information to $F^b$ while the standard algorithm does not. Can you provide the robust accuracy of the partially-trained network on another validation set? I am concerned whether more epochs used in algorithm 1 brings a larger generalization gap. 

**Missing Setup.** I found the experiment part also hard to follow as there is a lot of setup missing. What is the size of the $\epsilon$-ball used by the attacker? How many epochs, the values of $\lambda$ and other hyper-parameters in Algorithm 1 are used in your experiment. What is the absolute robust accuracy for experiments and datasets mentioned in Figure 2? How many steps do you take for each attack? What did you do for the FGSM attack and how do you adapt that to the $\ell_2$ case as in the $\ell_2$ space we do not take the sign of the gradients. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity 
The theoretical part of this paper is a bit dense and hard to follow. More high-level descriptions, motivations and smoother transitions between theorems should help to improve the readability. The technical part is very unclear to me as a lot of key parameters in the setup are missing. 

### Quality
The fact that the semi-robustness propagates through a subnetwork to the rest is not that surprising to me as the other half of the network is still a universal function approximator so I believe one can learn a robust top network given the bottom ($G$ always exists). Regardless, the contribution to finding the expression of $\gamma$ for the top network is solid. The quality of the empirical evaluation is not below the standard and reasons are mentioned in the previous review box.

### Novelty
This paper studies an interesting question and brings a novel setup for the robustness community. 

### Responsibility 
I did not find paragraphs or statements on how to reproduce the experiments. Important parameters are missing. I also did not see the code of the paper. I think a lot of improvement needs to be made on the axis of reproducibility. 

summary_of_the_review: For the current moment I incline to reject. I think the theoretical contribution may be over-claimed (but I am not sure about this). I will not exclude the experiment part from the major contribution of this paper as the authors indeed mention the empirical results in their contribution statement. However, the quality of the experiments is not ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new notion called the semirobustness of subnetworks. Then, the authors state how subnetwork semirobustness can be extended for the semirobustness of a bigger subnetwork under some condition that relates to the semirobust subnetwork and the rest part. Further theoretical analysis and empirical verification of the statement are provided.
strength_and_weaknesses: [[Strengths]]
1. The concept of semirobustness of a subnetwork is novel and has the potential for practical applications.
2. The proofs look correct, and the experiments well support the theoretical statements.
3. The experiment used three different models and four attacks, which looks enough for experimental verification.

[[Weaknesses]]
1. I suggest the authors recheck the formal part carefully. For example, the definitions of $g_\delta$ and $h_{\delta,j}$ in Assumptions should be based on subnetworks rather than layers because the input is taken from the input space.
2. The notion of semirobustness looks valuable, but I’m not fully convinced of its practical usage. The theoretical part contains proof, and the experimental results support the theory. However, what would be the practical usage of the concept? Could the authors add a corollary relating the robustness of the entire network to the semirobustness of its subnetworks? (For example, the authors may propose a bound to the robustness of a network containing subnetwork that is NOT semirobust?) Such a statement would indicate some potential applications in robust training, e.g., robustly stacking more layers to existing robust models, inductively training robust subnetworks, etc. However, how would such applications be more useful than just training a robust network without the knowledge about semirobustness?
3. In the experiment, the authors reported the accuracy of $f_b^* \circ f_a^*$ (robust network) and the best accuracy of $\tilde f_b\circ f_a^*$ from training the last half of the network with adversarial training. While this is enough to show the theoretical statement (because semirobustness only requires the existence of mapping function $G_j$ that is approximated by $\tilde f_b$), it would be interesting to compare the accuracy of $f_b\circ f_a$ and that of $f_b\circ f_a^*$ (before the additional training of $f_b$) to check the effect of the robust first half in classification. Could the authors add those accuracies to Table1, or justify some reason why such accuracies are not interesting?
4. The authors did not provide enough detaabout the experimental setup. For example, what are hyperparameters used for the adversarial training (both for the entire network and $\tilde f_b$)? How many epochs did the authors train $\tilde f_b$? These details are important for general readers to reproduce the result, and this raises a reproducibility issue.
5. The discussion in the Conclusion looks like the conclusion from the experiments. Move this discussion to the experiment section and add the conclusion of the entire paper in Conclusion. (If the page limit becomes a problem, I’d suggest the authors move the further discussion from the experiment to the Appendix.)

clarity,_quality,_novelty_and_reproducibility: The authors’ contributions look novel, and the writing is straightforward and understandable. There might be a few reproducibility issues.
summary_of_the_review: I consider the paper novel and valuable work. First of all, a theoretical finding is rare and precious in adversarial machine learning. The semirobustness concept looks to be precious to the field as it opens up new possible training frameworks based on semirobustness. The idea was tested on various models using several attacks, so the experimental support looks to be strong enough.

Regarding some negative factors, the paper did not show enough practical connection between their concept to the field of robust training. Though it is possible to imagine that this concept would yield frameworks for constructing a robust network by stacking semirobust subnetworks, showing such usage is also a part of the authors’ responsibility. Also, details about the experimental setup are missing, and this raises a reproducibility concern. However, I believe that these are amendable issues, and I’m willing to raise the score after they are fixed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper develops the concept of semirobustness, which indicates the adversarial robustness of a part of the network. The authors prove that if a subnetwork is robust and highly correlated with the rest of the network, then the remaining layers are also guaranteed to be robust. Empirical evaluations are done on CIFAR-10/100 and ImageNet.
strength_and_weaknesses: Strength:
- The idea of semirobustness is somewhat interesting to me and is novel as far as I know;

- The theoretical analyses and conclusions are intuitively reasonable (although I did not carefully check the proofs);

Weaknesses:
- Lack of details in the claims of Theorem (see Clarity, Quality, Novelty And Reproducibility);

- The attacks (e.g.,  FGSM, I-FGSM, PGD, CW) used for evaluation are weak. Stronger attacks like AutoAttack should be evaluated;

- The robust accuracy in Table 1 is quite high, which is partly because the authors apply $\ell_{2}$-norm perturbation (there is also no information on the value of perturbation budget $\epsilon$);

- It is surprising that most of the references cited in this paper are before 2020. This makes me wonder if there is any related work during 2020-2022?
clarity,_quality,_novelty_and_reproducibility: - In Theorem 1, there is no definition on what is semirobust for a single layer (Definition 1 only provides definition for subnetworks, with input $X+\delta$);

- In Theorem 2, there is not clear the relation between $\gamma_{a}$ and $\gamma_{b}$, as well as the relation between $\gamma_{b}$ and $\rho_{j}$ and $U_{j}$ in Assumption A1/A2;

- All the citation formats are \citet{}, which is not a common practice.
summary_of_the_review: While the idea of semirobustness is interesting, the empirical evaluation is the main drawback, and there require more accurate details in the theoretical analyses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes half adversarial robustness, or semi-robusness as a new concept, where the idea is to investigate whether a robust part of the network, i.e. the top of the network, will imply robust behavior of the entire network. The paper contributes a set of theorems and proofs that are interesting. They provide conditions under which semi-robustness would lead to robustness. 

The reviews and AC agree that 
+ The proposed concept is interesting and can also be intuitively followed (even though the presentation in the paper is very dense).
+ The contributed proofs also strengthen the paper.
+ The revised manuscript clarified many of the theoretical concerns the initial version had raised.

Yet, there was also a long and detailed review discussion where the weaknesses have been discussed. Most importantly, there are consistent concerns of three reviewers regarding the experimental evaluation of the proposed method with e.g. non-standard choices for adversarial attack epsilons and unclear/inconsitent results. 
Therefore, although there is an interesting concept proposed in the paper, we recommend to carefully revise the paper before submitting to another conference.

justification_for_why_not_higher_score: There are consistent concerns of three reviewers regarding the experimental evaluation of the proposed method with e.g. non-standard choices for adversarial attack epsilons and unclear/inconsitent results.
Decision: Reject