Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes two modified methods for adversarial training to improve robustness. The noise being added is constrained to low dimensional subspace, and it further changes to more delicate constraints.
strengths_and_weaknesses: Strength: the paper proposed a reasonable modification as there might be some subspace or clusters of pixels that are more irrelevant, so that they could be treated differently in adversarial training. 

It also includes a theory analysis with simple example.

Figure 3 provides an intuitive explanation.

weakness: The adversarial training overall seem to be very unstable, whether its sample-wise in figure1, or on OfficeHome in table 2, and in figure 4. The rank of the perturbation doesn't show a reasonable correlation with test accuracy.

The theory analysis is based on a simple binary classification, which might be intuitive. But the theorem proved here, I cannot find the role of this subspace of rank K here, which should be important.
questions: In line 245, if theorem 2 of [2] is so important, it should be explicitly written here. If the theoretical result is better than [2], then it should be specified what changes make the result better, i.e. subspace constraints, or the improvement in technical details to make inequalities tighter.

the paragraph from 202, if this is going to be a concrete example with "In the grass/desert domain, the cow/camel class predominates" specified, it might be better based on an image example showed in the paper, or not to be specified at all. I feel these examples come out of nowhere and leave for nowhere.
limitations: yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this submission, the authors proposed to focus on the problem of domain generalization based on adversarial training. Unlike existing adversarial training methods which inject sample-wise adversarial noise, a novel universal adversarial perturbation generating mechanism (with two variant) was proposed in this paper. Experimental results show the effectiveness of the proposed MAT and LDAT on domainbed benchmark. 
strengths_and_weaknesses: post rebuttal

My concerns have been addressed. 


Strength:

-	The proposed method is quite novel and interesting. 

-	The motivation and analysis of the proposed method is strong. 

-	Experimental results on benchmark justify the effectiveness of the proposed method.

Weakness:

-	My main concern is on the evaluation part. It seems that except Colored MNIST (which is quite toy), the improvement on other datasets are very limited. Moreover, the only baseline using data augmentation for comparison are conventional AT and Mixup, other adversarial augmentation for DG (e.g., GUT [Volpi, NIPS’18]) and conventional data augmentation methods (which have been shown to be quite effective [French, ICLR’18], [Tian, TPAMI’22]) should be discussed.

-	 In Fig. 4, it seems that the number of pertubations adopted between MAT and LDAT is not consistent. I am wondering if there are some guidance to choose the optimal number given source domains and model?


questions: Please see above. 

limitations: While the limitation of existing adversarial training has been discussed, the limitation of the proposed method is missed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces two methods for universal adversarial attack (as opposed to sample-wise) in order to improve the OOD-generalization of image classification CNN models. It also shows theoretically how spurious feature use is reduced through UAT as well as showing empirically how these UAT methods perform well on a variety of benchmarks.
strengths_and_weaknesses: Strengths:

The proposed method is novel, interesting, and relatively simple.

The paper is generally well written and easy to follow. I think the theoretical analysis is of interest to the broader community on how to apply UAT style training.

The experimental setup is comprehensive.


Weaknesses:

Unfortunately, the empirical results do not show a sufficient improvement over other baseline methods. The inclusion of colored mnist seriously alters the average, which is to be expected as the proposed method directly attacks only the color channels. When k=20, that's (in some sense) 20 different colors to be applied to each image. I do not find the colored mnist dataset to be generally insightful, therefore when I look at the other datasets, there is not a consistent improvement over other baselines. Normal AT, ERM, MAT and LDAT each get fairly good results (and often not outside the error margin). It is not clear to me when I should use MAT, LDAT or AT.

Minor:
Figure 2 has some typos (LRAT), and is a bit hard to parse. Having the blue arrow represent the entire NxN image is not intuitive. Perhaps turning the flat squares into 3d shapes would help.


Post rebuttal:
Thanks authors. While I would preferred LDAT or MAT to have a clear choice when to use one or the other, the rebuttal is sufficient reasoning. Overall, I agree with reviewer VkPQ, I have upgraded my score from 3 to 6.
questions: I am a bit confused as to how the parameters in the MAT/LDAT matrices are selected. Supplement algorithms 1 and 2 seem quite crucial to the function of the proposed method, and at least one of them should be in the main text. Could you give me a plaintext overview for how one of the algorithms works?
limitations: The authors do not mention potential negative impacts. A small paragraph could be included to describe how classifiers can be misused.

The weaknesses described above are not particularly referenced either.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper highlights the severe performance degradation of sample-wise AT  in OOD tasks due to the existence of much larger perturbations in OOD tasks. And the paper shows a theoretical perspective to the stable generalization performance of universal AT, discovering that low-dimensional domain-wise perturbation benefits removing the low-rank structured background information. Inspired by this, the paper proposes two specific constructions of low-rank structured perturbations along the sample number and input scale dimension respectively, and theoretically proves their enhancements to the OOD robustness using a simple OOD task. To verify the approaches, experiments on five benchmark datasets are conducted to show the OOD performance of proposed MAT and LDAT compared to existing algorithms.
strengths_and_weaknesses: Strengths:

1.  The paper provides an empirical insight into the limitations of sample-wise AT on OOD tasks, and highlights the importance of domain-wise AT. Then, two effective low-rank based domain-wise AT methods are proposed. Superior performance is observed on the OOD generalization ability comparing to baselines.

2.  The paper reveals some relationship between adversarial training and OOD generalization problem theoretically, allowing for deeper dive into the AT theory.



Weaknesses:


1. The theoretical results that reveal the low-rank perturbation can prevent the model from learning spurious features is obscure. It is better to show the results in a more friendly way. 

2. Some experimental results are not convincing. For instance, in Table 2, if not considering the colored MNIST datasets, the performance of the proposed model is on par with ERM. Without considering colored MNIST, the averaged gain is very limited. 
questions: nil
limitations: nil
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to utilize adversarial training to improve the test accuracy on the OOD dataset. They present the low-rank perturbations to perform adversarial training, inspired by the  better performance of universal adversarial training. Experiments on benchmark datasets veriy the effectiveness.
strengths_and_weaknesses: Strengths:
1. Adversarial attack for good is an interesting topic. In this paper, it utilizes the adversarial training to improve test accuracy of OOD dataset. I think this topic should be encouraged. 
2. The paper gives an empirical and theoretical demonstration for the proposed method. I think it is good
3. Experiments compared with many SOTA methods verify the better performance. 

Weakness:
1. My main concern lies in the assumption that because the OOD dataset (or universal perturbation) usually shows low-rank structure, inspired by this, the authors propose the low-rank adversarial perturbations to perform attacks. Actually, in the real images, many scenes are not low-rank. Therefore, I don't know whether the proposed method still works in this situation. Authors should give some examples and discuss it. 

questions: The aussmption of low-rank structure should be detailed discussed in the rebuttal. 
limitations: The aussmption of low-rank structure should be detailed discussed in the rebuttal.
review_title: null

Meta-review: metareview: The authors have provided a strong rebuttal addressing many of the concerns brought up by reviewers. Two of the reviewers have in response increased their scores significantly. At the end of the process most of the reviewers have converged in favour of acceptance.
confidence: Less certain
Decision: Accept