Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The proposed work uses a concept similar to adversarial training to improve robustness against image corruptions. The authors consider different basis perturbations which can account for different complex perturbations and vary their parameters during the min-max procedure to perform adversarial training.  Empirical results are shown on multiple datasets and different tasks such as classification, detection. 
main_review: Strengths:

— The proposed method is intuitively well motivated and empirical results are shown on multiple datasets and tasks to show the effectiveness of the approach.

— The idea of breaking down complex perturbations into different basis to simplify the problem is interesting. 

— The method is shown to perform better against unseen corruptions as well.

Weaknesses:

— Due the min-max nature of the training, the method seems to be computationally expensive.

— The presentation of the paper can be significantly improved. 

1. Precise definitions of MMAI and AMAI would have made the metrics easier to understand.
2. A formal description of the second stage which relies on the frequency component is missing from section A.5 in the appendix. 
3. Fig 6 from A.4 can be presented differently. It is difficult to distinguish between different curves.

— Other baselines such as the Data Augmentation and Adversarial training considered in Table 1, are not considered for the detection task.

— It is mentioned that 5 different levels of perturbations are applied for each type of corruption and other levels decreased performance. An experiment which illustrates this effect and explanation for this would improve understanding. 

Additional comments:

— Looking at Figure 7 in Appendix A.7, it is difficult to understand where the saliency map of the baseline model is focussed on. It also seems that the proposed approach is focussing on the center potion of the image. Other qualitative examples might be helpful here.

— An experiment where all the generated datasets are used in training, rather than choosing a dataset based on MA, serves as a baseline. It would be interesting to see if the improvement for the proposed approach is due to the additional data or min max procedure. 

— An experiment where different basis perturbations are considered during training( more or less in number, different subset) serves as an ablation and would motivate the use of the selected perturbations. 
limitations_and_societal_impact: Limitations and societal impact are addressed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces an adversarial training framework to help autonomous driving classification systems defend against image corruptions caused by internal and external factors. Motivated by the complex application scenario of auto-driving, the authors study the influence of low-quality images on auto-driving tasks. The authors employ the adversarial training concept of adversarial attack to stimulate image distributions in real-world image corruption to better train auto-driving models. 
main_review: This paper proposes an effective gradient-free adversarial training framework for improving the robustness of autonomous driving. The framework will generate a perturbed dataset to stimulate the real-world image corruption distribution and better train auto-driving models. The experimental results show that the proposed method can boost both the robustness and accuracy of the learning to steer models. 

Moreover, the authors apply FID to measure the distribution gap between the generated datasets and the real-world images.  Therefore, the authors also propose a comprehensive robustness evaluation standard for learning to steer models. 

Good things of this paper: 

(a)This paper is well-organized and easy to follow. The pipeline of figure 1 is helpful to understand the proposed research method. 

(b) The proposed method has been clearly stated and is easy to be implemented. The authors will also release their codes. 

(c) Extensive experiments demonstrate the effectiveness of the proposed methods. The improvement is significant. 

Questions of this paper: 

(a) The proposed method can be robust to the unseen image corruptions(snow, fog and frost). However, will the proposed training framework also be robust to gradient-based adversarial examples? If no, can the proposed framework add gradient-based adversarial examples for robust training? 

(b)In algorithm1, the finding of p is the augment to minimize the MA; how to select p without gradient information? 


limitations_and_societal_impact: The limitations are listed in the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary:  In this method, a new augmentation method is proposed to boost the robustness of the supervised “learning-to-steer” and classification tasks for autonomous driving. The proposed approach is based on finding a combination of common perturbations that minimizes the prediction accuracy, and the model is trained in an adversarial manner. The method was evaluated on the driving datasets (Honda, Audi and SullyChen) and CIFAR-100.
main_review: The paper is generally difficult to read and needs to be restructured (e.g., L185-193 should be in the experiments section). Some abbreviations and names are used without a clear definition or explanation (e.g. it is not obvious to the non-specialist in autonomous driving what “learning-to-steer” or “steering” is in this context). 

“Supervised Contrastive Learning”, Khosla et al. is a very relevant paper that should be included in the related work section.

It is not clear and never discussed in the paper whether the FID is computed using the target model that is currently being trained or a fixed pretrained model. If the latter is true, this approach would always result in a single worst-case composite perturbation. 
The overall description of the proposed method in section 4.3 is very confusing and lacks structure. For example, L200-201 state that in stage 2, the model is trained only on the clean data; then later in L204-207, it says that in the same stage 2, the model is trained in an adversarial fashion, and both the clean and perturbed data are used. The description in the Algorithm 1 is too vague: what exactly does it mean: “Conduct sensitivity analysis and discretize the parameters of n factors into their corresponding levels”? Or similarly, “Generate new datasets for each factor with the discretized values from the base dataset”? A reference to a section or equation that clearly describes what is done in this stage is needed.  An exact description of how exactly the new dataset is generated based on the sensitivity analysis is not provided in the paper.

The experimental setup is satisfactory and convincing. An appropriate set of metrics and baseline methods is used to support the main claim of the paper. The empirical results on all driving datasets as well as on CIFAR-100 indicate the efficiency of the proposed approach. 

The new metrics proposed in L 279-289, seem to be redundant without a supporting analysis or empirical evidence of their correctness for the evaluation of robustness and generalization. 

limitations_and_societal_impact: Some limitations, such as the need to store the augmented data, are included. However, no practical limitations concerning the speed and efficiency of the proposed method are discussed. Authors claim that their method with composite augmentations requires less training examples as opposed to the methods that use single perturbations. However, this has not been shown experimentally. Ideally, a plot with convergence speed of the proposed vs baseline methods should be included in the paper. Also, an additional preprocessing stage is introduced in the paper, as well as the adversarial training stage, which might be time-consuming, and how much it slows down the training compared to other methods must be shown.
review_title: null

Meta-review: This paper proposes a new approach for adversarial training. The idea seems novel and interesting and the result seems to be promising. The paper presentation and organization should be further improved. The meta reviewer recommends acceptance of this paper.
Decision: Accept (Poster)