Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a gradient-free adversarial training technique called AutoJoin, which attaches a denoising autoencoder to the original regression model. It shows superior performance compared with the baselines on the benchmarks. The joint learning of steering and denoising reinforce each other.
strength_and_weaknesses: ## Strength

1. This method is simple but demonstrates huge performance improvement.
1. This paper is well-written and easy to understand, especially with algorithm 1.

## Weakness

1. The novelty is limited. The introduction of a denoising autoencoder for adversarial robustness is already introduced in [1][2].

[1] Feature Denoising for Improving Adversarial Robustness
[2] Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser
clarity,_quality,_novelty_and_reproducibility: The method is simple and can be easily reproduced.
summary_of_the_review: Though the performance improvement is huge, the novelty of this paper is limited, which doesn't meet the ICLR standard.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is on the topic of  robust maneuvering in view of autonomous driving. The authors propose to augment the training images with gradient free perturbations. Furthermore, they propose to regularize the model training by adding a denoising task where a decoder should reconstruct the original images without the gradient free perturbations.
strength_and_weaknesses: Strength:
The paper is clear (Figure 1 resumes quite well the proposed method) and proposes an extensive set of experiments to support their claim: experiments on several datasets, an ablation study to show that both gradient free perturbations and the denoising auto-encoder regularization improves performance.

Weaknesses:
1) The authors should more clearly state the differences with Shen et al., 2021 which also propose gradient free perturbations but no denoising auto-encoders.
2) Maybe adding data augmentations such as RandAugment and adversarial perturbations methods like AdvProp could further support the experiments.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and somewhat novel with the refining of the perturbations and the auto-encoder regularization.
summary_of_the_review: The paper is clear and well supported with experiments, it reaches SOTA results but the only clear novelty seems to be the denoising auto-encoder as regularization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author introduces a new efficient adversarial training method, AutoJoin, to efficiently produce robust models for imaged-based maneuvering. Compared with Shen, the method adds a new denoising autoencoder to reconstruction the image so as to remove the noise added to the input image. The method achieves better empirical robustness and efficiency than baselines.
strength_and_weaknesses: Strength

1. The proposed method, AutoJoin, achieves better robustness and efficiency than baseliens.

2. The experiments and the ablation study are intensive, demonstraing the effectivenss the of proposed method.

Weakness

1. The result about the gradient-based adversarial examples in Table 8 is not complete. If the claim of the paper contains the adversarial robustness of this kind of adversarial examples, AutoAttack [1] should be applied to test the robustness in order to remove the effect of gradient masking.

2. As one of the paper's focus is the efficiency, it is better to list all the details about the training time in a table in the Appendix. Besides, I wonder whether the improvment of the efficiency comes from using less training data. If so, will applying the random select of the training data in line 4 of Alg 1 to Shen reduces as much as training time as AutoJoin?

3. The contribution of the paper should be highlighted. For example, which components are proposed in previous papers and which ones are newly proposed?

Minors

1. It would be better for the authors to add description about Single, Combined and Unseen in the main text.
clarity,_quality,_novelty_and_reproducibility: The paper needs some modification of writing to highlight the contribution. I think the algorithm is a little complicated with many augmentations. The authors need to release their code to make the result reproducible. 
summary_of_the_review: The proposed method, AutoJoin, achieves better robustness and efficiency than previous baselines. Some clarification and more experiments are needed. 

[1] Francesco Croce et.al., Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, ICML 2020
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of robustness against gradient-free perturbations. It formulates AutoJoin, which is a gradient-free technique consisting of two steps:
* Perturb the images using a perturbation from the predetermined perturbation set at a sampled intensity level and obtain the latent representations through the encoder, 
* The encoder representations are then passed to the decoder to jointly learn the denoising autoencoder (DAE) and regression model for steering angle prediction.

The proposed method is tested against gradient-free and gradient-based attacks, improving performance over prior methods while decreasing the per-epoch training time significantly.

strength_and_weaknesses: ## Strengths
 
* **Originality and Quality**
    * The paper focuses on an important problem of formulating efficient gradient-free adversarial training methods for robust maneuvering against image perturbations. This is the first step in that direction and would interest the community.
    * The proposed method is novel and simple, improving both the performance and computational cost.
    * The experiments cover benchmark datasets and compare them with state-of-the-art methods for gradient-free attacks.
    * The evaluation also spans gradient-based attacks, which strengthens the experimental results.

* **Clarity** Overall, the paper is well-written and easy to read. 

---

## Weaknesses

* **Originality and Quality**
    * **Sanity check for robustness:** While the proposed method is the first attempt for a gradient-free technique, many generative methods have been proposed to defend against gradient-based attacks and have later proven ineffective [1]. Can the authors evaluate the defense against random perturbations with random restarts as sanity check? 
   * **Importance of DAE and missing confidence intervals:** The ablation in subsection 4.2.2 shows that Autojoin obtains marginal better performance with DAE. The paper mentions that it is vital for clean and single MAE; however, without the confidence intervals, it is challenging to analyze the effectiveness of this significant component of the proposed method.
    * **Additional ablation studies:** The ablation Table 7 highlights the importance of RGB and HSV perturbations. Can the authors include the rows with only RGB and HSV perturbations to dissect the gains in the rows where they are combined with Gaussian noise? I also checked the appendix, but it only contains the same settings on different datasets.
    * **Gradient-based attacks:** Despite the paper’s focus on gradient-free attacks, the paper includes an evaluation on gradient-free attacks. However, the evaluation is extremely limited due to two reasons. First, the paper evaluates FGSM and PGD attacks. I’d suggest including the evaluation with AutoAttack – this should strengthen the results significantly. Second, the defenses used for comparison are relatively weak, instead the authors should evaluate with more recent methods (refer RobustBench [2] for the list of recent advances in gradient-based defenses).

* **Clarity**
    * **Redundant text:** The paper contains redundant text in a few places that can be removed to include more important material from the supplementary material. For example, the paragraph below subsection 4.2 is not required, and instead of having the following as subsubsections, I’d suggest including them as subsections directly. Further, the details in the first paragraph in subsection 3,1 can be moved to the appendix.
  * Third last line – Chen et al. in elated work is missing citation. 
   * The paper should include more details on the selection process for $\lambda_1$ and $\lambda_2$, along with an ablation showing their effect on the performance.
* **Reproducibility** The code was not provided with the submission. Since the main contribution of this paper is the empirical analysis of this new strategy, it is essential to provide the code and detailed implementation to evaluate the paper.

---
## References
[1] Tramèr et al. On Adaptive Attacks to Adversarial Example Defenses. NeurIPS 2020.  
[2] Croce et al. RobustBench: a standardized adversarial robustness benchmark, NeurIPS 2021   
clarity,_quality,_novelty_and_reproducibility: The review above elaborates on all these points in detail.
summary_of_the_review: The paper tackles a significant problem; however, analyzing the effectiveness of the proposed components in the current ablation settings is challenging. The evaluation of gradient-based attacks is not the paper's primary focus but could be stronger in its current form. Without the sanity checks/adaptive attacks, I am worried that the proposed method might be effective due to weak attacks. Therefore, my current score is weak reject, but I am happy to increase my rating during the discussion period if the authors address my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper discusses a new method called AutoJoin, which uses a denoising autoencoder to improve adversarial robustness in image-based maneuvering tasks. The method shows improved performance compared to the baselines, but the novelty is limited as denoising autoencoders have been used previously in similar contexts. The paper is well-written and easy to understand, and the method looks relatively simple. However, as pointed out by the reviewers, the paper requires a future revision to incorporate some feedback, such as the results are incomplete (averaged over two training runs and weak attacks for gradient-based attacks, and additional ablation studies on the hyper-parameters and the proposed components would strengthen the paper). At its current form, the paper does not meet the standard for ICLR.
justification_for_why_not_higher_score: With the limitation on novelty and some unaddressed issues regarding experiments, the paper does not meet the ICLR acceptance bar, as also shown in the reviewer ratings.
Decision: Reject