FLOAT:  FAST LEARNABLE ONCE-FOR-ALL ADVERSARIAL TRAINING FOR TUNABLE TRADE-OFF BETWEEN ACCURACY AND ROBUSTNESS

Abstract: Training a model that can be robust against adversarially-perturbed images with-out compromising accuracy on clean-images has proven to be challenging. Recent research has tried to resolve this issue by incorporating an additional layer after each  batch-normalization  layer  in  a  network,  that  implements  feature-wise  linear modulation (FiLM). These extra layers enable in-situ calibration of a trained model, allowing the user to configure the desired priority between robustness and clean-image performance after deployment.  However, these extra layers significantly increase training time, parameter count, and add latency which can prove costly  for  time  or  memory  constrained  applications.   In  this  paper,  we  present Fast Learnable Once-for-all Adversarial Training (FLOAT) which transforms the weight tensors without using extra layers, thereby incurring no significant increase in parameter count, training time, or network latency compared to a standard adversarial training. In particular, we add configurable scaled noise to the weight tensors that enables a ‘continuous’ trade-off between clean and adversarial performance.  Additionally, we extend FLOAT to slimmable neural networks to enable a three-way in-situ trade-off between robustness, accuracy, and complexity.  Extensive experiments show that FLOAT can yield state-of-the-art performance improving both clean and perturbed image classification by up to ∼6.5% and ∼14.5%,  respectively,  while  requiring  up  to 1.47x fewer  parameters with similar hyperparameter settings compared to FiLM-based alternatives.

Reviews and author responses:

This paper argues feature-wise linear modulation (FiLM) methods to perform in-situ calibration are too computationally expensive. Motivated by this, the authors propose the FLOAT method which adds scaled binary noise to the weight tensor instead of using extra layers. This paper also extends FLOAT to balance the 3-way-tradeoff between robustness / accuracy / complexity. The included experiments showcase the improvement in resolving the tradeoff between robustness and accuracy.   
This work is heavily based on the previous work Wang et al. 2020 <Once for all adversarial training>. At first glance, I could not even tell there are fundamental differences between this work and the previous work. Indeed, the improvement is natural but very limited. This severely diminished the novelty of this work. 

This paper contains lots of redundant information e.g. Algo 1 and Algo 2 are mostly similar with minor tweaks, and could very well be condensed. Figure 2 and 5 are redundant, and could be merged as well. 

I do see the improvement from FLOAT over OAT in terms of computation time. In figure 7, the performance of FLOAT seems to be better than OAT and PGD. But Figure 12 in the appendix tells me this is highly dependent on a good pick for lambda, which might not be generalizable. 

The authors tend to abuse some highly theoretical terms in this empirical paper without providing enough background/description such as Pareto Frontier. 
 
Overall, I think the paper is not at the publishable stage, I suggest the authors condense the information and add supporting theoretical thoughts behind the proposed modification.


 Dear Reviewers and AC panel,

Thank you again for your detailed reviews that have helped revise our manuscript. We are glad that the merits of our work have been recognized by reviewers dQEN and Xsuw. We recognize the final decision time is only two days away and we are yet to hear from a few reviewers, particularly reviewer DN3s and BuXN as we tried to address their concerns that we feel can play a key role in better understanding what we targetted to solve in this work. We are summarizing the major points about the paper (post-rebuttal) for a quick understanding of all:

**[Concern of novelty]** We elaborated in detail how our contribution is different from the existing approaches as we to the best of our knowledge are the first to successfully introduce and demonstrate weight conditioning in a *once-for-all adversarial training* framework. In particular, we have two different aspects of contributions: *1. A novel formulation of weight conditioned learning to solve various limitations of once-for-all adversarial training, 2. Conduct thorough empirical analysis to demonstrate the efficacy of FLOAT. In particular, we are the first to demonstrate FLOAT's performance at an extensive scale with five datasets including that with Tiny-ImageNet and on five different test time attacks. We also conducted an analysis with $\lambda_{th}$ and noise re-scaling factor.*

**[Concern of limited benefits]** Compared to existing SOTA, FLOAT is better in many metrics including training time, training parameters and hyperparameters, storage cost, potential inference latency, and speed, and last but not the least "classification performance". We would also like to highlight that, given the significant amount of time and energy consumed to train various models [1-3], a faster training alternative with much reduced trainable parameters can have significant environmental benefits.

**[Concern with a sharp drop-off in the plots]** We described possible reasons for this and highlighted this being a limitation of existing SOTA as well and not an explicit limitation of FLOAT. Having said this, in this paper, we have also highlighted possible ways to reduce the drop-off slope. 

**[Empirical evaluations to bolster our claims]** While we acknowledge theoretical understanding of machine learning phenomena are important, this paper does take the other route of thorough empirical evaluation of the novelty and benefits that have been highlighted. 

**[Difference from OAT, the existing SOTA]** While we understand our paper flows in a way that might be considered similar to that in OAT, however, this was intended to make the reader understand where FLOAT differ from OAT and in what aspect this work can yield benefits compared to OAT. We have clearly highlighted the significant differences between these two approaches in the rebuttal and how these differences make FLOAT achieve a unique combination of multiple potential practical deployable benefits for resource-constrained hardware.

**[Clarification in understanding plots and a few writing concerns]** We have clarified the plots where we recognized a potential misunderstanding (Fig. 7 and 12, in particular) might have happened. We made changes in the manuscript with suggested modification of certain terms and removal of semi-redundant information.

Finally, given the significant practical impact of the work as already highlighted by reviewer Xsuw and also noted by earlier conference in recognition of the previous SOTA [4], we genuinely hope in having a possible discussion for a clear and fruitful resolution of the concerns highlighted above. Please do not hesitate to contact us if there are other clarifications or experiments we can offer. Thanks!

Best wishes, 

Authors    

**References**

[1] Shrinking deep learning’s carbon footprint, MIT News, August 2020.

[2] Training a single AI model can emit as much carbon as five cars in their lifetimes, MIT Technology Review, July 2019.

[3] The carbon impact of artificial intelligence, Nature Machine Intelligence, August 2020.

[4] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free, NeurIPS 2020.


 We thank the reviewer for their detailed feedback. Here we address the concerns raised by the reviewer.

***On general summary: This paper argues feature-wise linear modulation (FiLM) methods to perform in-situ calibration are too computationally expensive.***

We thank the reviewer for the nice summary of our work. However, we would like to clarify that *we never claimed that the FiLM based approach is computationally too expensive* (than that without FiLM) as MLP layers do not significantly add to the total number of FLOPs *(highlighted in 1. Introduction, 4.3 Discussion on FLOPs, Appendix A.2)*.  Rather we highlighted two other drawbacks of this approach, a potential increase in inference latency and a significant increase in memory overhead, both of which are critical on resource-constrained applications that too at significantly reduced training time.

***The work being heavily based on the previous work Wang et al. 2020 <Once for all adversarial training>, and the improvement being natural but very limited and  hence severely diminished the novelty of this work.***

We respectfully disagree with the reviewer’s comment here. We believe our approach is significantly different from that of OAT from Wang et al 2020 [1]. We also empirically show that our approach provides significant improvements on several fronts: 1. FLOAT models provide similar or better performance compared to the existing SOTA, 2. FLOAT requires less training time than both PGD-AT and OAT. 3. FLOAT models potentially have less latency than OAT, 4. FLOAT models have fewer parameters compared to OAT. Thus we outperform OAT in multiple metrics and believe this demonstrates our contribution is significant. 

In addition to the significant results, we also believe that our approach has novelty. We briefly discuss our differences from existing SOTA here for clarification.

**Key novelty in our approach:**
The key principle difference between our approach and OAT[1] is that they apply conditional learning via linear transformation of the activation maps, whereas we apply conditioning on the weights. Apart from OAT, there have been prior works [2-4] that use various affine transformations like feature-wise linear modulation-based conditioning on activation maps and thus FiLM has remained a popular and well-researched area. On the contrary, to the best of our knowledge, we are the first to formulate a dual-weight tensor-driven conditional learning approach, relying on conditioning via noise tensor components on the **shared weight tensors**. Moreover, our problem formulation disentangles the additional training hyperparameter overhead of $\lambda$ (and its various conditional values) to only an inference hyperparameter ($\lambda_n$ in our case). This is primarily because of a simple yet effective novel contribution in FLOAT, where we propose rescaling the noise during inference using $\lambda_n$. This rescaling effectively controls the tradeoff between clean and adversarial image performance while reducing the training time over-head (up to 1.43x). 

**Key difference in architecture:**
[1] uses FiLM modules after every BN stage and thus for $b$ BN layers they require $b$ FiLM modules. In contrast, we do not require any such transformation module. Instead, for a convolution weight tensor at layer $l$, we introduce a trainable noise scaling factor $\alpha^{l}$. Hence, for $c$ convolutional layers we require only $c$ such trainable parameters that is negligible compared to the added parameter cost of FiLM modules.  

**Key practical impact:**
Each FiLM module has 2 MLP layers in [1]. Thus for $b$ FiLM modules it requires $2b$ MLPs that adds to the layer latency [5], which may significantly increase model latency [6] and limit its usage in real-time applications. Moreover, the models yielded by OAT are parameter heavy due to the presence of FiLM layers which can significantly increase the model’s memory budget (as exemplified in Table 1 in the Appendix). It is well known that the energy cost of memory accesses can be up to 200x higher than compute [7] and thus for such resource-limited devices FLOAT is a very attractive alternative.

*Broader goal of our research : We believe the novelty and inspiring results of this research will further open the door for exploration of efficient weight conditioning in various other applications, that otherwise often relied on activation conditioning.*

***paper containing redundant information e.g. Algo 1 and Algo 2 are mostly similar with minor tweaks, and could very well be condensed. Fig. 2 and 5 are redundant and could be merged.***

While we agree that Algorithms 1 and 2 are similar, merging them may confuse readers who may not be familiar with Slimmable network training [8]. However, we have followed the reviewer’s suggestion and have moved Algorithm 2 to the Appendix. We have kept Fig. 5 to help the reader understand the FLOAT Slim architecture without referring to the original slimmable paper [8].




 We thank the reviewer dQEN for the acknowledgment and recognition of our clear and detailed rebuttal. While we are glad to receive an **acceptance** from dQEN, we look forward to any further discussion if the reviewer feels something is necessary to make our contribution clear.

This paper generalized the conventional once-for-all adversarial training (OAT) approach [Wang et al., 2020] to FLOAT, the fast learnable once-for-all adversarial training method. The key technical innovation lies at the incorporation of $\lambda$-conditioned noise into model weights so as to achieve the purpose of OAT but not request additional layers to perform conditioning. To demonstrate the effectiveness of the proposed approach, extensive experiments are provided across datasets and model architectures.   Strengths:

+ Simple but effective model noise injection approach
+ Extention to slimmable networks is plausible
+ Sufficient empirical justification across multiple dataset and model types.

Weakness:

- The technical contribution is marginal compared to the OAT baseline. Eq. (4) is similar to $\lambda$-conditioned weight smoothing with a learnable variance scaled by $\alpha^l$, right? If so, during training, is only one Gaussian noise realization generated? Or are multiple noise copies used to smoothen the model weights when $\lambda = 1$?  

- I suggest incorporating experiments evaluated by non-PGD/FGSM attacks (e.g., CW attacks or ensemble attacks like AutoAttacks) at testing time.  In other words, it is worthwhile to examine if the superior RA performance (e.g., Fig. 7c) is correlated with test-time attack types. 

- Question on Fig. 7: Fig. 7a and 7c showed that the OAT baseline is worse than PGD-AT. Then, what is the benefit of proposing OAT in the literature? Does it outperform PGD-AT in other metrics? If so, should they be considered in the comparison with FLOAT?

 In summary, the paper contains some interesting results but the technical novelty is marginal and the effectiveness should be further justified through experiments. 

 ***Question on Fig. 7: Fig. 7a and 7c showed that the OAT baseline is worse than PGD-AT. Then, what is the benefit of proposing OAT in the literature? Does it outperform PGD-AT in other metrics? If so, should they be considered in the comparison with FLOAT?***

We thank the reviewer for asking for this clarification. It is true that for various $\lambda$ values the OAT baseline often performs inferior to the PGD-AT baseline.  However, for each $\lambda$ PGD-AT requires a separate model training and thus for deployment requires a separate set of trained weights. For $n$ $\lambda$ choices this effectively increases the training time and hardware storage cost by $n$ times, making PGD-AT an extremely inefficient and often infeasible alternative. Thus, the reviewer’s intuition about there being other metrics on which OAT outperforming PGD-AT is correct. and we have also compared FLOAT in these metrics. In particular,  we will modify Fig. 8 and included the training time and memory cost in PGD-AT to yield one model for any specific $\lambda$ choice (for $\lambda > 0$). Table R4T2 details these comparison results (lower is better) for CIFAR-10 with ResNet34. In table PGD-AT:1T and PGD-AT:4T yield one model and 4 models for one and four $\lambda$ choices, respectively.
### R4T2
|Method | Normalized training time | Single model storage | Storage cost to support $\lambda$ switchability |      
|----------|:----------|:----------|:-------------:|
| PGD-AT:1T | 1.12x | 21.28 MB | NA |
| PGD-AT:4T | ~4x | 21.28 MB | ~85 MB |
| OAT | ~1.25x | 31.4 MB | ~31.4 MB |
| FLOAT | $\textbf{~1x}$ | $\textbf{~21.28}$ MB | $\textbf{~21.28}$ MB |

***In summary, the paper contains some interesting results but the technical novelty is marginal and the effectiveness should be further justified through experiments.***

We thank the reviewer for finding interest in our work. As suggested by the reviewer, to provide further empirical evidence we have also evaluated FLOAT on Tiny-ImageNet, a popular large scale dataset. Table R4T3 details the performance on two different $\lambda_{th}$ choices. Compared to performance on other datasets, FLOAT’s similar performance trend on such a complex large scale dataset highlights the consistent effectiveness of our approach. 
### R4T3
|Model | Type/$\lambda_{th}$| $CA/RA  %$$(\lambda_{n}=0.0)$ | $CA/RA  %$$(\lambda_{n}=0.2)$ | $CA/RA  %$$(\lambda_{n}=0.7)$ | $CA/RA  %$$(\lambda_{n}=1.0)$| Normalized memory | 
|----------|:----------|:----------|:-------------:|------:|-------------:|-------------:|
|ResNet34 |FLOAT/0.0| $\textbf{66.92}$/$\textbf{1.98}$ | 60.46/$\textbf{20.42}$ | 57.99/$\textbf{22}$ | $\textbf{54.67}$/22.68 | $\textbf{1x}$
|ResNet34 |FLOAT/0.5| $\textbf{66.92}$/$\textbf{1.98}$ | $\textbf{66.63}$/3.27 | $\textbf{58.11}$/20.74 | $\textbf{54.67}$/22.68 | $\textbf{1x}$
|ResNet34 | PGD-AT/NA | 66.76/0 | 59.55/19 | 55.61/21.20 | 52.82/$\textbf{22.75}$| 4x

[1] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free, NeurIPS 2020.
[2] FiLM: Visual reasoning with a general conditioning layer, AAAI 2018.
[3] TENT: fully test-time adaptation by entropy minimization, ICLR 2021.
[4] Semantic Image Synthesis with Spatially-Adaptive Normalization, CVPR 2021.
[5] Weight Smoothing to Improve Network Generalization, IEEE Transactions on Neural Networks, 1994.


 We thank the reviewer for the constructive feedback and are happy to clarify the raised concerns below. 

***Better results compared to prior work***

We appreciate the reviewer’s appreciation of the results. We would like to highlight that our work not only provides better classification performance, but also improves the SOTA in terms of reduced memory, energy budget, and latency, which can be a bottleneck for real-time applications of conditional adversarial learning, particularly on edge devices. This makes FLOAT an efficient approach for a large variety of resource-constrained safety-critical applications.

***Specifically, adding parametrized noise tensor to weights improve robustness is shown in prior work..***

We agree that the earlier work of He et al. [1] has added parametric noise with the goal of improving robustness through standard adversarial training. However, their approach suffers from a significant drop in clean image accuracy. To the best of our knowledge, our’s is the first paper to show how noise can be added to the weights conditionally to create dual weight tensors in such a manner as to provide an in-situ tradeoff between clean and adversarial performance. Thus, our models not only have improved robustness but also achieve SOTA clean image classification performance. Another minor, but important, difference from [1] is that for an image batch of size N, [1]  used adversarial images for the entire batch of clean images thus effectively using 2N images per batch to train, whereas we evenly use N/2 clean images and use perturbed versions of remaining N/2 images to train our model. Thus, we speed up training compared to a single PGD-AT training(up to 1.37x).

***..and using different BNs for clean and adv inputs is known..***

We agree that Wang et al. (2020) [2] has leveraged the idea of dual BN [3] and conditional learning to improve clean and adversarial images. However, to ensure the dual BNs were properly trained, [2] relied on expensive FiLM modules controlled by the extra hyperparameter $\lambda$.  In contrast, our proposed structure shows that the dual BNs can be trained using dual weight tensors that are created by conditionally adding noise to the original weight tensor, yielding better accuracy with lower latency and lower memory consumption. 

Moreover, [2] needed many values of $\lambda$ to create various activation map transformations that correspond to different trade-offs between clean and adversarial training. On the other hand, our approach uses an inference hyperparameter that re-scales the noise to control the tradeoff between clean and adversarial training, and thus requires only binary $\lambda$ values during training. This significantly speeds up training compared to [2].

***Although the method as a whole seems new, each of its parts is known.***

In addition to the differences detailed above, we note that [1] and [3] have been in the literature for about two years. Yet, the SOTA conditionally robust models in [2] still relied on FiLM-based activation conditioning. Given the speed of which ML research is advancing, this suggests that the dual weight tensor approach we propose was not obvious. 
In summary, we believe, as emphasized by reviewer Xsuw, that the key contribution of this paper is that it demonstrates that the learnable additive noise in [1], which was originally proposed to improve model robustness, can also be effectively adapted to **efficient model conditioning on the weight tensors**.

*Broader goal of our research : We believe the novelty and inspiring results of this research will further open the door for exploration of efficient weight conditioning in various other applications, that otherwise often relied on activation conditioning.*

***concern regarding the continuum of the trade-off. First, it is not necessarily a trade-off as shown in Fig 6 and 7: sometimes one can achieve higher robustness and accuracy...***

We thank the reviewer for highlighting this important point. We agree that for some cases in the plots the robustness does not significantly increase despite an increase in $\lambda_n$. Consequently, in some cases, we obtain improved clean image performance without a significant drop in robustness. This can be attributed to the fact that, because the generation of adversarial images is noisy, there is no guarantee that increasing $\lambda$ will always increase robustness. 
However, we would also like to emphasize that this characteristic is not unique to our approach. PGD-AT and OAT [2] also suffer from a  lack of guaranteed increase in robustness with increasing $\lambda$ that can be attributed to the noisiness of adversarial image generation. 

Nevertheless, we will add a note in Sec. 4.2 of the original manuscript to clarify this point and tone down our claim of continuity.


 We thank the reviewer for a precise summary of our work and highlighting some key contributions. Here we address the raised concerns by the reviewer.

***The technical contribution is marginal compared to the OAT baseline.***

As we mentioned in our response to Reviewer BuXN, the key principle difference between our approach and OAT[1] is that they apply conditional learning via linear transformation of the activation maps, whereas we apply conditioning on the weights. Apart from OAT, there have been prior works [2-4] that use various affine transformations like feature wise linear modulation based conditioning on activation maps and thus FiLM has remained a popular and well researched area. On the contrary, to the best of our knowledge, this paper is the first one to formulate a dual-weight tensor-driven conditional learning, conditioned on noise tensor components. Moreover, our problem formulation disentangles the additional training hyperparameter overhead of $\lambda$ (and its various conditional values) from training to only an inference hyperparameter ($\lambda_n$ in our case). This is primarily because of a simple yet effective novel contribution in FLOAT, where we propose scaling the noise during inference using a rescaling factor $\lambda_n$. This rescaling effectively controls the tradeoff between clean and adversarial image performance. 

Moreover, [1] uses FiLM modules after every BN stage and thus for $b$ BN layers they require $b$ FiLM modules. In contrast, we do not require any such transformation module. Instead of a convolution weight tensor at layer $l$, we introduce a trainable noise scaling factor $\alpha^{l}$. Hence, for $c$ convolutional layers we require only $c$ such trainable parameters, that is negligible compared to the added parameter cost of FiLM. 

*Broader goal of our research : We believe the novelty and inspiring results of this research will further open the door for exploration of efficient weight conditioning in various other applications, that otherwise often relied on activation conditioning.*

***Eq. (4) is similar to λ-conditioned weight smoothing with a learnable variance scaled by αl, right?*** 

We understand weight smoothing is a network generalization strategy [5] that is often used to mitigate model overfitting. While our approach may be viewed as a form of weight smoothing, our goal of conditionally adding noise in this paper is different from that of weight smoothing. In particular, we aim at generating two separate variants of a weight tensor that are convolved with both clean and adversarial activation maps, respectively. In this way, our models can be trained on both clean and perturbed images somewhat independently.  The  $\alpha^l$ is a trainable scaling factor that controls the severity of the noise that is generated from a normal distribution having variance same as that of the weight tensor. We will be happy to clarify further.

***If so, during training, is only one Gaussian noise realization generated? Or are multiple noise copies used to smoothen the model weights when λ=1?***

During training, for each forward pass for each weight scalar we sample noise from a single Gaussian distribution with mean = 0 and variance that is the same as that of the weight tensor of the corresponding layer.

***Suggestion on evaluating performance on other attacks (e.g., CW attacks or ensemble attacks like AutoAttacks) at testing time.***

We thank the reviewer for the suggestion. Table R4T1 details performance comparison of FLOAT generated models with that generated via PGD-AT, on CIFAR-10 on both Autoattack and CW attack . As the results show, the trend is similar to that in Fig. 7. We will also add the results to the final manuscript in Sec 4.5 and Appendix, respectively.  
### R4T1
|Attack type |Method| $CA/RA  %$$(\lambda_{n}=0.0)$ | $CA/RA  %$$(\lambda_{n}=0.2)$ | $CA/RA  %$$(\lambda_{n}=0.7)$ | $CA/RA  %$$(\lambda_{n}=1.0)$|      
|----------|:----------|:----------|:-------------:|------:|-------------:|
|Autoattack |PGD-AT| $\textbf{95.46}$/0 | $\textbf{90.35}$/37.36 | 88.46/43.92 | 87.4/44.61 |
|Autoattack |FLOAT|  94.83/$\textbf{11.98}$ | 89.8/$\textbf{43.94}$ | $\textbf{89.5}$/$\textbf{44.82}$| $\textbf{89.3}$/$\textbf{45.08}$ |
|CW |PGD-AT| $\textbf{95.46}$/0.06 | $\textbf{90.35}$/52.2| 88.46/54.2 | 87.4/54.4 |
|CW |FLOAT | 94.83/$\textbf{28.8}$ | 89.8/$\textbf{54.9}$ | $\textbf{89.5}$/$\textbf{55.8}$| $\textbf{89.3}$/$\textbf{55.9}$ |


 ***In figure 7, the performance of FLOAT seems to be better than OAT and PGD. But Figure 12 in the appendix tells me this is highly dependent on a good pick for lambda, which might not be generalizable.***

We think there is some confusion here in understanding the plots. For each dataset we have 4 different $\lambda_n$ choices, and for each $\lambda_n$ we have CA and RA, and thus 8 different accuracy comparison points. As we used 3 datasets, we have 24 comparison points overall .
Of these 24 comparison points, FLOAT always provides improved (of up to ~14.5%) accuracy both in CA and RA with only one exception of CA on SVHN at $\lambda_n = 0.2$. Here, OAT obtains a negligibly superior CA of 95.84% compared to FLOAT’s 95.24%. Compared to PGD-AT trained models, OAT provided inferior performance on 11/24 points. On the contrary FLOAT has negligible inferior performance on 2/24 points only at $\lambda_n=0$, and 0.2 on CIFAR-10 CA. Hence, our empirical evaluation shows that FLOAT’s superior performance over existing SOTA (OAT) as well as PGD-AT is **not** a function of the choice of $\lambda_n$. We will be happy to clarify this further.

***Abusing highly theoretical term like pareto frontier***

We apologize for this and have replaced the sentences having the term “Pareto Frontier” with these:

1. To directly evaluate the robustness vs accuracy trade-off, we evaluate the clean and robust accuracy values of models generated through FLOAT at various $\lambda$ values compared with those yielded through OAT and PGD-AT.

2. Extensive experiments show FLOAT’s superiority in terms of an improved CA-RA performance, reduced parameter count, and faster training time.

We will be happy to correct any further terms that the reviewer considers necessary.

***Overall, I think the paper is not at the publishable stage, I suggest the authors condense the information and add supporting theoretical thoughts behind the proposed modification.***

While we agree that a strong theoretical foundation is important, the state of the art theory cannot predict or describe many important phenomena in SOTA large deep neural networks.  Consequently, we often rely on an alternative approach based on empirical analyses of a phenomenon as it occurs in practice, and we have done this in our paper.

We assert that this paper makes two clear contributions. 1. It is the first paper to propose a noise-conditioned dual weight-dual BN DNN training yielding models that can provide SOTA on both clean and adversarial images. 2. It describes thorough evaluations that show the superiority of the proposed method on memory footprint, latency, and training time along with superior CA-RA performance. 

We believe, with the proliferation of DNN-enabled edge hardware devices for various safety critical scenarios, FLOAT enables the use of conditionally-trained in-situ-adjustable models on devices with low memory budgets. With the detailed empirical analysis and broad applicability of our contribution, we believe the current manuscript is a significant contribution to the scientific community and that it is inappropriate to dismiss it due to the lack of supporting theoretical analysis. That said, we will indeed consider the theoretical analysis of conditional learning and this work as an interesting area of future research. We will add a note regarding this in Section 5.

***Empirical contributions are neither significant nor novel.***

We would like to take this opportunity to highlight  that, FLOAT has evaluation results on more complex datasets compared to that in the existing alternative (Fig. 6). We provide a unique  motivational analysis with noise-rescaling (Fig. 4). We have also demonstrated the impact of BN switching threshold in model CA-RA performance (Fig. 6), and showed results on two new test time attacks: AutoAttack and CW attack (for rebuttal to reviewer dQEN). Apart from these, we have done runtime analysis (Fig. 8), and memory budget analysis (Table 1) of all the models. Thus we feel this paper has presented more through empirical evaluations of conditional learning compared to that in existing alternatives.

[1] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free, NeurIPS 2020.
[2] FiLM: Visual reasoning with a general conditioning layer, AAAI 2018.
[3] TENT: fully test-time adaptation by entropy minimization, ICLR 2021.
[4] Semantic Image Synthesis with Spatially-Adaptive Normalization, CVPR 2021.
[5] Hetconv: Heterogeneous kernel-based convolutions for deep cnns, CVPR 2019.
[6] NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration, CVPR 2021.
[7] 1.1 Computing’s energy problem (and what we can do about it), ISSCC 2014.
[8] Slimmable Neural Networks, ICLR 2019.





 ***Additionally, almost in all figures, FLOAT has flat robustness changes followed by a sharp change.***

We thank the reviewer for this point and partly agree with this comment. The sharp drop-off in robustness is due to the discrete switch in BN selection that is controlled by $\lambda_{th}$. 

However, our approach offers a unique means of decreasing the slope of the drop off which can decrease the discontinuous nature of the tradeoff. In particular, by increasing $\lambda_{th}$ from the default choice of zero (that was used in OAT), we can reduce the drop-off slope. For example,  in Fig. 6, we have sampled values of $\lambda_n$ from {0.0, 0.2, 0.7, 1.0} and evaluated FLOAT’s performance on two different scenarios of the BN choice. The blue lines correspond to a $\lambda_{th}= 0$ (scenario $\mathcal{S}1$) and the red lines correspond to an increased $\lambda_{th}= 0.5$ (scenario $\mathcal{S}2$). It is clear from the plots that the slope of the red line ($\mathcal{S}2$) is generally less steep than the slope of the blue lines ($\mathcal{S}1$), increasing the continuity of the available tradeoff. 

We also would like to emphasize that *OAT [2] also suffers from a similar drop-off* (see Figure 7) due to the impact of a discrete threshold that determines which BN to use, despite $\lambda$ being a training hyperparameter in [2]. 
We will clariy these points in Section 4.3 of the paper.

***That is, practically speaking there are only two knobs on the trade-off curve***

We indeed agree to the reviewer’s comment on there being two knobs in FLOAT, one knob $\lambda_n$ that controls the noise scaling, the other knob being $\lambda_{th}$ that decides when to switch from using $BN_C$ to $BN_A$ during inference. In contrast to OAT which only exploited one knob $\lambda$, we show that the sharp drop-off can be further reduced with the help of our two knobs.  
If, on the other hand, by the term ‘two knobs’ the reviewer has hinted at FLOAT having two major operating regions, we partly agree. As described above, the sharp transition between the modes is generally observed due to switch in BNs and this is also observed in OAT [2]. However, as also described above, we can reduce the size of the drop-off and provide a more continuous tradeoff. 

***It is not clear how applying to Slimmable improves the understanding of the proposed method. They seem pretty orthogonal to me or perhaps I'm missing something.***

We agree that the idea of slimmable networks is indeed complementary to FLOAT. However, we note that the authors in [2] extended OAT to slimmable networks to provide a three-way in-situ trade-off between accuracy, robustness, and complexity. As we have proposed FLOAT as a potential replacement to OAT, a natural question is  “Can FLOAT models extend to Slimmable networks and what performance do they achieve?” We thus believe that showing empirical results for FLOAT applied to Slimmable networks has both academic and practical benefits. Our extension not only validates the superiority of our approach in providing a three-way in-situ trade-off, but also extends the application of such models into edge devices with tight resource constraints. Moreover, compared to OATS, FLOATS' even faster training by up to 1.9x marks a significant benefit both in terms of potential training cost and time. 
We might also emphasize the practical significance of Slimmable networks. It is well known that the same model may have different run times for a specific task based on the underlying hardware [5]. For example, a MobileNetV1 model takes 24 ms and 116 ms to run on the OnePlus 6 and Google Pixel, respectively [6]. Moreover, the compute budget for the same device may vary over time based on other applications that are running on the device [7]. Hence, an efficient deployment strategy of a model may also require the model to dynamically slim down to meet a specific target latency and/or compute budget. FLOAT Slim yields a class of networks that can provide dynamic slimmability along with an in-situ accuracy-robustness trade-off.

[1] Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack, CVPR 2019.
[2] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free, NeurIPS 2020.
[3] Adversarial Examples Improve Image Recognition, CVPR 2020.
[4] FiLM: Visual reasoning with a general conditioning layer, AAAI 2018.
[5] AI Benchmark: Running deep neural networks on android smartphones, ECCV W 2018.
[6] Slimmable Neural Networks, ICLR 2019.
[7] Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms, CVPR W 2021.



 We thank the reviewer for their insightful comments and wonderful summary of our work. We happily acknowledge the reviewer’s appreciation about the potential impact FLOAT can have in the research community. Here we address the two concerns raised by the reviewer.

***The results in Figure 7 shows OAT is outperformed by PGDAT on CIFAR10 and STL10. However, the original OAT paper showed they have similar performance. I suggest the authors to try more hyperparameters on OAT to better match the results in the original paper.***

We agree with the reviewer on this. In this regard we would like to clarify that there is a difference in hyperparameter setting between OAT trained in the original paper and that in our paper. In particular, the $S_{\lambda}$ for training in the original paper was [0.0,0.1,0.2,0.3,0.4,1.0], and that in our work is [0.0, 0.2, 0.7, 1.0]. With similar hyperparameter choices of training $\lambda$ s we indeed saw a similar trend as that reported in the original paper. We will be happy to clarify further related concerns.

***Please discuss and acknowledge more on He et al. (2019) in both related works and method sections.***

We thank the reviewer for highlighting this. Though we have mentioned that He et al. (2019) has adopted noise addition to weight tensor for improved adversarial robustness, we have now updated the manuscript with further details of this work in Sec. 2.2 and 3.

This paper presents an algorithm named FLOAT(S), which adds conditional perturbation to weights during training (add noise when training with adversarial examples). This method immediately improves over its predecessor OAT by removing extra FiLM layers. As demonstrated in the experiments, FLOAT achieves better training efficiency than OAT, and perhaps more surprisingly, it has better robustness over OAT. ## Strengths

- Better results compared to prior work

## Weaknesses

- Although the method as a whole seems new, each of its parts is known. Specifically, adding parametrized noise tensor to weights improve robustness is shown in prior work, and using different BNs for clean and adv inputs is known to be useful.

- I'm concerned regarding the continuum of the trade-off. First, it is not necessarily a trade-off as shown in Fig 6 and 7: sometimes one can achieve higher robustness and accuracy by changing $\lambda_n$. Additionally, almost in all figures, FLOAT has flat robustness changes followed by a sharp change. That is, practically speaking there are only two knobs on the trade-off curve.

- It is not clear how applying to Slimmable improves the understanding of the proposed method. They seem pretty orthogonal to me or perhaps I'm missing something. Overall, I think this paper is a bit incremental even though it improves over prior work.

This paper tackles the problem of in-situ trade-off between model robustness and accuracy, which is of great importance in real-world applications. The proposed method, FLOAT, makes improvements on the previous state-of-the-art method OAT (NeurIPS'20) by using a more efficient model conditioning method. Experimental results show that FLOAT outperforms OAT in terms of accuracy, robustness and model size. The FLOPs of FLOAT, OAT, and the PGDAT baseline are almost the same.  Pros:
1. The problem setting of in-situ trade-off between accuracy and robustness is of great practical importance for real-world high-stakes applications such as autonomous driving. 
2. The proposed method solve a major limitation of previous state-of-the-art method OAT: OAT leads to unneglectable overhead in model size by using the extra FiLM layers for model-conditioned adversarial training. The proposed FLOAT uses a more efficient model conditioning method, which adds configurable scaled noise on model weights. 
3. The experimental results show FLOAT outperforms previous state-of-the-art OAT and PGDAT by considerable margins on three popular benchmark datasets, while largely reducing the model sizes. 
4. Although the proposed "configurable scaled noise" method is inspired by He et al. (2019) and the model conditioned learning is inspired by Wang et al. (2020), the overall framework of FLOAT still has its own novelty. Specifically, it is good to see the learnable additive noise in He et al. (2019), which is originally proposed to improve model robustness, can also be adapted for efficient model conditioning. With that said, I do suggest the authors to discuss and acknowledge more on He et al. (2019).
5. The paper is well written and easy to follow. 

Cons:
1. The results in Figure 7 shows OAT is outperformed by PGDAT on CIFAR10 and STL10. However, the original OAT paper showed they have similar performance. I suggest the authors to try more hyperparameters on OAT to better match the results in the original paper. 
2. Please discuss and acknowledge more on He et al. (2019) in both related works and method sections.  The problem setting is of great importance to real-world applications. The proposed method solves a major limitation of previous state-of-the-art method, and empirically achieves good performance. 

Meta-review: This paper presents Fast Learnable Once-for-all Adversarial Training (FLOAT) which transforms the weight tensors without using extra layers, thereby incurring no significant increase in parameter count, training time, or network latency compared to a standard adversarial training. Compared to existing SOTA, FLOAT is better in many metrics including training time, training parameters and hyperparameters, storage cost, potential inference latency, speed, and task accuracy. 

This paper received highly mixed scores 8-6-5-3. During the private discussion, Reviewer DN3 stated that she/he was willing to raise score from 3 to 5. Although I am not sure why the reviewer did not actually make the change, I'm consider the rating increase as happened (i.e., "factually" 8-6-5-5).

After reading this paper, AC agrees that FLOAT solved an important limitation of the previous state-of-the-art method OAT: reducing the FiLM overhead by using a more efficient model conditioning method, i.e. adding configurable scaled noise on model weights. The lukewarm part is, the method is no doubt heavily based on the OAT paper. Even one can argue the "method as a whole" is novel, the contributions (despite interesting) remain slightly incremental.

In view of the above, AC currently places this paper as a borderline rejection.

Decision: Reject