FLOAT: FAST LEARNABLE ONCE-FOR-ALL ADVER-
Year: 2021

Abstract:
Training a model that can be robust against adversarially-perturbed images without compromising accuracy on clean-images has proven to be challenging. Recent research has tried to resolve this issue by incorporating an additional layer after each batch-normalization layer in a network that implements feature-wise linear modulation (FiLM). These extra layers enable in-situ calibration of a trained model, allowing the user to configure the desired priority between robustness and clean-image performance after deployment. However, these extra layers significantly increase training time, parameter count, and add latency which can prove costly for time or memory constrained applications. In this paper, we present Fast Learnable Once-for-all Adversarial Training (FLOAT) which transforms the weight tensors without using extra layers, thereby incurring no significant increase in parameter count, training time, or network latency compared to a standard adversarial training. In particular, we add configurable scaled noise to the weight tensors that enables a trade-off between clean and adversarial performance. Additionally, we extend FLOAT to slimmable neural networks to enable a three-way in-situ trade-off between robustness, accuracy, and complexity. Extensive experiments show that FLOAT can yield state-of-the-art performance improving both clean and perturbed image classification by up to ∼6.5% and ∼14.5%, respectively, while requiring up to 1.47× fewer parameters with similar hyperparameter settings compared to FiLM-based alternatives. Code for this project will be made available shortly.

heading: 1 INTRODUCTION
text: As artifical intelligence (AI)-enabled applications proliferate, the robustness concerns of deep neural networks (DNNs) has grown in importance Zhang et al. (2019). These concerns are largely associated with the vulnerability of machine learning (ML) classifiers to adversarial examples, i.e., images that have well-crafted barely-visible perturbations from corresponding clean images and have the ability to fool classifier models into making wrong predictions Carlini & Wagner (2017); Goodfellow et al. (2014). With the growing usage of DNNs in safety-critical and sensitive applications including autonomous-driving Bojarski et al. (2016) and medical image analysis Han et al. (2021), it therefore becomes crucial that they have high accuracy on both adversarially-perturbed and clean images. To improve the model performance of DNNs against such adversarial attacks, various defense mechanisms have been proposed including hiding gradients Tramèr et al. (2017), adding noise to parameters He et al. (2019), and detection of adversaries Meng & Chen (2017). In particular, adversarial training Madry et al. (2017) has proven to be a consistently effective approach.
These defenses, however, come at various costs. Firstly, most of these methods suffer from increased training time due to the additional back-propagation overhead to generate perturbed images. Secondly, adversarial defenses sometimes cause a significant drop in clean-image accuracy Tsipras et al. (2018), highlighting an accuracy-robustness trade-off that has been explored both theoretically and experimentally Sun et al. (2019); Tsipras et al. (2018); Schmidt et al. (2018). Moreover, the defenses rely on several hyperparameters whose settings force the model to work at a specific point along this trade-off. This is disadvantageous in applications in which the desired trade-off depends on context Wang et al. (2020).
A naive solution to this problem is to use multiple networks trained with different priorities between clean and adversarial images. However, this increases both training time and inference memory. Alternatively, recent work has proposed training a once-for-all adversarial network (OAT) that supports conditional learning Wang et al. (2020), enabling the network to adjust to different input distributions. In particular, after each batch-normalization (BN) layer, they add a feature-wise linear modulation (FiLM) module Perez et al. (2018) whose weights are controlled by a parameter λ. For inference, the user sets λ to enable an in-situ trade-off between accuracy and robustness. The disadvantage with this approach is that the added FiLM modules increase the parameter count, training time, and network latency, limiting applicability in resource-constrained, real-time applications.
Our contributions. In view of the concerns above, we present fast learnable once-for-all adversarial training (FLOAT). In FLOAT, we train a model using a novel mechanism wherein each weight tensor of the model is transformed by conditionally adding a noise tensor based on a binary parameter λ. This yields state-of-the-art (SOTA) test accuracy for clean and adversarial images by in-situ setting λ = 0.0 and 1.0, respectively. For inference, we further show that model robustness can be correlated to the strength of the noise-tensor scaling factor. This motivates a simple yet effective noise re-scaling approach controlled by an added floating-point parameter that can help the user to have a practical accuracy-robustness trade-off.
Because FLOAT does not require additional layers to perform conditioning, it incurs no increase in latency and causes only a negligible increase in parameter count compared to the baseline models. Moreover, compared to OAT, FLOAT training is up to 1.43× faster, as it does not explicitly require learning intermediate fine-grained conditioning using different λs.
Moreover, we show how FLOAT can be extended to slimmable networks Yu et al. (2018) to enable an in-situ inference trade-off across three dimensions, namely, accuracy, robustness, and complexity.
To evaluate the merits of FLOAT, we conduct extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, SVHN, and STL10 datasets with ResNet34 (on both CIFAR and Tiny-ImageNet datasets), WRN16-8, WRN40-2, respectively. As shown in Fig. 1, compared to existing methods, FLOAT models can provide improved accuracy of up to ∼6.5%, and ∼14.5%, on clean and perturbed images, respectively.
The remainder of this paper is organized as follows. Section 2 presents the necessary preliminaries and prior work. We present our approach in Section 3 and analyze experimental results in Section 4. Finally, the paper concludes in Section 5 with discussions of broader impact in Section 6.

heading: 2 PRELIMINARIES

heading: 2.1 NOTATION
text: Consider a model Φ with L layers parameterized by Θ that learns a function fΦ(.). For a classification task on dataset X with distribution D, the model parameters Θ are learned by minimizing the empirical risk (ERM) as follows
minimize Θ
L(fΦ(x,Θ; t)), (1)
where t is the ground-truth class label, x is the vectorized input drawn from X , and L is the crossentropy loss function.

heading: 2.2 ROBUST MODEL TRAINING
text: Several forms of adversarial training (AT) have been proposed to improve robustness Madry et al. (2017); Samangouei et al. (2018); Buckman et al. (2018). They all use both clean and adversariallyperturbed images to train a model. Projected gradient descent (PGD) attack, recognized as one of the strongest L∞ adversarial example generation algorithm Madry et al. (2017), is typically used to create adversarial images during training. The perturbed image for a PGD-k attack with k as the number of steps is given by,
x̂k = ProjP (x)(x̂ k−1 + σ × sign(∇xL(fΦ(x̂k−1,Θ; t))) (2)
Here, the scalar corresponds to the perturbation constraint that determines the severity of the perturbation. Proj projects the updated adversarial sample onto the projection space P (x) which is the -L∞ neighbourhood of the benign sample x1. σ is the attack step-size. For PGD-AT, the model parameters are then learned by the following ERM
minimize Θ [(1− λ)L(fΦ(x,Θ; t))︸ ︷︷ ︸ LC +λL(fΦ(x̂,Θ; t))︸ ︷︷ ︸ LA ], (3)
where LC and LA corresponds to the clean and adversarial image classification loss components, respectively, weighted by the scalar λ. Hence, for a fixed λ and adversarial strength, the model learns a fixed tradeoff between accuracy and robustness. For example, an AT with λ value of 1 will allow the model to completely focus on perturbed images, resulting in a significant drop in clean-image classification accuracy. Another strategy to improve model robustness is through the addition of noise to the model weight tensors. For example, He et al. (2019) introduced the idea of noisy weight tensors with a learnable noise scaling factor to improve model robustness against gradient-based attacks. However, this strategy also incurs a significant drop in clean accuracy.

heading: 2.3 CONDITIONAL LEARNING
text: Conditional learning involves training a model with multiple computational paths that can be selectively enabled during inference Wang et al. (2018). For example, Teerapittayanon et al. (2016); Huang et al. (2017); Kaya et al. (2019) enhanced a DNN model with multiple early exit branches at different architectural depths to allow early predictions of various inputs. Yu et al. (2018) introduced switchable BNs that enable the network to adjust the channel widths dynamically, providing a in-situ efficient trade-off between complexity and accuracy. Recently, Bulat & Tzimiropoulos (2021) used switchable BNs to support runtime bit-width selection of a mixed-precision network. Another conditional learning approach used feature transformation to modulate intermediate DNN features Huang & Belongie (2017); Yang et al. (2019); De Vries et al. (2017); Wang et al. (2020). In particular, Wang et al. (2020) used FiLM Perez et al. (2018) to adaptively perform a channel-wise affine transformation after each BN stage that is controlled by the hyperparameter λ of Equation 3. Each FiLM module is composed of two fully-connected (FC) layers with leaky ReLU activation functions and dimensions that match the output feature-map channel size. Despite requiring a relatively small number of additional FLOPs, however, the FiLM module can significantly increase the number of model parameters and associated memory access cost Horowitz (2014). Moreover, the increased number of layers can significantly increase inference latency Singh et al. (2019), potentially prohibiting its use in real-time applications.
1Note that the generated x̂ are clipped to a valid range which, for our experiments, is [0, 1].

heading: 3 PROPOSED APPROACH

heading: 3.1 FLOAT
text: In this section, we disclose our fast learnable once-for-all adversarial training (FLOAT). We first focus on defining the conditions for a model being trained on either clean or adversarial images, as the two boundary conditions for training. We use a binary conditioning parameter λ, that during training, forces the model to focus on either of these two conditions.
To formalize our approach, consider a L-layer DNN parameterized by Θ and let θl ∈ Rkl×kl×cli×clo represent the layer l weight tensor, where clo and c l i represent the number of filters and channels per filter, respectively, and kl represents the kernel height/width. We transform each parameter of θl, by adding a noise tensor ηl ∈ Rkl×kl×cli×clo scaled by a parameter αl and conditioned by λ, as follows,
θ̂l = θl + λ · αl · ηl; ηl∼N (0, (σl)2). (4)
Note that the standard deviation (σl) of the noise matches that of its weight tensor. For λ = 0 and 1, we obtain the original weight tensor or a noisy variant, respectively.
As illustrated in Algorithm 1, we train our models by partitioning an image batch B into two equal sub-batches B1 and B2, one with clean (IFMC) images and the other with perturbed variants (IFMA) (lines 4 and 6 in Algorithm 1). We use the PGD-7 attack to generate perturbations on the
image batch B2. As illustrated in Fig. 2(b), the original and noisy weight tensors are convolved only with clean and perturbed variants, respectively. Similar to He et al. (2019), we also train the noise scaling factor αl (line 9). As exemplified in Fig. 3, the post-convolution feature maps for clean and adversarial inputs can differ significantly in their respective mean/variances Xie et al. (2020); Xie & Yuille (2019). Therefore, the use of a single BN to learn both distributions may limit model performance Wang et al. (2020). To solve this problem, we extend the λ-conditioning to choose between two BNs, BNC and BNA, dedicated for IFMC and IFMA, respectively.
This approach differs from previous efforts in several ways. Earlier research performed noise-injection via regularization Bietti et al. (2018); Lecuyer et al. (2019) and perturbed weight tensors He et al. (2019) to boost model robustness at the cost of a significant accuracy drop on clean images. In contrast, we use noise tensors to transform a shared weight tensor yielding a model that can be configured in-situ to provide SOTA accuracy on either clean or perturbed images. Our approach is similar to λ-conditioning used by Wang et al. (2020). However, instead of transforming activations using added FiLMbased layers trained with multiple values of λ Wang et al. (2020), we transform weight tensors using added noise conditioned by binary λ. Compared to Wang et al. (2020), we thus require models with significantly fewer parameters and training scenarios, yielding faster training (up to 1.43×).

heading: 3.2 FLOAT GENERALIZATION WITH NOISE RE-SCALING
text: One limitation of the FLOAT approach as proposed above is that it allows the user to choose between two boundary conditions only. This limits applicability when the user is not confident about which condition to use during inference. To motivate more continuous in-situ conditioning, we analyze a ResNet20 model with noisy weight tensors trained with PGD-AT on CIFAR-10 He et al. (2019). Post-training, we re-scaled αl for each layer l, using a new floating-point parameter λn to yield λn · αl. Interestingly, as shown in Fig. 4, as the re-scaling factor decreases, the model robustness decreases and the clean-image accuracy increases.
Algorithm 1: FLOAT Algorithm Data: Training setX∼D, model parameters Θ, trainable noise scaling factor α, binary conditioning
parameter λ, mini-batch size B. 1 Output: trained model parameters Θ, α. 2 for i← 0 to to ep do 3 for j← 0 to nB do 4 Sample clean image-batch of size B/2 (X0:B/2, Y0:B/2) ∼D 5 LC ← computeLoss(X0:B/2,Θ, λ = 0,α;Y0:B/2) // condition to use weights w/o noise 6 X̂B/2:B ← createAdv(XB/2:B,YB/2:B) // adversarial image creation 7 LA ← computeLoss(X̂B/2:B,Θ, λ = 1,α;YB/2:B) // condition to use transformed weights 8 L ← 0.5 ∗ LC + 0.5 ∗ LA 9 updateparam(Θ,α,∇L)
10 end 11 end
Based on this observation, we enable practical post-training in-situ calibration by adding the rescaling parameter λn to the inference model2. This allows us to enable a practical accuracyrobustness trade-off in FLOAT during inference. We define a threshold λth such that for λn > λth we select BNA to perform inference and select BNC otherwise. Wang et al. (2020) selected BNC and BNA when λ = 0 and λ > 0, respectively. We follow a similar approach by setting λth = 0.

heading: 3.3 FLOAT EXTENSION TO DYNAMICALLY SWITCHABLE MODELS
text: As mentioned earlier, deployed DNN models, particularly on edge devices, can face dynamically changing resource budgets Lou et al. (2021). To better adjust to instantaneous budget changes, earlier research has proposed the idea of slimmable neural networks Yu et al. (2018) in which each layer’s channel width can be dynamically shrunk. Let Sf be the predefined set of supported slimming-factors. For each of the floating-point slimming-factors w(w ∈ (0, 1]) in Sf , the model shrinks each convolution layer by w using a different set of BNs.
To extend FLOAT to account for dynamically changing resource budgets, we herein present FLOAT Slim (FLOATS) to yield models that, in-situ, can trade-off between accuracy, robustness, and efficiency. Yu et al. (2018) introduced the idea of switchable BNs by having
a dedicated BN for each slimming factor. Similarly, we create a dedicated set of BNA and BNC for each slimming factor in Sf . In tandem, we scale the noise tensor along with the weight tensors by the same factor to ensure the transformed noisy weights follow the same scaling. Fig. 5 shows an example of a slimmable layer of FLOATS with Sf = 3 and slimming factors w1, w2, and w3, where w1 = 1.0 and 1 > w2 > w3 > 0 that operate on the clean and perturbed feature maps of scaled widths generated by the previous layer. Algorithm for FLOATS training is detailed in the Appendix.

heading: 4 EXPERIMENTAL RESULTS AND ANALYSIS

heading: 4.1 EXPERIMENTAL SETUP
text: Models and datasets. To evaluate the efficacy of FLOAT, we performed detailed experiments on five popular datasets, CIFAR-10, CIFAR-100 Krizhevsky et al. (2009), Tiny-ImageNet Hansen (2015) with ResNet34 He et al. (2016), SVHN Netzer et al. (2011) with WRN16-8 Zagoruyko
2During inference, we allow the user to control the trade-off using λn, since λ is binary in FLOAT. Also, λn = 0 and λn = 1, matches the training boundary conditions. OAT, on the other hand, can reuse the variable λ during both training and inference as it can assume any floating point value in [0, 1].
& Komodakis (2016), and STL10 Coates et al. (2011) with WRN40-2 Zagoruyko & Komodakis (2016).
Hyperparameters and training settings. In order to facilitate a fair comparison, for CIFAR-10, SVHN, and STL10 we used similar hyperparameter settings as Wang et al. (2020)3. For CIFAR-100, we followed same hyperparameter settings as that with CIFAR-10. For Tiny-ImageNet we trained the model for 120 epochs with an initial learning rate of 0.1 with cosine decay. For adversarial image generation, we used the PGD-k attack with and k set to 8/255 and 7, respectively, for all experiments. We initialized the noise scaling-factor αl for layer l to 0.25 as described in He et al. (2019). Further hyperparameter and model details are provided in the Appendix A.1 and A.2, respectively. We used the PyTorch API Paszke et al. (2017) to implement our models and trained them on a Nvidia GTX Titan XP GPU.
Evaluation metrics. Clean (standard) accuracy (CA): classification accuracy on the original clean test images. Robust Accuracy (RA): classification accuracy on adversarially perturbed images generated from the original test set. We use RA as the measure of robustness of a model. To directly evaluate the robustness vs accuracy trade-off, we evaluate the clean and robust accuracy values of models generated through FLOAT at various λ values compared with those yielded through OAT and PGD-AT. We used the average of the best CA and RA values over three different runs with varying random seeds, for each λ values to report in our results.

heading: 4.2 PERFORMANCE OF FLOAT
text: Sampling λn. Unless stated otherwise, to evaluate the performance of FLOAT during validation we chose a set of λns as Sλn = {0.0, 0.2, 0.7, 1.0}. Note that setting λn to 0.0 or 1.0 corresponds to the values of λ used during training. Also, we measure the accuracy of FLOAT using two different settings of λth, 0.0 (similar to OAT) and 0.5. For λth = 0.5, we update the noise scaling factor by using the following simple equation
αlnew = { αl · 2 · λn; if λn ≤ 0.5 αl · 2 · (λn − 0.5); if 0.5 < λn ≤ 1.0
(5)
As depicted in Fig. 6 (a)-(e), the FLOAT models generalize well to yield a semi-continuous accuracy-robustness trade-off. Also, across all the datasets, λth = 0.5 yields a more gradual transition between the two boundary conditions. Consider the setting where λn = 0.2. With λth = 0.5, we observe a 4.95% improvement in CA and a reduction in RA of 12.37% on average over all five
3We followed the official repository https://github.com/VITA-Group/Once-for-All-Adversarial-Training
datasets when compared with λth = 0.0. The improvement in clean accuracy here can be attributed to the use of BNC . However, this configuration shows a drop in CA and an improvement in RA when compared to the configuration where λn = 0.0. This can be attributed to the use of noisy weights (refer to Eq. 5) during inference. Thus, it can be concluded that a user who cares more about clean image performance than adversarial robustness should set λth > 0.0 to see a less abrupt drop in CA. Note that, because the generation of adversarial images is noisy, it is not always true that increasing λ will always significantly improve robustness. Consequently, in some cases, we obtain improved clean image performance without a significant drop in robustness.

heading: 4.3 COMPARISON WITH OAT AND PGD-AT
text: We trained the models by following OAT and PGD-AT with λs sampled from a set Sλ = Sλn on three datasets, CIFAR-10, SVHN, and STL10.
Discussion on CA-RA trade-off. Fig. 7(a)-(c) show the comparison of FLOAT with OAT and PGD-AT in terms of CA-RA trade-offs. The FLOAT models show similar or superior performance at the boundary conditions as well as at intermediate sampled values of λ. In particular, compared to OAT and PGD-AT models, FLOAT models can provide an improved RA of up to 14.5% (STL10, λ = 0.2) and 34.92% (CIFAR-10, λ = 0.0), respectively. FLOAT also provides improved CA of up to 6.5% (STL10, λ = 1.0) and 6.96% (STL10, λ = 1.0), compared to OAT and PGD-AT generated models, respectively. Interestingly, for both FLOAT and OAT, in all the plots we generally see a sharp drop in robustness while moving from top-left to bottom-right. This can be attributed to to the switch from BNA to BNC based on the λth, in the forward pass of the inference model.
Discussion on training time and inference latency. Due to the presence of the additional FiLM
modules, OAT requires more time than standard PGD-AT to train. However, a single PGD-AT training can only provide a fixed accuracy-robustness trade-off. For example, to have trade-off with 4 different λs PGD-AT training time increases proportionally by a factor of 4. FLOAT, on the contrary, due to absence of additional layers, trains faster than OAT. In particular, Fig. 8(a) shows the normalized per-epoch training time (averaged over 200 epochs) of OAT and PGD-AT are, respectively, up to 1.43× and 1.37× slower than FLOAT. More results on FLOAT-OAT training time is provided in Appendix A.6.
Network latency increases with the increase in the number of layers for both standard and mobile GPUs Li et al. (2021); Singh et al. (2019), primarily because layers are operated on sequentially
Singh et al. (2019). The additional FiLM modules in OAT significantly increase the layer count. For example, for each bottleneck layer in ResNet34, OAT requires two FiLM modules, yielding a total of four additional FCs per bottleneck. On the other hand, FLOAT, similar to a single PGDAT trained model, requires no additional layers or associated latency, making it more attractive for real-time applications.
Discussion on model parameter storage cost. Unlike OAT, where the FiLM layer FCs significantly increase the parameter count, the additional BN layers and scaling factors of FLOAT represent a negligible increase in parameter count. In particular, assuming parameters are represented with 8- bits, a FLOAT ResNet34 has only 21.28 MB memory cost compared to 31.4MB for OAT. Fig. 8(b) shows that FLOAT models, similar to PGD-AT:1T, can yield up to 1.47× lower memory. Discussion on FLOPs. Compared to the standard PGD-AT, FLOAT incurs additional compute cost of addition of noise with the weight tensor during forward pass. For example, for ResNet34 with ∼21.28 M parameters, FLOAT needs similar number of additions for noisy weight transformation. However, compared to the total operations of ∼1.165 GFLOPs, the transformation adds on 1.182% additional computation. Moreover, as a single addition can be up to 32× cheaper than a single FLOP Horowitz (2014), we can gracefully ignore such transformation cost in terms of FLOPs. OAT, on the other hand, also incurs negligibly less FLOPs overhead of up to only ∼1.7% Wang et al. (2020).

heading: 4.4 PERFORMANCE OF FLOATS
text: To generate results with FLOATS we sample two representative slimming-factors 0.5 and 1.0 for training and validations with ResNet34 on CIFAR-10. We further trained a model with OATS to support above mentioned slimming-factors. As depicted in Fig. 9, FLOATS provides significantly improved performance through-out the sampling λ trade-off for both the slimming-factors. In particular, a FLOATS model can provide an improved CA and RA of up to 3.71% and 14.5%, respectively, compared to a model trained with OATS. Moreover, FLOATS enjoys 1.90× faster training time compared to OATS along with the reduced latency and param. requirement as mentioned earlier.

heading: 4.5 GENERALIZATION ON VARIOUS PERTURBATION TECHNIQUES
text: To demonstrate the generalization ability of FLOAT models on different attacks, we show their performance on images adversarially-perturbed through PGD-20 and FGSM attacks. We follow Wang et al. (2020) to generate the PGD20 perturbations and use the number of steps as 20, keeping other hyperparameters the same as PGD-7. For FGSM, we make = 8/255 following Wang et al. (2020). As shown in Fig. 10, under both the attacks, FLOAT can achieve insitu accuracy-robustness trade-offs similar to that with OAT. To further demonstrate FLOAT’s generalization ability on even stronger attacks, we have also analyzed its robustness with an ensemble of parameter-free attacks, namely autoattack Croce & Hein (2020). In particular, we have used the
‘random’ variant of the autoattack4. Details of the autoattack hyperparameter is provided in the Appendix. As depicted in 11, compared to the PGD-AT yielded models, FLOAT model consistently provide better RA with similar or improved CA.

heading: 5 CONCLUSIONS
text: This paper aims at addressing a largely unexplored problem of achieving in-situ inference trade-offs between accuracy, robustness, and complexity. We propose a fast learnable once-for-all adversarial training (FLOAT) which uses a novel form of model conditioning to capture the different featuremap distributions corresponding to clean and adversarial images. FLOAT transforms its weights using conditionally added scaled noise and dual batch normalization structures to distinguish between clean and adversarial images. The approach avoids increasing the layer count, unlike other state of the art alternatives, and thus does not suffer from increased network latency. We further show how FLOAT can be coupled with recently proposed slimmable networks to extend the in-situ trade-off to include model complexity. Extensive experiments show FLOAT’s superiority in terms of improved CA-RA performance, reduced parameter count, and faster training time.

heading: 6 BROADER IMPACT
text: DNNs are well-known to be susceptible to adversarial images Szegedy et al. (2013). As their use grows in various safety-critical applications, including autonomous driving Bojarski et al. (2016), medical imaging Han et al. (2021) and household robotics Tritschler (2021) achieving model robustness without sacrificing clean image accuracy is increasingly important. This is particularly important for scenarios that can change frequently such as in the case of a household robot. Moreover, the increasingly portable nature of these AI-enabled devices introduces stringent storage and energy-budget limitations. To address these challenging problems, this paper presents FLOAT models which can perform in-situ configuration of the DNNs and allow an application to dynamically adjust the model’s accuracy, robustness, and complexity based on the context and scenario. Our approach does not require iterative (re-)training as with standard PGD-AT and trains significantly faster than SOTA alternatives. Moreover, for inference-only devices, our approach circumvents the need for re-loading alternate model parameters from the cloud to support different scenarios, avoiding the potentially significant data-transfer costs. With the increase of high-stake real-world applications that require robustness, we believe this research will form the foundation for practical AI-driven applications that can efficiently adapt to their environment. Finally, we hope this paper will motivate further work aimed at enhancing the continuity of the accuracy-robustness trade-off and developing a theoretical basis that can explain the benefits and limitations of FLOAT and conditional learning in general.

heading: 7 REPRODUCIBILITY
text: We have detailed our training algorithms for both FLOAT and FLOATS in Algorithm 1 and 2, respectively. We have further provided detailed hyperparameter and model information in Section
4We have followed the official repo https://github.com/fra31/auto-attack, to generate the attack.
4.1 as well as in the Appendix A.1. Finally, the code for this project will be made publicly available shortly.

heading: A APPENDIX
text: A.1 HYPERPARAMETERS
We trained the models for 200 epochs with a starting learning rate (LR) of 0.1 and a cosine annealing LR scheduler Loshchilov & Hutter (2016). We used the SGD optimizer with a weight decay and momentum value of 5e − 4 and 0.9, respectively. For CIFAR-10, CIFAR-100, and SVHN we used a batch-size of 128 and for STL-10 which has a comparatively larger resolution (96 × 96 as image height/width compared to 32 × 32 of others), we used a batch-size of 64. We used half of each image-batch as clean and the remaining half as perturbed (using PGD-7 attack). For testtime random autoattack Croce & Hein (2020) generation we used L∞ norm with as 8/255. To reproduce the results with OAT and OATS we used the official repository of Wang et al. (2020) and used their recommended λ distribution, encoding dimension, sampling scheme as well as σ value for attack.
A.2 MODELS
We used ResNet34 to evaluate CIFAR-10 and CIFAR-100, WRN16-8 to evaluate SVHN, and WRN40-2 to evaluate STL10. Table 1 shows the models’ parameter and FLOP count (contributed by the convolutions and FC layers as they attribute to majority of the FLOPs and parameters).
A.3 TRAINED SCALING FACTORS
The trained noise scaling-factor αl for each layer of a FLOAT model is shown in Fig. 12. The αl values are generally high at the initial layers of the models while and reduce to near zero at later layers. He et al. (2019) has also observed a similar trend in trained alpha values while training targeting only robustness.
A.4 ALTERNATE RESULTS PLOTS IN TERMS OF λ VS. ACCURACY
To bolster Fig. 7 in Section 4, Fig. 13 show an alternate view of the CA-RA comparisons between FLOAT, OAT, and PGD-AT.
A.5 FLOAT PERFORMANCE ON CW ATTACK
Here we present FLOAT’s performance on Carlini and Wagner (CW) attack Carlini & Wagner (2017). In particular, we used CW∞ attack (optimized by PGD) with an attack step size of 20. As depicted in 14, the FLOAT yielded models consistently outperforms than that generated through PGD-AT. For example for λn = 0.0, FLOAT provides an improved robustness of up to 28.74% with similar clean image performance.
A.6 TRAINING TIME AS A FUNCTION OF SIZE OF Sλ USED FOR TRAINING.
The comparison of training times of OAT and FLOAT as a function of different numbers of λs is shown in Fig. 15. Because FLOAT always has two possible training λs, the time is constant. However, for OAT, the training time increases when supporting more training λs. Note, in Section 4, we report the training time comparisons for OAT training with four training λs (0.0, 0.2, 0.7, 1.0).
A.7 FLOAT SLIM TRAINING ALGORITHM
Algorithm 2: FLOATS Algorithm Data: Training setX∼D, model parameters Θ, trainable noise scaling factor α, binary conditioning
parameter λ, mini-batch size B, slimming-factor set Sf . 1 Output: trained model parameters Θ, α. 2 for i← 0 to to ep do 3 for j← 0 to nB do 4 Sample clean image-batch of size B/2 (X0:B/2, Y0:B/2) ∼D 5 for slimming-factor w in sorted Sf do 6 sampleSBN(w) //sample the BNC and BNA corresponding to w 7 LC ← computeLoss(X0:B/2,Θw, λ = 0,αw;Y0:B/2) 8 X̂B/2:B ← createAdv(XB/2:B,YB/2:B) // adversarial image creation 9 LA ← computeLoss(X̂B/2:B,Θw, λ = 1,αw;YB/2:B)
10 L ← 0.5 ∗ LC + 0.5 ∗ LA 11 accumulateGradient(L) 12 end 13 updateparam(Θ,α,∇L) 14 end 15 end

Review summaries:
This paper argues feature-wise linear modulation (FiLM) methods to perform in-situ calibration are too computationally expensive. Motivated by this, the authors propose the FLOAT method which adds scaled binary noise to the weight tensor instead of using extra layers. This paper also extends FLOAT to balance the 3-way-tradeoff between robustness / accuracy / complexity. The included experiments showcase the improvement in resolving the tradeoff between robustness and accuracy. 

This paper generalized the conventional once-for-all adversarial training (OAT) approach [Wang et al., 2020] to FLOAT, the fast learnable once-for-all adversarial training method. The key technical innovation lies at the incorporation of $\lambda$-conditioned noise into model weights so as to achieve the purpose of OAT but not request additional layers to perform conditioning. To demonstrate the effectiveness of the proposed approach, extensive experiments are provided across datasets and model architectures.  

This paper presents an algorithm named FLOAT(S), which adds conditional perturbation to weights during training (add noise when training with adversarial examples). This method immediately improves over its predecessor OAT by removing extra FiLM layers. As demonstrated in the experiments, FLOAT achieves better training efficiency than OAT, and perhaps more surprisingly, it has better robustness over OAT.

This paper tackles the problem of in-situ trade-off between model robustness and accuracy, which is of great importance in real-world applications. The proposed method, FLOAT, makes improvements on the previous state-of-the-art method OAT (NeurIPS'20) by using a more efficient model conditioning method. Experimental results show that FLOAT outperforms OAT in terms of accuracy, robustness and model size. The FLOPs of FLOAT, OAT, and the PGDAT baseline are almost the same.