The Effectiveness of Random Forgetting for Robust Generalization
Authors: Vijaya Raghavan T Ramkumar
Bahram Zonooz
Elahe Arani
Venue: ICLR 2024 poster (2024)
Keywords: Adversarial training
robust overfitting
forgetting
reinitialization
robust accuracy
generalization
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A method alternates between the forgetting phase, which randomly forgets a subset of weights and regulates the model's information through weight reinitialization, and the relearning phase, which mitigate robust overfitting.

Abstract: Deep neural networks are susceptible to adversarial attacks, which can compromise their performance and accuracy. Adversarial Training (AT) has emerged as a popular approach for protecting neural networks against such attacks. However, a key challenge of AT is robust overfitting, where the network's robust performance on test data deteriorates with further training, thus hindering generalization. Motivated by the concept of active forgetting in the brain, we introduce a novel learning paradigm called "Forget to Mitigate Overfitting (FOMO)". FOMO alternates between the forgetting phase, which randomly forgets a subset of weights and regulates the model's information through weight reinitialization, and the relearning phase, which emphasizes learning generalizable features. Our experiments on benchmark datasets and adversarial attacks show that FOMO alleviates robust overfitting by significantly reducing the gap between the best and last robust test accuracy while improving the state-of-the-art robustness. Furthermore, FOMO provides a better trade-off between the standard and robust accuracy outperforming baseline adversarial methods. Finally, our framework is robust to AutoAttacks and increases generalization in many real-world scenarios.

Published as a conference paper at ICLR 2024
THE EFFECTIVENESS OF RANDOM FORGETTING FOR ROBUST GENERALIZATION
Vijaya Raghavan T RRamkumrr' Bahram Zonooz 1. & Elahe Arani "Eindhoven University of Technology TTomTom 'Wayve raghavija jay95 gmail com, b. zonoozltue. ni, e e.arani@gmail.cmm
ABSTRACT
Deep neural networks susceptible to adversarial attacks, which compro- mise their performance and accuracy. Adversarial Training (AT) has emerged as a popular approach for protecting neural networks against such attacks. However, key challenge of AT robust overfitting, where the network's 's robust performance test data deteriorates with further training, thus hindering generalization. Mo- tivated by the concept of active forgetting in the brain, we introduce a novel learn- ing paradigm called "Forget Mitigate Overfitting (FOMO)". FOMO alternates between the forgetting phase, which randomly forgets a subset of weights and regulates the model's information through weight reinitialization, and the relearn- ing phase, which emphasizes learning generalizable features. Our experiments on benchmark datasets and adversarial attacks show that FOMO alleviates robust overfitting by significantly reducing the gap between the best and last robust test accuracy while improving the state-of-the-art robustness. Furthermore, FOMO provides better trade-off between standard and robust accuracy, outperforming baseline adversarial methods. Finally, our framework robust to AutoAttacks and increases generalization in many real-world scenarios
INTRODUCTION
Deep neural networks (DNNs) have demonstrated outstanding performance various domains, ranging from computer vision natural language processing and speech recognition. However, recent studies (Szegedy et al. 2013 Goodfellow et al.] (014)) have revealed the susceptibility of DNNs to adversarial attacks. These attacks initiated by adding small, yet intentionally crafted, imperceptible perturbations to input data, resulting in erroneous predictions by DNNs. The adver- sarial attacks pose serious security threat in applications such autonomous vehicles, medical diagnosis, and other areas where DNNs are used to make critical decisions. Adversarial Training (AT) (Madry al., 2017 Zhang et al., 2019a) has emerged as promising solution address the issue of robustness and security of DNNs against adversarial attacks. It involves training DNNs using adversarial examples to improve their resilience against such attacks.
Recently, Rice et al. (2020) reported "robust overfitting" in AT, where the robust performance on test data degrades with further training. (left) illustrates this phenomenon, where the adversarial test accuracy significantly lags behind the adversarial train accuracy, leading to robust overfitting. Although this phenomenon is present in AT, conventional methods to prevent benign overfitting in standard training, such explicit regularization and data augmentation, do not improve performance compared to the best accuracy achieved with early stopping in AT. While carly stopping is a useful technique to prevent robust overfitting, it may not be desirable due to the occurrence of the "double descent" " phennmmnon in AT (Rice et al. 2020 Nakkiran et al. 2002). Thus, the potential existence of robust overfitting in AT, and the failure of conventional methods to mitigate it, present a striking lacuna in building robust machine learning systems.
"Contributed equally.
I CCode available at https:s//// / thub. com www./Neuraaa. -
DDouble descent is phenomenon in deep learning where model's test error initially increases, decreases, and then increases again model complexity dataset size increases (Nakkiran al. 2021

---

Published as a conference paper at ICLR 2024
Figure 1: Comparison of robust overfitting in base- line PGD-AT (left) and our method FOMO (right,, highlighting the gap be- tween robust training and test accuracy on CIFAR-10 with
(Baseline)
(Ours)
50
Acc
Robust Train Ace
Robust Acc (Stable model)
Accuracy
60-
50-
//mm
~
FOMO significantly re- duces robust overfitting compared to the baseline's best early-stop checkpoint.
100
120 140
160
180
200
60
100
120
140
160
160
200
Epachs
Epochs
Unlike DNNs, humans excel in generalization in dynamic environments, facilitated by the interplay of remembering, forgetting. and relearning processes in the brain (Richards & Frankland 2017). As the eminent psychologist William James noted, "If we remembered everything, we should on most occasions be as ill off if we remembered nothing." Paradoxically, one condition for remembering is that we should forget. Similarly, our brain has the remarkable ability to remember and actively for- needed, which necessary for learning and achieving generalization. Although
get information the underlying mechanism of active forgetting remains elusive, neuroscience and cognitive psychol- ogy research (Gravitz, 2019) Izawa al.) 2019) provides growing evidence that the brain actively forgets by pruning neurons, shaping the learning-ment memory process (Shuai et al. 2010; Hardt et al.) 2013 Davis Zhong 2017 Richards & Frankland] 2017). Furthermore, the interaction between forgetting, remembering, and relearning is reinforced by the presence of multiple memory systems. Consolidation through long-term memory storage enables us to preserve crucial knowledge for fu- ture use and retrieval, while relearning reinforces previously learned information (Bjork & Allen 1970). Together, these three components work in harmony in the regulation of the learning process to achieve better generalizability in the real world. Therefore, emulating these aspects in DNNs might hold the key to achieving robust generalization in AT.
Therefore, propose general learning paradigm, which refer FOrgetting for Mitigating Overfatting (FOMO), to address the problem of robust overfitting of parameterized networks dur- ing AT. We consciously simulate the process of active forgetting in the DNNs by re-randomizing random subset of weights periodically during AT. Each forgetting phase is followed by a relearning phase, which call 'interleaved training'. Our method alternates between the forgetting and re- learning phases while consolidating generalized features. With extensive experiments on multiple datasets, we show that our proposed training paradigm boosts the robust performance and general- ization of AT models to a greater extent by alleviating robust overfitting. Our main contributions are as follows;
FOMO, adversarial training paradigm to improve the performance and generalization of DNNs through the lens active forgetting and relearning.
We demonstrate the efficacy of FOMO against the AutoAttacks.
Our method alleviates robust overfitting and achieves significant results across multiple architectures and datasets.
Our proposed training paradigm is robust to natural corruptions and leads to flatter minima.
RELATED WORK
2.1 ADVERSARIAL LEARNING
Adversarial training (AT) has been shown to be an effective method of countering adversarial at- tacks; thus we define adversarial learning settings as envisioned in Madry et al. (2017). We consider classification task with a given K-class dataset D {(2;, 2: FT X x Y, where E Rd in an i..d. manner, and represents an input sampled from a certain data- genteaagnnaaingg distribution
Y : 1, ... K represents set of possible class labels. Let fo E IRd be neural network modeled to predict classes. The notion of adversarial robustness requires fo to perform well not only on P but also on the worst-case distribution near P under a certain distance metric. More concretely, the
2

---

Published as a conference paper at ICLR 2024
adversarial robustness we primarily focus on in this paper is the ep-robustness: that is, for given p 2 and a small E > 0. we aim to train a classifier fo that correctly classifies æ + 8 for any | S é as y, where (zz y) ~ P.
In AT, the training data is sampled from adversarial regions incorporated to train the classifier. Madry et al. (2077) formulated AT' as a min-max optimization problem:
min Σ
Ladv (fo + v1). St
max
where fo is the DNN with parameters 0, and (.dv(.) is the typical classification loss function (e.g.. the cross-entropy (CE) loss). The inner maximization is to find an adversarial example zú that maximizes the loss. The outer minimization is to optimize network parameters 0 that minimize the loss on adversarial examples.
2.2 ROBUST GENERALIZATION.
Unlike in standard training, where longer training results in near-zero training and test error, Rice iet al. (2020) observed that the test error increases in adversarial training. This phenomenon is called robust overfitting. [Schmidt al. 2018) established that achieving an adversarially robust generalization is challenging and requires more training data. Zhhang et al. (2019b) pointed out the limitations of adversarial training blind spot attacks. Subsequently, several empirical approaches have been proposed to improve generalization. such as adversarial training with semi/unsupervieed learning (Carmon et al. [019 Zhai et al. a 2019 Zhang Xu 2019), AVmixup (Lee et al. 2020), and robust local feature (Chen et al. 2021). However, we distance ourselves from these works, as these data interpolation methods rely heavily on the requirement of large datasets to mitigate overfitting.
In contrast, Rice et al. (2020) systematically investigated various techniques used in deep learning, including & and f2 regularization, cutout, mixup, and early stopping, and found that early stop- ping was the most effective approach remedy robust overfitting. While early stopping may not be the optimal solution for robust overfitting, other approaches mitigate this by promoting model flatness (Wu et al.
2020 Chen al., 2020). Chen al. (2020) achieves model flatness by lever- aging knowledge distillation to smooth the logits space while applying stochastic weight averaging (Izmailov et al. 2018) to smoothen the weights space. However, their method is computationally
expensive as it necessitates pre- pre-training additional models to mitigate overfitting. Dong et al. (2021) incorporates the temporal ensembling (TE) technique (Laine Aila 2016) into the AT Trameworks to regularize the predictions of adversarial examples from becoming overconfident. On the other hand, Adversarial Weight Perturbation (AWP) (Wu et al., 2020) explicitly regularizes the flatness of the weight loss landscape by proposing double-perturbation mechanism that adversarially per- turbs both inputs and weights. However, AWP requires an additional maximization step to compute the adversarial noise to perturb the network weights during AT. IDBH (Li & Spratling 2023) un- derscores the importance of diversity and hardness in data augmentation, showing that diversity improves accuracy and robustness, while hardness enhances robustness in adversarial training. Un- like the other approaches, we tackle the problem of robust overfitting in AT through the lens of active forgetting. Given the ability of DNNs to memorize noise in the training data (Dong et al.) [0211) we hypothesize that actively forgetting and relearning during AT may help consolidate generalizable features and achieve robust generalization.
2.3 BIOLOGICAL UNDERPINNINGS FOR ROBUST GENERAIIZATION
We begin by motivating our approach through an examination of learning dynamics within the hu- man brain. Intelligent decision-male makinn in noisy, dynamic environments emerges from the interplay between memory retention and forgetting mechanisms. As highlighted by Davis & Zhong (2017), the ability of humans to generalize from new experiences is, in part, attributable to the phenomenon of active forgetting. Active forgetting plays a pivotal role in the selective regulation and rebalanc- ing of the learning-mentt memory process, thereby guarding against overfitting to individual experiences (Gravitz] 2019). These insights from neuroscience provide substantial evidence for the existence of a symbiotic relationship between generalization and active forgetting in biological neural networks, relationship that remains notably absent in DNNs.
3

---

Published as a conference paper at ICLR 2024
opoch
Leats
Cor
Stari
(a) Learning
(b) Consolidation
(c) Forgetting
(d) Relearning
Trained Parameters
Exponential Moving Average weights
Reinitialized parameters (Forgetting)
Figure 2: Schematics of the proposed FOMO framework illustrating its three pivotal phases during the AT. Beginning with standard learning, FOMO sequentially incorporates consolidation, unique forgetting phase, and relearning stage. This cyclic process enhances the robustness of the model by addressing adversarial overfitting through active forgetting and relearning.
In line with this, Richards & Frankland (0117) present compelling evidence supporting the essential role of forgetting f'acilitating adaptive behavior within dynamic settings. Active forgetting within memory models offers functional advantages for robust generalization in such contexts, including (1) bolstering behavioral flexibility by diminishing the impact of obsolete information on memory- driven decision-making, and (2) averting overfitting to past events, thereby fostering generalization. Consequently, we underscore the importance of incorporating active forgetting mechanisms into computational models to effectively mitigate robust overfitting and promote robust generalization.
METHODOLOGY
We propose FOrget to Mitigate Overfitting (FOMO), a training paradigm improve generalization in adversarial learning. FOMO interchanges between the consolidation, forgetting, and relearning phases. More details can be found FigureZJan in Algorithm [ in Appendix
3.1 FORGETTING
What is forgetting? We define the "forgetting step" as any process that results in a reduction in robust training accuracy. Specifically, let denote the training dataset as D and neural net- work parameterized by as So. The robust training accuracy of the network So expressed as AccR(fe) 1 for a randomly initialized neural network (f%) on the dataset D. We define a forgetting step as any function satisfying two conditions: (i) P(C < Accn(((0) Accn(f0))
8) y4- Now, consider the robust chance accuracy denoted C
1, ensuring that the forgetting stage allows relearing due to a decrease accuracy, and (ii) mutual information [ff; D) > 0 for the network concerning the dataset remains positive after the forgetting step. This concept captures the notion that forgetting entails the partial removal of information from the net- work, not complete erasure. Put simply, after any forgetting operation, the training accuracy of the adversarially trained network should fall between the chance accuracy and the robust accuracy achieved before forgetting. Thus, the forgetting step aims to strike balance between preserving essential information from the previous step and facilitating relearning.
Where to forget? Now that we have established the process of forgetting, the subsequent ques- tion arises: Where in the neural architecture should forgetting take place to achieve generalization? Although our goal is to mitigate overfitting in AT, our aim is to prioritize characteristics that are simple, general, and beneficial for generalization (Valle-Perez et al. 2018). Therefore, we limit the forgetting process to the later layers the network. For example, in the case PreAct-ResNetll (He et al.) 2016), the last two layers are considered later layers. This is because these layers have more capacity to memorize and tend to overfit the training data, while the earlier layers in the network tend learn generalized representations (Dong et al. 2021 Neyshabur et al. 2018
4

---

Published as a conference paper at ICLR 2024
Arpit al., 2017 Yosinski et al.) 2014 Zhang al. 2021 Geirhos et al. 2018). Furthermore, studies in AT such as (Bakiskan et al. |022 Siddiqui & Breuel 2021) have shown that the network depth has significant impact adversarial robustness, and early layers tend to contribute more to robustness than later layers. Therefore, by limiting the forgetting process to the last two layers, we aim to regularize the weights in the later layers while retaining more generalized features learned in the earlier layers to facilitate robust generalization.
How to forget? Having established the concept of forgetting and identified where in a neural net- work to apply forgetting, the next critical step is to determine how to effectively emulate the for- getting process DNNs. To this end, we consider a deep network (fo) with layers, each with parameters fj. Initially, the network undergoes ad adversarial training during a warm-up period, em- ploying the adversarial example generation technique outlined in Section 2.1
Our forgetting approach starts by splitting the network fo into two hypotheses during the AT as outlined by binary mask M: (a) Retain hypothesis F2, and (b) Reset hypothesis Fg. The network parameters Or belonging to the retain and reset hypotheses are selected randomly using binary mask M, such that sumn( (M) = 8 * [0r], where & is the sparsity rate that determines the percentage of parameters belonging to the reset hypothesis Fg. We prefer random selection as it is simple and efficient. Note that the forgetting step applies only to the parameters of the later layers defined by the layer threshold L. Therefore, by default, the parameters O corresponding to the early layers of the network belong to the retain hypothesis FA throughout the AT. Finally, these hypotheses are outlined by binary mask; for FA and 0 for Fv, that is, FA = M O fo and Fg = (1 M) so-
After randomly selecting the subset of weights belonging to the retain and reset hypothesis, the parameters belonging FA from the previous learning step are retained, while F& is reinitialized or reset to a random value sampled from a uniform distribution. Formally, we reinitialize parameters in the layer, as follows 6; Mi @ 02 + (1 Mi) @ By.. where 6r. is randomly initialized tensor. Thus, we emulate the aspect of active forgetting through random reinitializatton of the connections and regularize the parameters the later layers throughout the AT to achieve robust generalization.
Finally, the new network, after forgetting, contains an amalgamation of retained and reinitialized parameters that undergo relearning until the onset of the next forgetting step.
3.2 CONSOLIDATION
Since our FOMO method alternates between the forgetting and relearning phases during the AT, it is important to assimilate the generalized information that is relearned after each forgetting step. As the information is encoded in the parameters of the network (Krishnan et al., 2019). we intend to consolidate this information after each relearning step by employing another network fo called stable model similar to fa. The knowledge learned by Se is consolidated in the stable model after each learning session (before each forgetting step), thereby serving as a long-term memory. This newly learned information is consolidated in the stable model by taking an exponential moving average of the so weights with decay parameter az: Φ = a.. + (1 - -o,)0, where Φ and @ correspond to the weights of the stable and the current network, respectively. It should be noted that the stable model is not subjected to training. while the forgetting operation is exclusively applied to fe. Thus, this consolidation step can be considered as forming a self-ensemble of the intermediate model states obtained after multiple relearning steps that leads to generalized representations.
3.3 RELEARNING
Recent studies in cognitive neuroscience provide significant evidence of the existence of a symbiotic relationship between learning and forgetting (Graviiz) 2019; Richards & Frankland] 2017). The "spacing effect" is a well-known example of this relationship. which shows that long-term recall is improved when learning sessions are spaced out rather than massed together (Bjork & Bjork 2019). Also, BBjork& & Allen (970) showed that the reduced accessibility of information between learning sessions is the key to regulating the important information in long-term memory. Therefore, we exploit this symbiotic relationship in AT by introducing interleaved training session after each forgetting step. The network with new initialization undergoes a relearning phase wherein it is trained adversarially for e, epochs, where e, is less than the total number of AT epochs.
5

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison the CIFAR-10 using the PreActResNet-18 and NeteResNet-
34-10 architectures under a perturbation norm of Coo 8/ 8/255.
ProActResNet-18
WidcReaNet-34-11
Method
Natural
PGD-20
Trade-off
PGD-20
Trade-off
Bost
Last
à
Best
Last
A
Best
Last
A
Bost
Last
A
PGD-AT
82.08
83.98
1.90
52.32
44.44
-7.88
58.12
86.90
86,38
-0.52
56.45
48.16
-8.29
61.84
TRADES
80.72
82.61
1.89
52.66
49.75
-2.91
62.10
84.73
84.62
-0.11
56.50
47.28
-9.22
60.66
KD+SWA
83.82
84.43
0.61
54.59
54.42
-0.17
66.18
86.85
88.03
1.18
56,92
55.74
-1.18
68.25
PGD-AT+TE
82.15
82.59
0.44
55.03
53.79
-1.24
65.14
86.20
85.63
-0.57
56.89
53.49
-3.4
65,84
AWP
81.25
81.56
0.21
55.39
54.73
-0.66
65.50
86,28
86.27
-0.01
58.85
58.76
-0.09
69,90
TOMO (Ours)
81.84
82.51
0.67
56.68
56.46
0.22
67.04
87.31
87.08
.0.23
59.69
59.23
-0.46
70,50
Table 2: Performance comparison on CIFAR-100 and SVHN datasets, using the PreActResNer18
architecture and perturbation norm of Co 8/255.
CHFAR-10U
SVHN
Method
PGD-20
Trade-off
Trade-off
Best
Last
Best
Last
A
Best
Last
A
Best
Last
A
PGD-AT
55.52
57.35
1.83
27.22
20.82
-6.4
30.54
87.93
89.90
-1.93
52.60
45.
-7.47
60.09
TRADES
35.53
-3.48
35.80
52.50
47.50
62.48
KD+SWA
57.23
57.66
0.43
30.06
30.02
-0.04
39.48
90.40
91.70
1.30
53.65
50.65
-3.00
65.25
PGD-AI-TE
56.52
-2.67
66.30
AWP
53.92
54.81
-0.89
30.70
30.28
-0.42
39.00
93.85
92.59
-1.26
59.12
55.87
-3.25
69.68
POMO
57.45
57.07
32.07
31.67
-0.40
59.63
72.44
Regularization. To expedite the relearning process, we propose a consistency loss that regularizes the output of the stable model. As shown Equation [ the function of this consistency regulariza- tion is to provide guidance to fe after each forgetting step. \1 and X2 in the Equation 2 denotes loss balancing parameters. The network (fo) () updated so that it acquires new knowledge while aligning its decision boundary with the stable model that contains information consolidated across multiple relearning steps. This further prevents fe from robust overfitting during the relearning phase. The overall loss LFOMO used to train the network during relearning phase shown in Equation 3 We also detach the gradients from the stable model. Thus this regularization provides more detailed supervision from stable model than Loe loss, which helps avoid overfitting. Once the network completes the relearning phase, the newly acquired knowledge is integrated into the stable model, which then proceeds to the forgetting step.
LCR U1 -KL((f(=))llf((e)) + d2 DKI(fol# + 601lfo( 6))
LFOMo(I, y: Cade (fo(r 6), + LCR
(3)
Thus, by cycling through forgetting, relearning, and consolidation, we introduce behavioral flexibil- ity that helps in learning generalized information. Finally, for inference, rely on the stable model, which serves as a long-term memory and holds the generalized knowledge that is consolidated after multiple relearning steps during the AT.
TACKLING ROBUST OVERFITTING
We compared our proposed method against previous AT methods on the CIFAR-10 dataset, utilizing two popular backbone networks, namely PreAct-ResNet18 (He et al. 2016) and WideResNNt-344 10 (Zagoruyko Komodakis 2016). The performance of each approach was evaluated on the test set against the PGD-20 attack. As presented in Table [ our experimental results demonstrate that FOMO outperforms all the baseline approaches regarding the best and last robust test aceu- racy for all architectures. Specifically, our method achieves significant improvement of 12.02% and 11.0% in last-epoch robust test accuracy over PGD-AT on 34-10, respectively. Notably, the last-epoch robustness of FOMO even consistently outperforms the best-epoch robustness of previous approaches that employ early stopping. Thus, early stopping may not be necessary, saving computation that goes into validating every epoch during AT. More- over, FOMO achieves better trade-off between standard and robust generalization than many AT methods. Lastly, FOMO mitigates robust overfitting by reducing the gap between the best and last
and WideResNet-

---

Published as a conference paper at ICLR 2024
Table 3: White-box/Black-boox (Auto-attack) performance comparison on CIFAR-10 and CIFAR-
100, using the PreActResNet-18 architecture and a perturbation norm of exo 8/255.
CIFAR-I0
CIFAR-IOR
Method
Best
Last
A
Best
Last
A
ICLR'18
PGD-AT
47.72
42,60
-5.12
24.53
20.21
-4.32
ICML'19
TRADES
48.37
46.94
-1.43
24.51
22.86
-1.65
NeurlPS'20 20
AWP
50.34
49.64
-0.70
25.26
25.07
-0.19
ICLR'21
KD+SWA
49.87
49.74
-0.13
26.04
25.99
-0.05
ICLR'22
PGD-AT+TB
50.11
49.14
-0.97
26.04
25.13
-0.91
ICML'22
MLCATwP
50.70
50.32
-0.38
25.86
25.18
-0.68
ICLR'23
IDBH[Strong]
50.74
49.99
-0.75
ICLR'24
FOMO
51.37
51.28
-0.09
27.57
27.49
-0.08
robust test accuracy. Therefore, iterating between forgetting, relearning, and consolidation learns the generalized features that facilitate robust generalization AT.
Performance across different datasets. To further evaluate the scalability of our proposed method, we conducted experiments on two additional benchmark datasets, CIFAR-100 and SVHN, which are more complex than CIFAR-10. The results, presented in Table indicate that our method achieves the highest level of robustness in both the best and the final epoch, demonstrating its ability to effec- tively scale to larger datasets compared other baselines. Thus, selectively forgetting information periodically through weight reinitialization regulates the weights and brings discernable benefits to the model regarding robust generalization.
TRAINING ROBUST ACCURACY
Here, study the impact of FOMO on robust training accuracy. The variation in robust accuracy on the training data during AT is shown in Figure [ Our proposed method (FOMO) effectively suppresses the robust training accuracy from the level attained by PGD-AT (from 92% to 69%). This illustrates that randomly forgetting a percentage of parameters in the later layers during AT inhibits the model from overfitting to the training data, leading significant reduction in the robust generalization gap (from 47.56% to approximately 12.54%) and, thus, mitigating robust overfitting.
EVALUATION WITH AUTOATTACK
DNNs deployed in real-world settings where they face more realistic and challenging scenarios, including sophisticated attacks that can target the model. It is, therefore, essential to evaluate pro- posed approaches against strong attacks that effectively compromise the model's 's robustness. By doing so, we can ensure that the adversarial methods can effectively enhance the model's security and resilience in real- world settings. Recently, Auto/ttack (Croce & Hein (020) has been more effective in uncovering vulnerabilities in DNNs, making it popular choice for evaluating the ro- ensemble of several state-of-the-art white-box and black-box attacks
bustness of models. It to generate adversarial examples that are more transferable and harder to defend against. As shown in Table we evaluate our proposed method against the Attoct on CIFAR- and CIFAR-100 datasets with the PreActResNett-1 architecture. Compared to the standard AT (PGD-AT) and other robust generalization methods, our approach largely alleviates robust overfitting under Auto-Attack on both datasets. These results indicate that training with FOMO leads to robust, consistently gen- eralizable features across challenging adversarial attacks.
7 ROBUSTNESS AGAINST COMMON CORRUPTIONS
DNNs are commonly deployed in real-world settings, where they are exposed to dynamic environ- ments influenced by factors such lighting and weather. As result, it is crucial to assess the robustness of DNNs to handle data distributions susceptible to natural corruption in real-world set- tings. Here, evaluated the efficacy of our method the corrupted CIFAR-10 dataset (Hendrycks)
7

---

Published as a conference paper at ICLR 2024
Adversarial perturbation
Robustness to natural corruptions
PGD-AT
AWP FOMO
80
Adversaria Accuracy
60
65
Attack Strength (Epsilon)
Figure 3: (Left) Robustness to adversarial attacks; (right) Robustness to Natural corupption. In both robustness analyses, FOMO shows significant performance improvement compared to the baselines considered.
& Dietterich 2019), which includes 19 types of corruptions. Models are trained on clean images and tested on CIFAR-1O-C. The mean corruption accuracy (mCA) of each method presented in Figure (Right). The results reveal that the mCA consistently improves with FOMO compared to PGD-AT across corruptions. Periodically iterating between forgetting, relearning, and consolidation during AT brings discernible benefits regarding robustness to natural corruptions.
ROBUSTNESS TO INCREASE IN ADVERSARIAL ATTACK STRENGTH
To further assess the efficacy, we conducted experiments employing PGD-20 attack with perturba- tion strengths incrementally spanning from an c value of 0.25/255 8/255. The outcomes, depicted in Figure 3 (Left), reveal pronounced decline in PGD-AT's robustness perturbation strength increases, whereas FOMO consistently surpasses baseline adversarial techniques across strength levels, showcasing its stable performance. These results suggest that the process of forgetting and re- learning within the FOMO framework consolidates high-level abstractions capable withstanding minor data perturbations, in contrast to standard AT (PGD-AT) and AWP methodologies.
CONVERGENCE TO FLATTER MINIMA
DNNs that converge flatter minima in the loss landscape demonstrate superior generaliza- tion. according to Neyshabur et al. 2018). Ad- ditionally, models that reside flatter minima resilient since slight perturbations do not significantly affect their predictions. To evaluate the robustness of our method, in-
100
FOMO
PGD AT
Top-1 train tr-in Accuracy (%)
80
AWP
60
corporate independent Gaussian noise into all
40
parameters of the trained CIFAR-I0 model,
outlined in Alabdullmossin et al. (202I).
Figure 4 illustrates that the solution achieved by FOMO exhibits greater robustness to model perturbations compared standard AT. Our method demonstrates significantly reduced sen- sitivity to perturbations. Specifically, for any level of noise introduced into the model pa- rameters 6, FOMO training accuracy exhibits smaller variations than standard training, sug- gesting that the FOMO solution may reside within flatter local minima. We posit that the training regimen involving alternating stages of for-
Perturbation (ax )
Figure 4: Robustness of the model perturbed by varying degrees of Gaussian noise. Our method is considerably robust to Gaussian perturbations, as the decline in performance is gradual, suggesting convergence to flatter minima.
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study of our proposed method on the CIFAR-10 dataset using PreActRes 18 with a perturbation norm of Eoc 8/255. The numbers in green represent the relative gain of each step compared to the previous one.
PGD-20
AA
Method
Best
Last
A
Rel. Gain
Best
Last
A
Rel. Gain
PGD-AT
52.32
44.44
-7.88
47.72
42.60
-5.12
+ Forgetting
52.32
48.86
-3,46
156.12%
47.72
44.63
-3.09
139. 65%
+ Consolidation
55.41
52.80
-2.61
124.57%
50.89
49.30
-1.59
148.54%
+ Regularization
56.68
56.46
-0.22
191.57%%
51.37
51.28
-0.09
194.34%
getting and relearning induces broader valley, potentially elucidating our model's capacity solidate generalizable features.
10 ABLATION
We analyze the effect of forgetting, consolidation, and consistency regularization on the overall performance by incrementally adding each component to the baseline AT (PGD-AT), Trepors the robust performance of the PreActResNet-IM 18 network against PGD-20 and AA on the CIFAR-10 dataset.
Firstly, we evaluate the effect of forgetting by comparing the performance of our method with and without forgetting (PGD-AT). We periodically forget 3.5% parameters corresponding the param- eters in the later layers ( 3 and 4 in PreAct-ResNet every five epochs. Our observations indicate substantial reduction the gap between the best and last robust test accuracy when the forgetting component is incorporated, compared to PGD-AT. Introducing the forgetting component aids in the regularization of the weights and facilitates the release of the network's capacity learn generalized features during relearning.
Furthermore, we extend our investigation by introducing a stable model for consolidating the fea- tures. Our findings indicate notable improvement (approximately 3%) in the best test robust curacy compared to PGD-AT. This improvement is attributed to the ability of the stable model to consolidate critical information learned during each releaming phase.
Lastly, we assess the role of the consistency regularization loss on the AT. Our observations re- veal that the loss component is vital in mitigating robust overfitting by decreasing the gap between the best and last accuracy to -0.22. Thus, consistency regularization guides the network after each forgetting step by providing more precise supervision from a stable model, which helps prevent overfitting to training labels. This further incentivizes the learning model to acquire generalized features consolidated in the stable model the end of each relearning phase, resulting an im- proved final performance. Our ablation study indicates that all individual components included in our proposed method are essential for robust generalization.
CONCLUSION
We introduce Forget to Mitigate Overfitting (FOMO), adversarial training paradigm, improve DNN performance and generalization through the lens of active forgetting. FOMO alternates be- tween the forgetting phase, which periodically forgets undesirable information in the model through the reinitialization of weights, and the relearning phase, which emphasizes learning generalizable features. These features are later consolidated into stable model after each relearning phase. Em- pirical results show that the proposed framework improves both standard and robust performance and generalization across a wide range architectures, datasets, and perturbation types. Our framework robust to auto attacks and increases generalization in many real-world scenarios. Overall, FOMO presents a promising solution for achieving better robust generalization in adversarial training. In future work, will aim to further develop theoretical understanding of this issue and how it relates to the effectiveness of AT, as the underlying cause of this robust overfitting phenomenon is not yet fully understood.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization on generalization in convolutional neural networks. arXiv preprint arXiv:2109.0022 2021.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International conference machine learning, pp- 233-242, PMLR, 2017.
Can Bakiskan, Metehan Cekic, and Upamanyu Madhow. Early layers more important for adver- sarial robustness. In ICLR 2022 Workshop on New Frontiers in Adversarial Machine Learning, 2022.
Robert A Bjork and Ted W Allen. The spacing effect: Consolidation or differential encoding? Journal of Verbul Learning and Verbul Behavior, 95):5677577, 1970.
Robert A Bjork and Elizabeth L Bjork. Forgetting as the friend of learning: Implications for teaching and self-regulated learning. Advances Physiology Education, 1342):164-167, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. leee, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Chen Chen, Jingfeng Zhang, Xilie Xu, Tianlei Hu, Gang Niu, Gang Chen, and Masashi Sugiyama. Guided interpolation for adversarial training. arXiv preprint arXiv:2102.07337, 2021.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020.
Ronald Davis and Yi Zhong. The biology of forgetting-aa perspective. Neuron, 95(3):990-503 2017.
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang. Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. arXiv preprint arXiv:2106.01606, 2021.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6572. 2014.
Lauren Gravitz. The forgotten part of memory. Nature, 571(7766):512-S1) -12,1122--2222211 2019.
Oliver Hardt, Karim Nader, and Lynn Nadel. Decay happens: the role of active forgetting in memory. Trends in cognitive sciences, 17(3):111-22) 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. /2261, 2019.
10

---

Published as a conference paper at ICLR 2024
Shuntaro Izawa, Srikanta Chowdhury, Toh Miyazaki, Yasutaka Mukai, Daisuke Ono, Ryo Inoue, Yu Ohmura, Hiroyuki Mizoguchi, Kazuhiro Kimura, Mitsuhiro Yoshioka, et al. Rem sleep- active mch neurons are involved in forgetting hippocumpus-deeendeen memories. Science, 365(6459):
1308-1313, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05447,.2 2018.
Giri P Krishnan, Timothy Tadros, Ramyaa Ramyaa, and Maxim Bazhenov. Biologically inspired sleep algorithm for artificial neural networks. arXiv preprint ExXiv1900.02220 02240, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for sune-supeeeieed learning. arXiv preprint arXiv:1610.0224. 2016.
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 272-281, 2020.
Lin Li and Michael W. Spratling. Data augmentation alone can improve adversarial training. In
The Eleventh International Conference Learning Representations, 2023. URL https://
openreview net met/foounm Add
Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13409-13411 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.060m 2017.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 212: 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of over paramertmtiiinnioo in generalization of neural networks. arXiv preprint Xrv: 1805. 1e..11105...7.7 12076, 2018
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104. PMLR, 2020.
Blake A Richards and Paul W Frankland. The persistence and transience of memory. Neuron, 94 107:10714 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neurul information processing systems, 31, 2018
Yichun Shuai, Binyan Lu, Ying Hu, Lianzhang Wang, Kan Sun, and Yi Zhong. Forgetting is regu- lated through rac activity in drosophila. Cell, 140(4):579 2010.
Shoaib Ahmed Siddiqui and Thomas Breuel. Identifying layers susceptible to adversarial attacks. arXiv preprint arXiv: 1nto::21777772 2021.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,6/99 /99 2013.

---

Published as a conference paper at ICLR 2024
Guillermo Valle-Perez, Chico Camargo, and Ard A Louis. Deep learning generalizes because the parameter-fume funnctioon map is biased towards simple functions. arXiv preprint arXiv: (rXiv:880500822. 2018.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in Neural Information Processing Systems, 33:2958-2969, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Chaojian Yu, Bo Han, Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Under-
standing robust overfitting of adversarial training and beyond. In International Conference Machine Learning, pp. 25595-25610. PMLR, 2022.
Sergey Zagoruyko and Nikos Komodakis. 1r5iv:1605.07146 2016.
Wide residual networks.
arXiv preprint
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hoperoft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv: (cXiv:1906.0055: 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Haichao Zhang and Wei Xu. Adversarial interpolation training: A simple approach for improving model robustness. 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482. PMLR, 2019a.
Huan Zhang, Hongge Chen, Zhao Song. Duane Boning, Inderjit Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. arXiv preprint arXiv: 19901.00844. 2019b.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
Algorithm Adversarial Training with FOMO
Require:
Training data = (x, )i I", number of training epochs N, batch size B, base adversarial training algorithm A with e, perturbation norm and steps, network fe. stable model fø. warm-up epochs ewarm-uwy number of epochs the network undergoes relearning ey, decay parameter dee sparsity rate that determines the percentage of parameters to be forgotten in the later layers defined by layer threshold L. Initialize model parameters
2: for epoch + to N do
3:
if epoch Cwarm- up and epoch == then
4:
Consolidate(fa. fo. ac))
5:
Random forgetting(fe. s)
b Randomly reinitialize the parameters in the later layers
6:
Sample a mini-batch B = (æz, y4)
from D
7:
B A(fe, Ladus e, steps, norm)
b adv samples > forward pass with adv samples b forward pass with std samples
8:
B' fo(B)
9:
B' + (B(
10:
if epoch Cwarm-un then
3 + (B)
forward pass using stable model
12:
8" - (B(
LCR = \1 (B + 22 (KI(B')1"))
14:
LFOMO = Ladv + LOR
>> refer Eq
2 & Eq
else
16:
PFOMO = Lado
17:
Compute the gradients
Update the parameters fo
A.1 IMPLEMENTATION DETAILS
We follow the standard adversarial training (AT) procedure used in previous research (Wu et al. 2020). The model was trained for a total of 200 epochs using the stochastic gradient descent (SGD) optimization algorithm with a momentum of 0.9, weight decay of 5 10 and an initial learning rate of 0.1. For standard AT, we reduced the learning rate by a factor of 10 at the 100 and 150th epochs, respectively. We applied standard data augmentation techniques, including random cropping with 4-pixel padding and random horizontal flipping, to the CIFAR-10 10 and CIFAR- 100 datasets, while no data augmentation was applied to the SVHN dataset.
For training the other baseline methods (Wu et al. 2020 Chen et al. 2020 Dong al. 2021) used the exact same procedure and hyperparameters as specified in those methods. For the FOMO method proposed in Section 3 began at epoch 105 (ewarm-up), little later than the first LR decay where robust overfitting often occurs. For Pre we forgot fixed = 3.5% of the parameters in the later layers (Block-3 and Block-4) of the architecture, while for WiderResNet- 34-10, we forgot 5% as it has larger capacity to memorize. Each forgetting step was followed by relearning phase that lasted for e, Section A.6 For the consolidation step, we chose a decay rate of the stable model of -__ = 0.999. During relearning phase, the stable model through the regularization loss (Len), and we chose regularization strengths of λ1 and X2 equal to 1. We ran the experiments for three seeds, and the average the results reported in the table.
epochs. The relationship between s and e, studied in
Datasets. For our experiments, we use three datasets: CIFAR-10 (Krizhevsky et al. 2009 CIFAR-
100 (Krizhevsky et al. 2009). SVHN (Netzer al., (01T). We randomly split the original training sets for these datasets into training set and validation set in 9:1 ratio. Our ablation studies and visualizations are mainly based on the CIFAR- dataset.
Baseline. We compare the results against various baseline methods such as vanilla PGD-AT Madry
et al. 2017). TRADES (Zhang et al. 2019a), KD+SWA (Chen et al. 2020), PGD-AT+TE (Dong al. 2021), AWP Wu al. 2020) that are proposed to mitigate robust overfitting. To assess the model 's robust overfitting ability, we compare the robust test accuracy between the best-epoch and the last-epoch. The difference between the best and the final robust test accuracy is denoted A.
13

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of test robustness (%) between MLCAT and FOMO under Autoattack.
CIFAR-10
CIFAR-100
Method
Best
Last
A
Best
Last
A
MLCAT 1.8
28.12
27.03
-1.29
13.41
11.37
-2.04
MLCAT w
50.70
50.32
-0.38
25.86
25.18
-0.68
FOMO
51.37
51.28
-0.09
27.57
27.49
-0.08
Table 6: Auto-attack FOMO using the PreActResNet-18..
CIFAR-10
CIFAR-100 100
Tiny- Ima-lmmgeNet
Method
Best
Last
Best
Last
Best
Last
WA
49.92
43.82
25.95
21.02
19.76
15.82
KD+SWA
49.87
49.74
26.04
25.99
19.78
19.76
PGD-AT+TE
50.11
49.14
26.04
25.13
18.16
15.88
FOMO
51.37
51.28
27.57
27.49
20.23
19.85
The results averaged three seeds. In addition to robust overfitting, achieving optimal balance between natural and robust test accuracy crucial for an effective AT method. However, there is currently standardized method for measuring this trade-off in the adversarial trade-off literature. l'herefore, we propose trade-off measure that offers a formal approach to compare how well different methods maintain this balance. The Trade-off is measured follows:
x As, x R.A.
Trade-off = NAL + RAA
(4)
where NAL and RAA, stand for last-epoch natural test accuracy and robust test accuracy, respec- tively.
A.2 COMPARISON WITH MLCAT
Yu et al. (2022) proposed method called MLCAT alleviate robust overfitting by analyzing the roles of easy and hard samples and regularizing samples with small loss values using non non-robust features. In contrast, we approach this problem from the lens of active forgetting, which provides different perspective for addressing robust overfitting.
According to the results presented in Table 5 our FOMO approach outperforms MICAT consis- tently in terms of robust improvement against AutoAttack across datasets. Furthermore, FOMO exhibits lower levels of robust overfitting compared both variants of MLCAT. MICAT corre- sponds to loss scaling, and MLCATwP corresponds weight perturbation. As result, FOMO is not only more effective than MLCAT, but also shows a reduced tendency toward overfitting. Thus, iterating periodically between forgetting, relearning, and consolidation during AT brings discernible benefits with respect to the robustness to AT.
A.3 EXTENDED COMPARISON WITH BASELINES ON LARGE DATASETS
We conducted extensive comparisons with KD-SWA, weight averaging (WA), and model ensem- bling approaches CIFAR10/100 SVHN, and Tiny-Image.4 consisting of 64x64 images. FOMO algorithm outperforms these baselines. demonstrating superior robustness and im- proved perfor- mance as shown in Table 6 Furthermore, as shown in Table 7 FOMO outperforms subspace ad- versarial training (Sub-AT) et 2022). This shows the effectiveness of FOMO even on larger datasets.
A.4 EVALUATION ACROSS PERTURBATIONS
We assess the versatility of our proposed FOMO framework by evaluating across perturbation (E, norm using 18 He et al. (0016) CIFAR-10. Table demonstrates the best and
14

---

Published as a conference paper at ICLR 2024
Table 7: PGD-attack on FOMO against Subspace AT.
CIFAR-10
CIFAR- 100
Method
Best
Last
A
Best
Last
A
Sub-AT
52.79
52.31
0.48
27.50
27.02
0.48
FOMO
56.68
56.46
-0.22
32.07
31.67
-0.40
Table 8: Test robustness (%) of AT and FOMO across different datasets and threat models.
CIFAR-10
CIFAR-100
Threat Model
Method
Best
Last
Best
Last
A
PGD-AT
52.32
44.44
-7.88
27.22
20.82
-6.4
t0
FOMO
56.68
56.46
-0.22
32.07
31.67
-0.40
PGD-AT
69.15
65.93
-3.22
41.33
35.27
-6.06
82
FOMO
72.69
72.28
-0.41
45.60
45.16
-0.44
final performance of the FOMO network compared against vanilla adversarial training (PGD-AT) with l perturbation norm. We use of 128/255 and a step size of 15/255 for evaluation with & perturbation. The training/test attacks are PGD-I0/PGD20, respectively. The results demonstrate that FOMO significantly outperforms vanilla adversarial training across perturbations. Therefore, forgetting and relearning are more effective in mitigating robust overfitting across perturbations.
A.5 EFFECT OF FORGETTING IN DIFFERENT LAYERS ON ROBUST OVERFITTING
Training DNNs adversarially often results in predominant phenomenon known robust overfit- ting. Current learning techniques generally analyze learning behavior by treating the network as whole unit, which disregards the ability of individual layers to learn adversarial data distributions. We suggest that different layers possess unique capacities to learn information, and it is crucial to comprehend these patterns to develop training schemn that mitigates robust overfitting. There- fore, we investigate the layer-wise characteristics of a network by analyzing the effect forgetting parameters at different layers robust forgetting. For this analysis, architecture, where each block (4 blocks) is considered a layer along with an additional linear clas- sification layer. Figure demonstrates the effect of forgetting in different layers on the robust train and test accuracy.
the PreActResNett
Our analysis revealed that forgetting on early layers often results in decreased performance on robust test accuracy. Additionally, it fails to regularize the training accuracy, leading to a larger robust generalization gap. This possibly due the lower capacity of earlier layers to accommodate new information. Moreover, since earlier layers learn generalized features compared to later layers, the other hand, later layers have more capacity forgetting them results in a loss generalization. Therefore, forgetting layers 3 and 4 leads to to memorize and tend to overfit the training data. reduced robust generalization gap, which mitigates robust overfitting. By limiting the forgetting process to the last two layers, aim regularize the weights in the later layers while retaining more generalized features learned in the earlier layers to facilitate robust generalization.
A.6 STUDY THE SYMBIOTIC RELATIONSHIP BETWEEN FORGETTING AND RELEARNING
Recent advances in cognitive neuroscience have shed light on the interdependent nature of learning and forgetting, with mounting evidence indicating a symbiotic relationship between the two phe- nomena Bjork & Allen 1970); Bjork & Bjork (2019). In the context of DNNs, the dynamics of forgetting and relearning are of particular interest, as they have been shown to play a critical role in mitigating the deleterious effects of overfitting, Specifically, the percentage of forgotten parame- ters and the duration of the relearning phase (e,)) are important factors to consider to achieve robust overfitting.
15

---

Published as a conference paper at ICLR 2024
Train Robust Accuracy
Test Robust Accuracy
60
80
Accuracy
45
PGD-AT
Layer
Layer 162
Unear
Layer Layer
Layer 263
Layer
Layer
30
80
100
120
140
160
180
200
40
60
80
100
120 Epochs
140
160
180
200
Epochs
Figure 5: The impact of forgetting 50% of parameters in each layer on robust generalization using PreAct-ResNet-18 on CIFAR-IO illustrated in the figures. The figure on the left shows the impact on train robust accuracy and the figure on the right shows the impact on test robust accuracy. It is evident from the figures that forgetting in the later layers regularizes the train robust accuracy and mitigates robust overfitting when compared to forgetting in the early layers.
Robust Test Accursry
Robust Test Accumsiy
6-115
40
100
-25
150
175
180
125
150
175
Epochs
Epochs
(a) Forgetting 3% parameters in later layers.
(b) Forgetting 10% parameters in later layers.
Robust Test Accuroty
Robust Test Accuracy
22
4--25
6-225
Epochs
Epochs
(c) Forgetting 30% parameters in later layers.
(d) Forgetting 50% parameters in later layers.
Figure 6: Study the symbiotic relationship between forgetting and relearning during adversarial training.
To investigate the impact of forgetting and relearning on overfitting, we varied the percentage of forgotten parameters (3%, 10%, 30%, and 50%) and adjusted the duration of the relearning phase over six different intervals (1, 5, 15, 20, and 25 epochs) within total training duration of 200 epochs. Figure 6 presents the relationship between forgetting different parameters and the relearning phase during the training process.
Our results suggest that forgetting only a small percentage of parameters can effectively mitigate overfitting when combined with relatively short relearning phase. However, forgetting many pa- rameters with short relearning phases can make training DNNs challenging. This implies that the percentage of parameters forgotten directly correlates with the duration of the relearning phase; the higher the percentage parameters forgotten, the longer the relearning interval required to relearn the necessary information from the previous step. Therefore, is essential to balance the amount
16

---

Published as a conference paper at ICLR 2024
Table Training time per epoch on CIFAR-10 under 6x28/225 perturbation using eet 18 architecture.
Method
Computation Time per Epoch (s)
PGD-AT
132.6
WA
133.1
KD+SWA
132.6+ 132.6+16.5+141.77
AWP
143,8
FOMO
137.1
Table 10: Evaluation under CW attack on CIFAR-I0 and CIFAR-100 using PreActResNet-IB archi-
tecture.
CIFAR-10
CIFAR-100 100
Norm
Radius
Methods
Best
Final
Best
Final
PGD-AT
67.18
64.29
37.16
33.43
L
128 255
KD+SWA
68.87
68.90
40.56
40.61
FOMO
70.52
70.35
42.73
42.35
PGD-AT
47.00
39.96
22.73
18.11
KD+SWA
49.35
49.44
25.42
25.35
FOMO
52.14
51.95
26.98
26.61
of forgetting and the duration of the relearning phase enable effective relearning and retain the critical information necessary for optimal performance.
A.7 COMPUTATIONAL. EFFICIENCY
Here, we compare the computational efficiency FOMO with the baseline methods. To ensure fair comparison, all methods were integrated into universal training framework, and each test was performed on single NVIDIA GeForce 3090 GPU. T compares the computational time per epoch of FOMO with the considered baselines. Notably, FOMO and our baselines were trained for the same number of epochs (i.e., 200 epochs for CIFAR-10/100), From Table 9 it is evident that FOMO imposes almost additional computational cost compared to vanilla PGD-AT, with specific values of 137.1s for FOMO and 132.6s for vanilla PGD-AT per epoch. This implies that FOMO is an efficient training method in practical terms. It is important note that KD+SWA, formidable method designed to counter robust overfitting. comes with an increased computational cost. This arises from its approach, which entails the pretraining of both robust and non-robust classifier, serving as the adversarial and standard teacher, respectively. In addition, the method incorporates the process of distilling the knowledge of these teachers. Moreover, KD+SWA employs stochastic weight averaging to smooth the weights of the model, further contributing to its computational demands. We believe that this addition improves practical insight into the efficiency of FOMO and its comparison with existing methods.
A.8 PERFORMANCE UNDER CW ATTACK
Table 10 presents the evaluation results under CW attack (Carlini Wagner 2017) on CIFAR- 10/100 using the PreActRReNNet-B 18 architecture. The robust accuracy is assessed under CW attacks, and checkpoints with the best robust accuracy under PGD-20 attacks on the validation set are se- lected for comparison. FOMO consistently outperforms both baselines across different datasets and attack scenarios, demonstrating its effectiveness in enhancing robust accuracy under CW attacks. The best and final robust accuracies for FOMO are generally close, indicating that FOMO maintains its performance during training and does not suffer from significant overfitting under these attacks. These results emphasize the promising performance of FOMO mitigating adversarial attacks, particularly under CW attack scenarios.
17


Reviews:
[{"review_id": "UO6vOHeGec", "reviewer": "Reviewer_Jeha", "summary": "This paper addresses the challenge of robust overfitting in adversarial training of deep neural networks, which affects their generalization performance. The authors propose a new method called \"Forget to Mitigate Overfitting (FOMO),\" drawing inspiration from the brain's mechanism of active forgetting. FOMO operates by periodically resetting a subset of the network's weights to promote the learning of more generalizable features. The approach suggests a promising direction for enhancing neural network robustness against adversarial attacks by mitigating overfitting through controlled forgetting and relearning. Experimental results show that FOMO is a promising method to improve model robustness.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors conducted comprehensive experiments to demonstrate the effectiveness of FOMO. The proposed method is effective and outperforms the existing method, according to Table 3 and other experimental results.  \n* The proposed method is intuitive and easy to implement.", "weaknesses": "* My main concern is that the proposed method seems to be heuristic and empirical. there is not enough discussion on its intuition or theoretical foundation.\n* I don't think the running time and convergence analysis are well-studied in this paper, the authors may need to provide a table showing how many epochs are needed to converge and compare the running time with the existing methods. \n* Minor: Please refrain from only using color to distinguish curves and bars as in Figures 3, 4, 5, and 6, as it is not friendly to readers with color blindness.\n* Minor: Missing reference on robust generalization: Zhang, et al. \"The limitations of adversarial training and the blind-spot attack.\" ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fvOpNBFB8X", "reviewer": "Reviewer_WNii", "summary": "This paper proposed a method, FOMO, to deal with the adversarial overfiting issue. The proposed method alternates between the forgetting phase and the relearning phase.", "questions": "In this paper, the author only shows the result under a combination of white box and black box attacks, i.e., Autoattack. However, this cannot show \"the efficacy of FOMO against the black box and white box attacks\". Standard Autoattack has 4 adversarial attacks: three white-box attacks and one black-box attack. It is possible that FOMO has a strong resistance against white-box attacks while being vulnerable to the black-box attack.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is in good structure and easy to follow. \n\nThe topic, which is to deal with adversarial overfitting, is interesting.\n\nThe method is simple yet effective. \n\nAn ablation study is provided.", "weaknesses": "The description of the method is too intuitive. \n\nIn Table 1, the delta, which measures the adversarial overfitting, never favors the proposed method. This cannot show that the proposed method is good at dealing with adversarial overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zD6gDu31la", "reviewer": "Reviewer_FBLx", "summary": "This work aims to address the generalization gap in adversarial training. The authors exploit the random forgetting to adjust the  weights of models. Three datasets and two adversarial attacks are used to evaluate the proposed method. The experimental results show that the method can improve the robust accuracy.", "questions": "Please see weaknesses.\n\n============After rebuttal============\nThe authors provide adequate explanations for most of my questions, so I am willing to raise the rating score to 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work proposed a new method to solve the generalization gap in adversarial training.  The perspective of random forgetting is interesting.\n\n2. The introduction to the forgetting mechanism in Methodology is clear.", "weaknesses": "1. The description in the caption of Figure 2 is inconsistent with the content of the image. The former states that the consolidation phase is behind the forgetting phase, while the latter expresses that theconsolidation phase is before the forgetting phase. In addition, please present more clearly in the figure what the generalized information is.\n\n2. In Figure 1, FOMO is only compared with standard adversarial training, but not with methods that aim to reduce the generalization gap (such as AWP). This result may not appreciably represent the effectiveness of the proposed method.\n\n3. The authors use two adversaial attacks to evaluate the proposed method, they can consider more adversarial attacks (such as L2-norm CW, DDN) to conduct a more comprehensive evaluation.\n\n4. Figures can be clearer and more aesthetically pleasing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)