Easy incremental learning methods to consider for commercial fine-tuning applications
Authors: Udhaya Ravishankar
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Incremental Learning
Neural Networks
Classification
Regression

Abstract: Fine-tuning deep learning models for commercial use cases is growing exponentially as more and more companies are adopting AI to enhance their core products and services, as well as automate their diurnal processes and activities. However, not many countries like the U.S. and those in Europe follow quality data collection methods for AI vision or NLP related automation applications. Thus, on many of these kinds of data, existing state-of-the-art pre-trained deep learning models fail to perform accurately, and when fine-tuning is done on these models, issues like catastrophic forgetting or being less specific in predictions as expected occur. Hence, in this paper, simplified incremental learning methods are introduced to be considered in existing fine-tuning infrastructures of pre-trained models (such as those available in huggingface.com) to help mitigate the aforementioned issues for commercial applications. The methods introduced are: 1) Fisher Shut-off, 2) Fractional Data Retention and 3) Border Control. Results show that when applying these methods on vanilla pre-trained models, the models are in fact able to add more to their knowledge without hurting much on what they had learned previously.

Easy incremental learning methods to consider for commercial fine-tuning applications
Anonymous Author(s)
Affiliation
Address
email
Abstract
Fine-tuning deep learning models for commercial use cases is growing exponen- tially more and more companies are adopting AI enhance their core products and services, as well as automate their diurnal processes and activities. However, not many countries like the U.S. and those in Europe follow quality data collection methods for AI vision or NLP related automation applications. Thus, on many of these kinds of data, existing state-of-the-ar pre-trained deep learning models fail to perform accurately, and when fine-tuning is done on these models, issues like catastrophic forgetting or being less specific predictions as expected occur. Hence, in this paper, simplified incremental learning methods are introduced to be considered existing fine-tuning infrastructures of pre-trained models (such as those available in huggingface com) to help mitigate the aforementioned issues for commercial applications. The methods introduced are: 1) Fisher Shut-off, 2) Fractional Data Retention and 3) Border Control. Results show that when applying these methods on vanilla pre-trained models, the models are fact able to add more to their knowledge without hurting much on what they had learned previously.
10
11
12
14
Introduction
16
Many companies and organizations today are adopting Al in automation, automating their daily 18 processes and activities, as well as offering them in their core products and services. Automation has traditionally been in the industry for many years,
means for which economics of scale could 20 be acheived so as to remain competitive in the market. Now with AI, more and more intelligence is being brought into automation, and in countries like India, organizations are beginning adopt AI
21
22 for this particular purpose.
23 With recent advancements in Al vision and NLP models such as the GPT-3, Jurassic-1 1, and so 24 organizations today are using AI for 1) Document Reading and Understanding, 2) Online Proctoring, 3) Chatbots, 4) Intelligent Information Parsing and other application related process automations. Given these use cases, AI solutions need to be specific to their processes, but yet be an addition to their generally known formats. This in sense, is more like making use of human employee who 28 has some kind of general education on various tasks or processes but still required to learn the companies counterparts well and in detail before he/she is allowed to execute them. These processes can include between, reading customer emails for entering relevant information about their product requirements onto structured database, understanding various types of printed documents for information parsing, and to identifying newer objects for either document filtering or malicious
26
30
32
38
activity detection.
34 For natural language related tasks, powerful models like the GPT-3 are now being widely used, but they require good prompt engineering skills to get the best out of them. Also, given that they are probabilistic models, the generated outputs can sometimes falter away from what expected, and
36
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 this can become problem when selling it to customers, because even the slightest faltering may not
38 be acceptable to them all. Hence, reduce this, more and more examples have to be provided in
39 the prompt, and this can come at high cost not suitable for low cost of living countries like India.
40 The other workaround to fine-tune the model on the new datasets, but this has epoch limitations
41 on how deeply it can fit on the new dataset without hurting the body of general knowledge it gained
42 carlier. Also, fine-tuning models like the GPT-3 comes at a very high cost now-a-days, and is no more an option. This leaves the automation builders to use huggingface com transformers instead.
43
44 In vision, although state-of-the art pre-trained deep learning models are able to achieve human level performance on variety of inputs, they can only perform so in upto close to high quality inputs. If the quality goes lower, they fail terribly. Not all organizations have a good quality data collection process involved for applying automation, and this is ubiquitously the case in many parts of the world.
45
46
47
48
So it becomes quite difficult to sell Al as a human-level performer, and at this point Al becomes of lesser use than it could potentially be.
49
50
Another approach typically used to resolve such problems is employ transfer learning, which typically involves replacing the last layers of the model with a new model to get the specific outputs required. Some examples done in research are Too, et al. (2019), Dif & Elberrichi (2020), Alshalali & Joysula (2018), Jung, et al. (2015), Qian, et al. (2021) and Vrbančić & Podgorelec (2020), While this may not seem be problem with vision based tasks, it is definitely a problem with natural language based tasks. This is because the final layers of the natural language models have all the vital information of language structure that help with the language generative process. When this is to be changed, catastrophic forgetting can happen. Catastrophic forgetting is a phenomenon in which previously learned knowledge is lost partly by the application of new data for training. Also, with vision based tasks, when the requirement is to just improve the performance on lower quality data, transfer learning may not be the appropriate approach. Fine-tuning for these must involve the final layers of the model which could inevitably lead to catastrophic forgetting on the higher quality inputs.
51
52
58
54
55
56
57
58
59
60
61
62
This brings the only solution towards incremental learning. This type of learning all about learning on newer datasets without having the side-effects catastrophic forgetting, and there has been substantial amount of research done in this area. Luo, et al. (2020) summarizes all the work that has happened in this area so far. There are several approaches to implementing incremental learning on pre-trained models, some of which will be discussed in the forthcoming sections. In this paper, a few of these approaches will be simplified for commercial applications along with novel intuitive additions to further help the learning process. The paper introduces: 1) Fisher Shut-off which is a simplification of the work done by Kirkpatrick, et al. (2017), 2) Fractional Data Retention which adopts ideas from Castro, et al. (2018), and 3) Border Control which is an extension to the idea
63
64
65
66
67
68
69
70
71 outlined by Ren, et al. (2018) on reweighting examples by employing a method similar to Adaboost.
72 The last one is the novel addition as formulates a different approach to retaining salient examples
73 for incremental learning. It is based on the work by Ruping (2001) on incremental learning with
74 SVMs. But since SVMs are too complex in the context on neural networks, a similar but simplified
75
approach is proposed.
76 The purpose of this work is to initiate the development of a new infrastructure for commercial fine-tuning of pre-trained models with simplified incremental learning methods.
77
78
The rest of this paper proceeds follows:
Section will provide a brief discussion on incremental learning methods developed so far, followed by the proposal of simplified incremental learning methods in Section 3. Section 4 will show sample results of the proposed methods on vanilla pre- trained model using toy dataset. A toy dataset is used for the only purpose of providing visualizations on the performance of the proposed methods. Nevertheless, these methods can be extended on to real world datasets. The paper then concludes in Section 5 discussing steps forward for implementation.
79
80
81
82
83
84
85 2 Incremental Learning
86 This section a summary of the review published by Luo, et al. (2020). In this review, four different types of strategies for incremental learning are highlighted, and every work published in this area 88 uses either one or more such strategies. Some examples are Castro, et al. (2018) and He, et al. (2020). The four strategies are:
2

---

90
Architectural
81
Regularization
92
Rehearsal
93
Breudo-Rehearge
94
The following subsections will disccuss these briefly.
95
2.1
Architectural Strategy
This strategy is similar to boosting techniques where multiple models trained. But when used in the context of incremental learning, each model is trained on different task separately. Then another meta-model that effectively selects which model to use for inference is trained. The work done by Poliker, et al. (2001) resembles this in many ways. In this work, multiple classifiers are trained with different training sets, and then Adaboost style of ensemble learning employed combine the model outputs.
97
98
89
100
101
Another interesting work is by Rusu, et al. (2016) on Progressive Neural Networks (PNN). In this work, a neural network is trained sequentially on different tasks or training sets. However, each time, new neurons are added in each layer with weights, and the weights of the previously learned neural network are frozen. Then, to prevent catastrophic forgetting, the outputs of each layer of the previous neural network the earlier training
103
105
used in addition to the new task or training set, when training the new layer neurons. The results on this type of incremental learning were quite direction in the research of dynamically expanding networks that could make better use the neural networks capacity than the PNN. In fact, it will be seen later that the Fisher Shut-off method proposed in this paper inherently employs the idea of PNNs.
107
108
encouraging that set
109
110
2.2
Regularization Strategy
112 In this strategy, as the name suggest, a regularization term is added in the loss function that measures
113 the importance of old knowledge when learning on new training set. The representive work done in
114 this is Kirkpatrick, et al. (2017). whereby they introduce the concept of Elastic Weight Consolidation
115 (EWC) by means of Fisher Information Matrix. The EWC brings about the regularization term in
116 the loss function as
A R(w) = Σ , Fr(w; F((ss - wixold)?
117 where F; is the Fisher Information Matrix which suggests the importance the i-th weight trained
118 on the old (or previous) training set. Here, as one could speculate, the term Fisher Shut-off proposed in this paper actually derives itself from the Fisher Information Matrix, meaning that this matrix is new set.
119
120
used as the basis for shutting off the training of certain weights when training
121
Another popular type of regularization strategy is Knowledge Distillation introduced by Hinton, et al. (2015). In this method, knowledge from an ensemble of models trained on different tasks (or training sets) separately are distilled into a smaller model that can be deployed much easily for inference.
122
123
124
There are many buggingface com transformers that are product of such knowledge distillation.
125
The distillation ensures that the smaller model holds all the knowledge of the ensemble, and that can infer as good as it. Distillation is done by setting soft-targets on the smaller network from all the earlier training sets of the ensemble. The soft- targess are the output logits from the ensemble models on their respective trained datasets.
128
127
128
2.3
Rehearsal and Pseudo-Rehearsal Strategies
130
Rehearsal strategies in incremental learning make use of the earlier training sets when training a model on new tasks or training sets. This by far is the simplest of all incremental learning strategies that ensures catastrophic forgetting is prevented. The only issue that when this strategy is used for deep learning models trained on large datasets, the training on new datasets could become extremely
132
3

---

134
slow and even time consuming before any fruitful results are achieved. Hence, newer research work in this area formulate methods for retaining only the most important data points to prevent catastrophic forgetting. The work done by Castro, et al. (2018) 1) is example of this. In this work, selection and removal mechanisms on data are introduced for assimilation into memory network.
135
137
138
Talking about memory networks, the Pseudo-Rchearsal strategy involves training an additional data generator generate the samples, the neural network was trained on earlier. Hence, newer research in this area involve GANs for data generation. Examples are Odena, et al. (2017) and Wu, et al. (2018).
139
140
141
3
Proposed Incremental Learning Methods
142
143
Commercial applications always require simplistic implementations of advanced methods no matter how complex they may be. Therefore, is for this purpose alone this paper proposes some simplified methods for implementing incremental learning. As metioned earlier in Section [ these methods are: Fisher Shut-off, 2) Fractional Data Retention, and 3) Border Control. This section covers them in detail.
144
146
148
3.1
Fisher Shut-off
149
As mentioned in the previous section, the term Fisher Shut-off derives itself from the Fisher Infor- mation Matrix which weighs the importance of weights trained on previous datasets, Hence, in this sub-section, a brief overview of the details behind this matrix covered with the help of Aich (2021).
150
151
152
Let D represent a dataset coming from a stream of data for incremental learning. Then p(w) 'D) represents the model trained on data DD This means that to train a model following posterior must satisfy:
dataset, the
154
p(w(DDeww) (new) D(Dnew w) Dota) p(Dnew)
155
Note here that ((w[Doid is written in place of p(w) because when Dew is applied to the model, the 158 weights w have already been trained with Dotd. Hence, given the model, p( (Pota)) the log-likelihood
157 loss on Dnew becomes,
Drww (w) log(p( (w u(Dnew))
log(p(( (D (newlw)) + (og(p(w (Doaa)))))))))))))))))(()))))( - log(p( (p((new))) log(p( new/ww) (og(p( (ota))
(3)
158 Here, the (Dpp((Deelll)))) equals the cross-entropy loss of the model on Dnew while (og(p(w) (ota))
159 loss of the model on Dota- To ensure that catastrophic forgetting does not occur on Datd in its
absence while training on Dnews the loss on Dota will have to be approximated using alone. To do
161 this, the Taylor's expansion on log(p(w:) (Dtt)) taken as,
Loan (w) ((w))lpa + @L(w)
0 (L P2
+ 2(1 w/ wlDaw)" )r
(w
w/ Dand)
Iw
Doial
Dota
8ww (P
(0PC(w)
~ [(w) Dars
+ 2
(w- w Dasd yr
(w - w/ (
Data
182 since technically dL(w) Ou
0, if the model is trained well on Datd Then, noting that the last term in (4) is equivalent to regularization term, this term alone could be considered the loss Doid
Datd

---

184 for preventing catastrophic forgetting. In doing so, the Fisher Information Matrix will equal to the
0² (2 This Hessian, H. can be simply computed by the model gradients
@C(w) tw
185
Hessian, 8P
Daid
Data
186
assuming that not all gradients are zero, as,
H @L(w)
al(w)
(5)
Ow
Sw
Data
Dald
167 Doing so, and keeping only the diagonal terms, would imply that the model gradients are more than enough weigh the important weights of the model trained Dotd- Replacing (5) in (4) and 189 substituting in (3) would give the loss on Dnww as,
[D[(w) aC(w) T Sw
Drccw (w) ze log(p(Dneellu)) 5(" w Dasu )²
(w (w-w
Doad (6)
Ow
Dota
170
which to an extent implies that if the model gradients on Dotd are absolutely zero, they get trained on Deww without regularization, while those that are not, get regularized towards w Data"
172
This is what the proposed Fisher Shut-off exploits. In Fisher Shut-off, all weights of the model trained on Dota that do not have absolute zero gradients get shut-off for training on Dnew while remaining that do take part. Also, since in practice Re Lu functions are commonly used in deep learning models as the activation functions of neurons, shutting off these weights becomes as simple as setting condition. Figure shows sample performance of Fisher Shut-off on a regression model trained sequentially on mutually exclusive batches of data. These batches could represent the different tasks or training sets.
173
175
177
179
However, when it comes to classification, simple shut-off does not work completely. This is because, while in regression problems datasets could inherently employ some kind of piece-wise nonlinear fit their distributions, the same cannot always be guaranteed in classification. Thus, in classification,
180
181
Batch #1
Batch #2
Batch #3
0.7125819637966035
0.826204337645094
0.444389413643661
2.0
10
1.5
05
0.
0.5
0.0
-15
-15
-2.
-2.0
Batch #4
Batch
Batch #6
0.1257310699423718
0.0981010900204433
0.0001652057010500744
1.00
0.75
0.50
0.25
00
0.0
0.0
0055
-0.
-0.50
-0.75
-1.0
-1.0
-1.00
-2
Figure 1: Fisher Shut-off on a regression model on six mutually exclusive batches of data. Blue dots represent the overall dataset, while green square dots are the batch or task data. The red line is the model's 's output after each batch is fed to it. Fisher Shut-off is used from Batch #2 onwards.
5

---

182
the shut-off weights must also take part in training. And, as per (1), there is a learning constant required in the regularization to ensure that the right balances between D Pnew and D Dota are met on these weights. This paper provides a novel learning constant determination for this regularization. This is detailed in Appendix A
183
184
185
186
Also in regression problems, if datasets have batch distributions that are quite far apart from each previous batch, then Fisher-Shutoff may not fully work too. Appendix B shows some of these
187
188
examples
3.2
Fractional Data Retention
190
This is a very simply proposal. The idea is to retain only a fraction of the data trained on the neural network on the earlier tasks training sets. There is nothing more to this. However, banking on the ideas of selection highlighted in Castro, et al. (2018), whereby data is selected based on their proximity to cluster centers, to be more representative of the classes, this paper uses this as the baseline idea behind its proposal on Fractional Data Retention. Thus in Fractional Data Retention, a fraction of the data within the data cluster is retained and appended in every stage of incremental learning.
192
193
194
195
196
197
3.3
Border Control
198
The most important requirement when incrementally learning classes is to ensure that the decision boundaries of the carlier training tasks are protected much as possible when training on new sets. If data points are used for this purpose, it would seem that, those that lic closest to the decision boundaries after training would be the most important ones retain, for any succeeding incremental learning tasks. Thus, the Border Control method proposed in this paper exploits this. Ruping (2001) used SVMs to identify these data points the support vectors that helped define the overall decision boundaries. But with deep learning models or vanilla neural networks, SVM is quite complex and therefore in order be able retain data points closest to the decision boundaries, a different selection mechanism is required. This selection mechanism could instead be based on selecting data points on how large the absolute errors in sigmoidal outputs for the applied dataset, as the data points closest to the decision boundaries have this inherent property.
200
202
204
205
206
207
208
Furthermore, since real world data can be quite complex, it would be necessary to not only select data points based on how large their errors in sigmoidal outputs are, but also those points that are far away from them. This is because, given the context of incremental learning where there is high chance that newer training sets may have data points that could potentially set newer decision boundaries in those fartherest regions, these data points would help protect those.
210
212
214
Hence Border Control, the top-k data points that have the largest absolute errors in the sigmoidal outputs and their respective top-k fartherest data points are retained in every task or training set for further incremental learning. These points are appended to the newer training sets before further training is applied.
216
Sample Results
218
The proposed methods are tested a toy dataset, as mentioned in Section only to provide some visuals on how the incremental learning progresses using the proposed methods. Figure [ this dataset. vanilla deep neural network of size, (000-1000-1000-1000-- is used for incrementally
220
221
222
learning batches of data from this toy dataset. The activation functions for all layers are LU except for the output which is a so ftmoz. All weights are uniformly but randomly initialized with a single random seed to make the results comparable. The weights are also scaled by 2 factor to ensure that minimal overfitting occurs during training. Here is the layer fan-in.
223
224
226
To visualize incremental learning on the proposed methods, the dataset is divided into 6 batches with mututally exclusive data points. This gives roughly between 100 to 200 data points in each batch, a size that is commonly used when training neural networks of this size. Fiouss this.
227
228
In this figure, it can be clearly seen that the batch distributions on the class data for incremental learning do not always form a piece-wise nonlinear fit, and therefore, plain shut-off of weights cannot fully retain knowledge learned carlier. Also, among these distributions, some allowed incremental
230

---

Figure 2: The toy dataset having three nonlinearly arranged classes.
Batch #1
Batch #2
Batch #3
Batch
Batch
Batch
Figure 3: Batches on the toy dataset.
232
learning to happen easily, while others did not, and the distribution shown in Figure 3 is one such. Table summarizes the results of the proposed methods on this particular distribution. For other batch distributions, similar results could be achieved. Note here that quite some ML-Ops were required to achieve the results in Table [] This was especially the case for those that employed Fisher Shut-off, since this method has a regularization constant that requires adapting on each batch. Furthermore, training on each batch was stopped once 100.0% accuracy was obtained on the batch. This left quite
233
235
236
237
some data points to lie very close to the boundary lines or in some cases just right them. Thus, the neural network was very vulnerable to catastrophic forgetting when succeeding batch trainings occurred as part of incremental learning.
239
241
However, taking a look Table it can be seen that when Fisher Shut-off is applied, additional leverage against catastrophic forgetting occurs on each incremental batch, than when it is not used. And, among the three methods proposed in this paper, the Border Control method shows much stronger performance. In Figure sample decision boundaries learned when each incremental batch applied to the neural network using Fisher Shut-off and Border Control together is shown. topk value of 5 is used for the Border Control. Also, note in Figure 4hure the red circles mark the border points accumulated on each batch. can be seen that they clearly assume the data points closest
242
243
244
245
246
247
to the decision boundaries, as well as those far away from it. All with respect to their batches. For the far away data points, their purpose can be clearly seen between batches #1 and #2, where the fartherest points of class 2 in Batch #1 helped protect the decision boundaries from the data points
249
7

---

Table 1: Performance of proposed methods on the dataset of Figure 2
Sample accuracy accumulated dataset after Batch
!
Method
#1
#2
#3
#4
#5
#6
No Incremental Learning
100.0%
90.90 9898%
92.76%
94.89%
98.32%
96.11%
Fisher Shut-off (FS)
100.0%
99.74%
98.
98.88%
98.96%
96.89%
Frac. Data Ret. (10R)
100.0%
97.94%
98.39%
98.89%
98.71%
98.67%
FDR (20%]
100.0%
98.71%
98.59%
99.36%
98.97%
98.78%
Border Ctrl. (BC) = 5]
100.0%
100.0%
100.0%
99.84%
100.0 %
100.0%
BC|topk 10
100.0%
100.0%
100.0%
99.84%
100.0 %
100.0%
FS + FDR [10%
100.0%
99.74%
98.79%
99.52%
99.23%
98.78%
FS + FDR [20%]
100.0%
100.0%
99.19%
99.52%
99.48%
99.11%
FS + BC|topk = 5
100.0%
100.0%
100.0%
99.84%
100.0% %
100.0%
FS + BC/topk 10)
100.0%
100.0%
100.0%
100.0%
100.0% %
100.0%
Batch #1
Batch #2
Batch
ACC 0.636666666666667
0,923833333333333
ACC
0.98222222222222
it
-1
Batch
Batch
Batch ACC 1.0
ACC 0,98333333839333
ACC = 0.9911111111112
0-
0-
-3
-3
-4
Figure 4: Incremental learning using Fisher Shut-off and Border Control together [topk 5]. Black circles mark the batch data, while the red circles mark the accumulated border points.
251
of class 0 in Batch #2 This means that more complex datasets can be accommodated by simply applying Border Control. More examples shown in Appendix C
253
Also, to add on further to this, for the most difficult incremental learning applications such as learning new classes as highlighted in Castro, et al. (2018) and He, al. (2020), Border Control can help leverage the many issucs associated with like class imbalance, concept drift and so on.
255
5
Conclusion
To summarize the work in this paper, three simplified methods for implementing incremental learning for commercial fine-tuning of pre-trained models was proposed. Results showed that while Border Control performed the best, Fisher Shut-off was able to leverage the performances. However, dataset used in this paper was a toy dataset and not one the benchmark datasets typically used for
258
280
Incremental learning method aplied from Batch #2 onwards.
Indicates catastrophic forgetting
8

---

261
incremental learning. Hence, testing these methods on the benchmark datasets is potential next step forward. Then. preparing the prerequisites for each model available, like say wwggingface.ccommmm com for incremental learning must be done so that automation companies or any other Al organization can make use of them. From the methods proposed this paper, the prerequisites would be: 1) the Shut-off matrix for the neural network weights, and 2) the border points for each of the learned classes. Additionally, an ML-Ops infrastructure can be provided to optimize the performances of
262
264
265
286
the models that employ the Fisher Shut-off method. Metries like the Backward Transfer (BWT) and Forward Transfer (FWT) proposed in Lopez- Paz & Ranzato (2017) can be used for this purpose.
268
References
269
270
[1] Aich, A. (2021) Elastic weight consolidation (EWC): Nuts and bolts, arXiv preprint arXiv:2105.0400)
271
[2] Alshalali, T. & Joysula, D. (2018) Fine-Tuning of Pre- Pre-Trained Deep Learning Models with Extreme Learning
272
Machine. IEEE International Conference on Computational Science and Computational Intelligence (CSCI), pp.
273
469-473.
274
[3] Castro, F., Marín-Jiménez. M.J.. Guil. N., Schmid, C. & Alahari, K. (2018) End-to-end incremental learning. Proceedings of the European Conferenec on Computer Vision (ECCV), pp. 233-248.
[4] Dif, N. & Elberrichi, Z. (2020) A New Intra Fine-Tuning Method. International Journal of Service Science, Management, Engineering, and Technology, 11(2). pp. 16-40.
277
278
[5] Hc, Mao, R., Shao, Z. & Zhu, F. (2020) Incremental learning online scenario. IEEE/CVF Conferenee on Computer Vision and Pattern Recognition, pp. 13926-13933
279
280
[6] Hinton, G., Vinyals, O. & Dean, (2015) Distilling the Knowledge a Neural Network. arXiv preprint Xiv: 1503.02531, 2(7).
281
282
77] Jung. H., Lee, S., Yim, J., Park, S. & Kim, J. (2015) Joint fine-tuning in deep neural networks for facial expression recognition. Proceedings of the IEEE International Conference on Computer Vision, pp. 2983-2991.
284
[8] Kirkpatrick. J. Pascanu, R., Rabinowitz, N. Veness. J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Baan Barwinska, A. & Hassabis, D. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings the National Academy Sciences, 114(13), pp. 3521-3526.
288
287
99 Lopez- Paz, D. & Ranzato, M.A. (2017) Gradient episodic memory for continual learning. Proceedings of Neural Information Processing Systems (NTPS), pp. 6467-6476.
288
289
[10] Luo, Y., Yin, I.., Bai, W. & Mao, K. (2020) An Appraisal of Incremental Learning Methods. Entropy, (11,, pp. 1190-1216.
290
281
[11] Odena, A., Olah, C. & Shlens, J. (2017) Conditional image synthesis with auxiliary classifier GANs. International Conference Machine Learning (ICML), pp. 2642-2651.
293
[12] Polikar, R., Udpa, L., Udpa. S. & Honavar. V. (2001) Learn++: An Incremental Learning Algorithm for Supervised Neural Networks. IEEE Transactions Systems, Man, and Cybernetics, part C (applications and reviews), 31(4), pp. 497-508.
295
296
[13] Qian, X., Zhang, C., Yella, J., Huang, Y., Huang. M.C. Bom. (2021) Soft sensing model visualization: Fine- Fine-tuming neural network from what model learned. IEEE International Conference on Big Data (Big Data),
297
298
pp. 1900-1908.
289
[14] Ren, M., Zeng, W., Yang, B. & Urtasun, R. (2018) Learning to reweight examples for robust deep learning. International Conference Machine Learning (ICML). pp. 4334-4343
[15] Ruping, S. (2001) Incremental learning with support vector machines. IEEE International Conference on Data Mining, pp. 641-642.
302
[16] Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R. Hadsell, R. (2016) Progressive neural networks. arXiv preprint arXivv1606.04671
304
305
[17] Too, E.,, Yujian, L.., Njuki, Yingchun, L. (2019) A comparative study of fine-tuning deep learning models for plant disease. Computers and Electronics in Agriculture, 161, pp. 272-279.
306
307
[18] Vrbančic, G. & Podgorelec, V. (2020) Transfer learning with adaptive fine-tuming. IEEE Access, Volume 8,
308
Pp. 196197-196211.
309
[19] Wu, Y., Chen, Y.P.. Wang, L.J., Ye, Y.C., Liu, Z.C,, Guo, Y.D., Zhang, Z.Y. & Fu, Y. (2018) Incremental Classifier Learning with Generative Adversarial Networks. arXiv preprint 100vv.180.

---

Checklist
312
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
313
314
315
316
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
318
319
Did you include the license to the code and datasets? [N/A]
320
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
322
323
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
325
326
(b) Did you describe the limitations of your work? [Yes] See Sections 4 and 5
327
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
328
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
329
330
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Appendix [A]
332
335
(b) Did you include complete proofs of all theoretical results? [Yes] See Section S and Appendix A
335
3. If you ran experiments..
336
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] As supplemental material
338
339
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [No] But it's in the supplemental material
(c) Did you report error bars (e.g., with respect the random seed after running experi-
342
ments multiple times)? [No] But additional examples are shown in Appendix C
343
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
344
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
348
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
349
350
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
352
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
353
354
5. If you used crowdsourcing or conducted research with human subjects...
355
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
356
357
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
358
359
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
10

---

A
Appendix
382
Here, the derivation of regularization constant for Fisher Shut-off method is detailed. To start with, let the neural network be defined as,
363
y - w 202 w)
Here, w is the weights of the output layer, <(-) is the output of the preceding layer, I is the input and 385 represents the rest of the weights of the neural network. Throughout the derivation,
384
will be dealing with only the output layer, and so the o(x, w) will be written in short form as & from here on. Let eg denoted the error of fitting the k-th iteration, and 9k denote the error gradient. This would mean that Ok: = CPC
386
367
Then, given the regularization term in (6), let v denote difference in weights, between the new training and the previous training. This would give the weight updation policy as,
370
Wk+1
Wi;
79h
(8)
371
cj= = / 2x Y, then the error ck+1 after the weight updation would equal,
Ek+1 wh+1 1x+1 - Y
(wk 19kh BBūx) )d #x+11 - Y
(9)
wx 4641 199 x++1 Bwg 1c+1 Y
372
Asumming for simplicity sake that Px+1 so tb + 8, then (9) can continue as,
ex+1 RC wI db nok 4x - Bwg 24 - +A
(10)
ee no d; Bwh 4x
373
Taking the square norm of ex+1 in (10), would equate this to,
||ext1 2 = ||ex|1 + 172 2x11 + 3 22
(11)
- 2 gx 28e 4. ux + 2n 2 9k
We require that || ||ex| at all times, that regularization does not affect the fit at any 375 point during the training. Applying this condition in (11) would give,
+
1+11 2n v 9b 23 o5 we + 2n 4
< 0
(12)
376 Then, taking the partial derivatives of (12) w.r.t 1) and 3 would give the following equations to be satisfied:
7| 9 o - of + BoT o 0
(13)
31|08 dh 11? 2 -e--e TT th + noσ 2r
(14)

---

378 Solving, (13) and (14) can result in negative 77 and 3, which is not acceptable, and so to simplify the
379 solution, we neglect the 3-term in (13). Doing so we get,
c $ g 11 12 11
=
(15)
2 og wk g PN D
B
1 1 1
380
Then, substituting for and openning up the norms, we get,
TT PRCR
77
6 ( 4 ( 4 lek
(16)
c/
T Tb Uh qoo (D PRCR
B
W ( 2/ yuu
Simplifying (16) gives.
381
c ck
= T o DEER
k
(17)
I R
8 (1 11)
W thk
Equation (17) gives the raw form for both 77 and 3 to be regulated. However, this will be further simplified for computating purposes, but will be used as a basis.
383
384
Since the errors e get smaller as the neural network fits the data, using them in learning constants will only slow down the fits. common way to overcome this is by replacing e with all ones. Similarly, for the Wk all weights that are to be regularized are replaced with ones. If we denote
385
386
387
the weights to be regularized as wy, and there are m patterns the dataset with a weights to be regularized, the y and B computations become,
1
17
p 1
(18)
8
389
Here, a represents the (1 - -eem in (17). This constant will not neccessarily take the computed when being regulated. Instead, this constant will have to be adapted each time for every incremental batch applied to the neural network.
391
382
The reason why the computed r is not used for the adaptation is because this 77 can sometimes become too small in the adaptation, that the - y would always tend towards When this was empirically tested on the toy dataset, the regularization was found at times to have gone too strong
395
385
that the never happened. MI.-Ops on the found that this constant is not always 1, and can be anywhere between 0 and 1, or higher in some cases.
12

---

B Appendix
397
396
Additional examples on the regression problem with Fisher Shut-off. Fisher Shut-off could not be used completely, and regularization had to take over for some batches. Figures 5 and 6 show this.
MSE 0,4281102081889777
MSE 0,2151558244249M66
MSE 0.10485160224617718
1.0
1.00
0.75
0.5
0.50
0.00
-0.5
-0.5
-0.75
-2
Batch #4 2
-2
Batch #5
-2
Batch #6
MSE 0.1266359810533611
MSE 0.10710626561226235
MSE 8.000966115716552212
1.0
1.00
05
0.50
0.5
0.0
-0.5
-0 0.50
0.75
Figure 5: Complete Fisher Shut-off is used in Batches #2 and #6. Batches #3, #4 and #5 are regularized.
Batch #1
Batch #2 MSE 0.6803429560061659
Batch #3
MSE
3.24911809480033
MSE
0.28210500594440866
1.00
1.00
75
10
0.25
0.25
0.00
0.00
0.0
-0 50
-0 50
-0.5
75
Batch #4
Batch #5
Batch #6
MSE
00019049479998888777
MSE
1.00208242322733548888
MSE
1.01114842374832551444
0.75
0.75
0.5
850
0.00
0.0
0.00
-0 25
-025
-0.5
-1 88
-1.0
-1.00
-2
Figure 6: Complete Fisher Shut-off used in Batches #2 and #6. Batches #3 and #4 are regularized. Batch #5 is fine-tuned
13

---

C
Appendix
In this appendix, additional examples on the toy dataset classification is shown. Figures and show the batch distributions considered. Among these, Figure R has more cases in which the farthest points in Border Control can play vital role in retaining previously learned knowledge. Tables and summarize their performances.
402
403
Batch #1
Batch #2
Batch #3
Batch
Figure Another batch distribution on the toy dataset.
404
Table 2: Performance of proposed methods on the dataset of Figure 7
Sample accuracy on accumulated dataset after Batch
Method
#1
#2
#3
#4
#5
#6
No Incremental Learning
100.0%
73. 14%
92.07%
97.56%
88.33%
91.67%
Fisher Shut-off (FS)
100.0%
99.43%
98. 98.26%
99.54%
92.85%
97.78%
Frac. Data Ret. (FDR)]10%)
100.0%
97.43%
96.13%
98.93%
98.75%
96.78%
FDR[20%]
100.0%
98.86%
98.84%
99.69%
99.75%
98.89%
Border Ctrl. (BC)[topk 5]
100.0%
100.0%
100.0 %
100.0%
100.0%
99.78%
BC|topk = 10
100.0%
100.0%
100.0 %
100.0%
100.0%
100.0% %
FS (10%
100.0%
99.71%
98.84%
99.54%
98.75%
98.33%
FS + FDR[20%)
100.0%
100.0%
99.23%
99.85%
99.87%
98.89%
FS BCitopk = 5]
100.0%
100.0%
100.0 %
100.0%
100.0%
100.0% %
FS + BCitopk = 10]
100.0%
100.0%
100.0 %
100.0%
100.0 %
100.0 %
14

---

Batch #1
Batch #2
Batch #3
-1
-2
Batch #4
Batch #5
Batch #6
Figure 8: Yet another batch distribution on the toy dataset.
Table 3: Performance of proposed methods on the dataset of Figure 8
Sample accuracy accumulated dataset after Batch
Method
#1
#2
#3
#4
#5
#6
No Incremental Learning
100.0%
89.34%
93.80%
95.60%
97.81%
84. 84.78%
Fisher Shut-off (FS)
100.0%
95.36%
99.59%
99.55%
99.36%
94.56%
Frac. Data Ret. (FDR) (10%]
100.0%
95.08%
96.07%
97.42%
99.61%
98.67%
FDR [20%]
100.0%
98.36%
98 08.97%
99.85%
100.0%
99.67%
Border Ctrl. (BC)[topk 5]
100.0%
100.0%
99.79%
99.85%
100.0%
100.0 %
tC = 10
100.0%
100.0%
100.0% %
100.0%
99.74%
100.0% %
FS FDR[10%)
100.0%
95.36%
99.79%
98.48%
99.61%
98.89%
FS + FDR 20%]
100.0%
98.36%
99.79%
99.69%
100.0%
99.67%
FS BCitopk =
100.0%
100.0%
100.0
100.0%
99.74%
100.0 %
FS + BC[topk- = 10)
100.0%
100.0% %
100.0 %
100.0%
100.0 %
100.0 %
15


Reviews:
[{"review_id": "v68PMXKalQN", "reviewer": "Reviewer_wa3e", "summary": "This paper introduces three simplified methods for incremental learning. The three methods are Fisher Shut-off, Fractional Data Retention and Border Control, which are all adapted from previous works.  Fisher Shut-off select weights in models that take part in fine-tuning. Fractional Data Retention and Border Control retains a fraction of data from earlier tasks and dataset. All the methods are illustrated with a toy dataset, and the author claims that they can be used in real world datasets.", "questions": "1. All three methods are adapted from previous works, what are the key differences?\n\n2. Experiments are conducted on a toy dataset. Please provide experimental results on real-world datasets to support the claims in this work.\n\n3. Even for the toy dataset experiment, \n    - Why there are no existing techniques used for comparison? \n    - I am not sure if the experiments in Table 1, i.e, FDR 10% and FDR 20%, or BC (topk=5) and BC (topk=10), have a different percentage of retained data. If so, a direct comparison between them is not fair.\n    - Experiments with incremental learning without FS are essential.\n\n3. FDR requires data clustering, which technique do you use in the experiment? Do different clustering techniques largely affect the result?\n\n4. In BC, can you provide some proof that selecting top-k farthest data points are more effective than random selection? Since the paper is claimed for real-world datasets with low-quality data, it is likely that the top-k farthest data points are out of distribution, which provides limited help.", "limitations": "The limitation of this paper is partly addressed in Conclusion.", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe proposed methods are simple and easy to implement.\n\nWeakness:\n\n1. Lack of originality, the methods proposed in this paper are mainly built on other works. The differences are not clearly shown in the paper.\n\n2. The evaluation is on a toy dataset and it is too simple to support the claims that the proposed methods can be used in real-world applications. \n\n3. The data selection method designed in Fractional data retention and Border control is not well studied and evaluated in this paper. Deep learning data selection is studied in many topics, i.e, active learning, model repair, and test data selections. To evaluate the effectiveness of the Fractional data retention and Border control, the authors should show that their methods are better than random selection and other commonly used data selection methods. [[1](https://arxiv.org/pdf/1708.00489.pdf),), [2](https://arxiv.org/pdf/1906.03671.pdf)]\n\n4. The presentation of this paper is poor.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_IkoFCytqcy", "reviewer": "Reviewer_ocES", "summary": "This paper aims to address the incremental learning of pre-trained models for practical commercial scenarios. No novel method is proposed, and this work seems like a course report. ", "questions": "None", "limitations": "See the Cons above.", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- None\n\nCons:\n- There is not even a novel method proposed to address the problem of focus.\n- Plenty of existing works in the NLP and even the CV domain have been ignored. See the references [1-6] below.\n- All the results are evaluated on very small synthetic datasets; none of the existing benchmarks has been investigated. Moreover, the existing state-of-the-art baselines have not been compared against.\n\n[1] Jang, Joel, et al. \"Towards Continual Knowledge Learning of Language Models.\" International Conference on Learning Representations. 2022. \n[2] Lin, Bill Yuchen, et al. \"On Continual Model Refinement in Out-of-Distribution Data Streams.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n[3] Ermis, Beyza, et al. \"Continual Learning With Transformers for Image Classification.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[4] Jin, Xisen, et al. \"Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora.\" Challenges & Perspectives in Creating Large Language Models (2022): \n[5] LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5\n[6] Pelosin, Francesco, et al. \"Towards exemplar-free continual learning in vision transformers: an account of attention, functional and weight regularization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D8PyJntYJNB", "reviewer": "Reviewer_EYaL", "summary": "Incremental training via fine-tuning of large-scale deep learning models often face challenging issues like catastrophic forgetting. As a remedy, this paper proposes three methods 1) Fisher Shut-off, 2) Fractional Data Retention and 3) Border Control for commercial deep learning fine-tuning scenarios. The effectiveness of the proposed remedies are tested on a toy dataset. ", "questions": "N/A.", "limitations": "N/A.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nIncremental learning on commercial applications is an important topic. \n\nWeakness:\n\nThe originality of this paper is not sufficient. While certain ideas are definitely interesting (like the Fisher shut off), most of the proposed strategies already exist, so the value of information from this paper would be quite limited. \n\nExperiment section must be further enriched. I don't quite get the logic of illustrating the effectiveness of the proposed methods on a toy dataset, while one of the selling points of this paper is \"commercial fine tuning applications\".\n\nI think the presentation of this paper can be much improved. Details should be added to the included strategies, e.g., section 3.3", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject