LOIRE: LifelOng learning on Incremental data via pre-trained language model gRowth Efficiently
Authors: Xue Han
Yitong Wang
Junlan Feng
wenchun.gao
Qian Hu
Chao Deng
Venue: ICLR 2025 Poster (2025)
Keywords: Lifelong learning
Model growth
Function-preserving
Efficient pre-training
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A framework for lifelong learning that enables PLMs to effectively grow their capacity using incremental data

Abstract: Large-scale pre-trained language models (PLMs) require significant computational resources to train from scratch on large volumes of data. But in the real world, emerging data from diverse sources may not be initially available for pre-training. Recent studies on lifelong learning have tried to solve this problem by exploring the use of model growth techniques to effectively incorporate new knowledge without the need for complete re-training. However, model growth approaches utilized have issues with growth operators that do not ensure strict function preservation or growth schedules that only include a few growth dimensions, reducing lifelong learning's effect. Furthermore, existing approaches often assume that emerging data has the same distribution as pre-training data, causing catastrophic forgetting of previously acquired knowledge. To address the aforementioned issues, we introduce LOIRE, a framework for lifelong learning that enables PLMs to effectively grow their capacity using incremental data. LOIRE employs growth operators for all feasible dimensions and a growth schedule to generate the optimal expansion sequence in the field of lifelong learning. Specifically, we present a novel plug-in layer growth operator with residual connections that skip the newly added layer during initial training while ensuring function preservation. We additionally propose an iterative distillation strategy for LOIRE that allows an intermediate model in the growth stages to switch between being a student and a teacher, reducing catastrophic forgetting during growth. Experiments show that LOIRE can reduce computational expenses by an average of 29.22\% while retaining equivalent or better downstream performance.

Published as a conference paper at ICLR 2025
LOIRE: LIFELONG LEARNING ON INCREMENTAL DATA VIA TRAINED LANGUAGE MODEL GROWTH EFFICIENTLY
Xue Han , Yitong Wang , Junlan Feng
Wenchun Gao, Qian Hu & Chao Deng
JIUTIAN Team
China Mobile Research Institute
Beijing, China
(hanxuea1, xongyitoonyyy,, fongjunlon, gacwenchun, huglanet, dengchao 8ch nomobi le.com
ABSTRACT
Large-scale pre-trained language models (PLMs) require significant computa- tional resources to train from scratch large volumes of data. But in the real world, emerging data from diverse sources may not be initially available for pre- training. Recent studies by exploring the use of model growth techniques to effectively incorporate new knowledge without the need for complete re-training. However, model growth approaches utilized have issues with growth operators that do not ensure strict function preservation or growth schedules that only include few growth dimen- sions, reducing lifelong learning's 's effect. Furthermore, existing approaches often assume that emerging data has the same distribution as pre-training data, causing catastrophic forgetting of previously acquired knowledge. To address the afore- mentioned issues, we introduce LOIRE, a framework for lifelong learning that enables PLMs to effectively grow their capacity using incremental data. LOIRE employs growth operators for all feasible dimensions and a growth schedule to generate the optimal expansion sequence in the field of lifelong learning. Specifi- cally, we present a novel plug-in layer growth operator with residual connections that skip the newly added layer during initial training while ensuring function preservation. We additionally propose an iterative distillation strategy for LOIRE that allows an intermediate model in the growth stages to switch between being a student and teacher, reducing catastrophic forgetting during growth. Exper- iments show that LOIRE can reduce computational expenses by an average of 29.22% while retaining equivalent or better downstream performance.
lifelong learning have tried solve this problem
INTRODUCTION
Large-scale pretrained language models (PLMs), such as GPT, BERT, and T5 (Devlin et al. 2018 Brown et al. 2020 AAchiam et al. 2023 Raffel et al. 2020), have demonstrated impressive perfor- mance across various tasks. Large PLMS can require long time and lot of computation resources to train from scratch on huge amounts of data. However, when used in practice, PLM's performance may decline since new data from other sources frequently comes in that was not previously available for training. This alters the domain of data distribution (Guo & Yu 2022). The mismatch hinders the widespread use PLMs in real life. However, training from scratch is time and computation inten- sive. particularly if the PLM is large. Although researchers investigated involving lifelong learning (Mehta et al. 2023) for the PL pre-training phase to prevent periodic re-training of PL.Ms from scratch for new data, which has the potential to reduce computing and time needs, there still re- mains an efficient problem unsolved. Recent research (Jin et al.) 2021 Chen et al. [2023] Qin et al.) 2022) have developed effective lifelong learning pipelines for PLMs by employing model growing approaches (Chen et al. 202T Wang et al. 20233 Yao et al.) 2023 to gradually increase model ca- pacity, thereby enhancing PLMs' ability to quickly adapt to knowledge. SOTA model growth
'These authors contributed to the work equally.
1 Corresponning author.

---

Published as a conference paper at ICLR 2025
strategies rely sequential training with scaling parameters. During growth, each larger model inherits knowledge from the preceding smaller model by reusing its parameters to initialize.
Despite their success, the model growth strategies adopted for lifelong learning continue to face chal- lenges. 1) Growth schedules, which specify when and where to expand the model structure, have mostly been examined for Transformer considering the number of layers and the width of the Feed- Forward Network (FFN) (Gong et al., 2019 Gu et al. 2020). However, devising an effective sched- ule that includes all possible growth dimensions remains difficulty. Growth operators, which refer to the steps taken during growth to inherit knowledge from the previous model, have to assure function preservation. Function preservation is a theoretically essential property for growth oper- ators since it ensures that the initialized larger model behaves identically to the preceding smaller model (Chen et al. 2013). This property has proven beneficial for both knowledge inheritance (Chen) et al. 2015: 2021 Shen et al. 2022) and training stability Yao et al. 2023). Existing efforts that merely aim for non-strict function preservation may not inherit all relevant information, resulting in inadequate growth. This, in turn, has an impact on the lifelong learning effect because it creates a wide function gap during rapid growth, preventing further gains training dynamics. 3) a lifelong learning scenario, the data distribution is constantly shifting new domains keep emerging (Ke et al., 2023). Existing model growth methods frequently assume that pretraining follows the same data distribution, resulting in catastrophic forgetting of previously learned knowledge.
We propose the LOIRE (Lifelong learning framewOrk on Incremental data via PLMs gRowth Efficiently) address these issues. Our contributions are listed as below:
(1) We present a novel plug-in layer growth operator that replicates the selected layers and inserts them between the original layer and the subsequent layer. This design is inspired by recent research findings that show similarity in pattern classes between neighboring Transformer layers (Delétang al. 2023). We adopt the concept of residual connection by introducing a set of gates to intention- ally skip the newly added layer in the initial training. thus ensuring the function function-preserving property. prreerinp.....ving..
We also show that this layer growth is theoretically
(2) In addition to the proposed layer operator. we provide a systematic definition for multi- dimensional operators that includes the hidden dimension, the FeedForward Network dimension, and the number of heads in the multi-head attention. Next, we build growth schedule that gener- ates an optimal growth sequence by combining multi-dimensional operators in the field of lifelong learning, allowing for efficient PLM training.
(3) Finally, we suggest an iterative distillation warmup strategy for LOIRE. According to a previ- ous study, the difference in model size between teacher and student can affect the distillation performance (Mirzadeh et al. 2020). In our case, the difference in growth dimensions during every stage of growth results in a sumilar phenomenon. We suggest allowing an intermediate model cre- ated throughout the growth steps to switch between being a student and a teacher during the iterative distillation process. According the iterative distillation warmup. LOIRE could accommodate the new data distribution without forgetting earlier distributions during model growth.
2 METHODOLOGY
Fig T is an overview of the whole paradigm. LOIRE combines multi-dimennional function- preserving growth operators with an optimized growth schedule. Each forward pass in the growth schedule yields a larger model by increasing the capacity of the previous smaller one and training on newly acquired data. To reduce the disparity between the data distribution before and after model growth, we use an iterative distillation strategy to warm up the larger model generated after each growth step.
2.1 PRELIMINARY
In this study, focus on the Transformer architecture VVswwan et al. 2017) that is prevalent in existing PLMs.
Hidden states H' represents the input for the Transformer layer I, which is bi-dimensional tensor with s and h being the sequence and hidden dimension. When the h changes, it affects every module of the Transformer structure. We overlook the position embedding in this work it does
2

---

Each Transformer layer | contains the modules that are important for the growth approach, which are described below: Published as a conference paper at ICLR 2025
Uiwer
Seiu Uieer
M
M.
LayerL
Mas-
Operators
Growth
D2
Lepu
sttettonn
Layar_1
Layer_1
f.
N,
Layer
D2
multi-stages Innitial Hidden dimension FFN Multi-head
(a) Optimized growth operators and schedules with strict function preserving
(b) Multi-stages Model Growth
The overall framework of the proposed LOIRE. (a) Illustrates the growth operators and schedules over all four possible Transformer dimensions, including multi-head attention, hidden dimension, FFN, and plug-in layer growth operators. (b) Describes the multi-stage model growth procedure with an interactive distillation approach over emerging datasets.
Figure 1:
not affect the expansion process. The hidden states are iteratively passing through the Transformer layers: H' Trans (H'-),, [1, L]. where L denotes the total number of the layers.
Multi-head attention (MHA): Multiple parallel self-attention heads make up MHA. The input H of each layer is fed into the MHA mechanism, which can be formulated as follows:
K: Q Vi
W/K///V H.
axd exa 6xh
hxd
Hasad, = Attention (Q., K., Vi) = softmaz
Q. QQ
(I)
vd
H MAA .ch
MHA(H) [Haoods Haeadyl W (αxл)
where H is applied to linear projection for generating queries, keys and values(Q/K/V)) utilizing different weights(W K/Q/V) for each transformation respectively. Hhead, signifies the output of the i-th attention head with being the total number of heads. The output linear matrix po generates the final result H MHH which is then delivered to the Feed-forward network.
Feed-forward network (FFN) is a Multi-Layer Perceptron responsible for applying non-linear transformation to I MHA if FFN's dimension its internal representation):
H PPN = FFN(H MHA) GELU MHA W AXT
+ bf1) W'2 8/2 Xx3
(2)
fxh .8
Layer normalization (NORM) and residual connections interconnected, facilitating the miti- gation of internal covariate shift and enhancing gradient flow in both MHA and FFN.
MORMMRA
Loyer + H MMA) H NORMPFN Layeer.orrm(H MHA H MPN)
(3)
where H NORAMMA is the normalization output of MHA the same for FFN module.
2.2 PROBLEM FORMULATION
Lifelong learning involves training an initial PLM (1) on corpus Dj and then updating it to (0) with each D; E D.. .r. learn new knowledge while retaining old knowledge. D [D..... Dr} } is an incremental data stream that consists of corpus from different stages T. For each D., there may contain data from multiple domains. We aim efficiently initiate M reducing the gap with earlier (t-1)) in acquired knowledge.
Lifelong learning with an efficient model growth strategy can be further formulated as below. Growing the ((1-1) to M0) is process with multi-stages, which means (((-1) needs go through
3

---

Published as a conference paper at ICLR 2025
K stages eventually grow into M M M M Let L. be the target loss function while T be the total pre-training time, the optimized objective of lifelong training can be defined as:
E (M; 0x)] 0% - ((0+) + -x-11
(4)
s.t.
where ε is the mtati-stage growth schedule. is the growth operator set with a as operator for one dimension, which will introduce in subsections 2.5 and 2.3 separately.
2.3 GROWTH OPERATORS WITH STRICT FUNCTION PRESERVING
We introduce the growth operator set e defined in equation 4 The term "growth operator" yp denotes the actions performed during the growth phase to transfer knowledge from the prior model. The growth operator set consists of several growth operators yp, defined v {ølk E w.. In this work, we consider all four Transformer dimensions for growth, including layer, multi-head attention, feed-forward network, and hidden states, defined with & {Slayer, Pmnaa i Fhidden )
Function preservation is critical property for each growth operator yp because it ensures that the initialized larger model performs exactly the same as the preceding smaller model (Chen et al.) 2015).
Specifically, suppose PLM trained from the previous stage is represented Ma-1( 0x-1) with input x and parameters 0x-1- The growth operator yp constrains the larger model's increased param- eters 0 This is achieved by choosing new collection of parameters 02 for the larger model Ma that satisfy specific criteria defined as below:
Ve, Mu 0x) (x: ) 02_1) M2:1(2;00-)
(5)
Next, elaborate all four growth operators vé (Player, Prahas ff f Phidden) ) and demon- strate the validity of strict function-preserviin transformations for each.
We first introduce imhha fnr Phiddenn leaving the layer operator Gllager introduced in the next subsection. Pmho: ff and Phidden are defined according to Gesmundo & Maile (2023)'s 's work. We simplify each layer's initial input into H. Due space limitations, only present the straints of growth operators with strict function preservation on the parameters of each module and provide proofs of function preservation in the appendix
MHA growth operator 9mhha refers to the act of introducing new heads within the multi-head attention module. As mentioned in equation [T] the hyper-parameter a controls the scaling of the multi-head attention dimension. When the head number increases from to a2, we keep the weights of the former heads fixed while assigning random values to the weights of the new heads.
w K/q/V any-value
w K/Q/V-
is a1
(6)
a; < 1<15aa
As the number of heads increases, alterations are also observed in the size of the corresponding weight matrix 0 in equation
We set the expanded portion of O to be zero matrix N as:
wo
Wo (og x
(W
(#2xd))x
L((ag x1(x))xb
FFN growth operator Pffn can be scaled up by increasing its internal representation's dimension- ality. In equation 2 the scaling of FFN expansion is controlled by the hyper-parameter f. Given example, when the FFN's hidden dimension is increasing from fi to f2.
Transformer layer we set the weights of extended W to be N, while the extended part of Wh and 6 are initialized arbitrarily, written as R:
w'1 (W'1)' Axfi 8xf2
W hx
Wig
(111) 2x12
y==
b/1 px/2
R Wig
xx(
axfi
xx
W
(8)
(W'2))
12N
Lia 11
4

---

Published as a conference paper at ICLR 2025
Hidden dimension growth operator Phidden is used to expand the dimension of the representation, which is originally sent into the Transformer layers. The scaling of hidden dimension expansion is controlled by the hyper-parameter h. When the hidden dimension the representation is increasing from b to h2, we set the extended portion of H to be N:
=
H xx
(82-h11)
(9)
Then each module in Transformer exhibits variations in the scaling for the parameters with hidden dimension expansion.
In the MHA module, we the extended portion of W to be N, and the extended weight matrices of K, Q, and for each head are initialized randomly:
WK/R/V hyxd
w//q/v = (W//Q/V)'
hyxd
(h2- xxx
R
(10)
W (xx
wo xxz
N
(uxd)x(hgg
k1)
In the FFN module, we set the extended portion of W and b/2 to be N, while the extended Wh is initialized randomly:
Wh by x R (h2 -
Wha t (W")) [
Raxs
(II)
w'a 1x
(prlt)'
8/2
8/2
6'2 pxkg
N (M2
h1)
According to (Gesmundo & Maile 2023), with an expansion of the above four dimensions, the outputs of these modules remain unchangeable, given the same input. The detailed proofs are listed in Appendix G
2.4 LAYER GROWTH OPERATOR
Typically, mainstream layer growth operators in the depth dimension stack the entire Transformer layer itself until they achieve the target layer number, breaking the integrity of full-dimensional cunction-preserving (Gesmundo & Maile 2023). On the other hand, recent studies have demon- strated that adjacent Transformer layers have similar pattern classes (Delétang al. 2023 which are overlooked by existing growth operators.
Considering the factors mentioned above, LOIRE employs an optimized plug-in layer depth growth operator Player that replicates the selected layer and inserts it between the original layer and the subsequent layer. Motivated by the advantages of residual connections in improving the model's representation and learning capabilities, we include them in the layer growth operator to deliberately skip the newly added layer during initial training, ensuring function preservation.
For a model with L layers, as illustrated in Fig [aa,, (),, we replicate the selected layer and place between Layer_/ and Layer-(l + 1), recognizing it as Layer-Cl. The output of each duplicated Layer_Cl is computed as follows:
n = A + + (1- Tiaasas [1,
(12)
In equation 12 we simulate the process of residual connections using the hyper-parameters
{Ai.... } to assure function preservation in the layer operator. As result, the newly added
layers have the least impact the model during the initial training phase following expansion.
With X À setting to 1, the ln layer is skipped leveraging the residual connection, as proven below. Noticing that we simplify f(a (x:) (((()) to express nested formula and H' is the input of layer defined in 2.1
achieve function-preservin
transformation
CL ((t-)( Trane(H"~1
HOL HoLL
H
Trans 0L 2) Trans (H
H0I H²
5

---

Published as a conference paper at ICLR 2025
During training, X eventually drops to O, and residual connections vanish, resulting in the same structure as vanilla Transformers. Practically, duplicating all layers is unnecessary. You can repeat any of the layers depending the model's growth requirements.
2.5 OPTIMIZED MULTI- STAGE GROWTH SCHEDULE
We introduce the details of the growth schedule defined in equation 4 consists of multiple stages of sequentially growing the model in different dimensions. As we focus on the Transformer-bast architecture, dimensions include layers, MHA, FFN, and hidden states, as described in Sectior Previous studies on lifelong learning have not fully explored the impact of growing sequentially throughout multiple stages(Qin et al. 2022). In order enhance efficiency, it is advisable create a well-optimized growth schedule that involves a sequence of gradual growth stages, each of which demonstrates rapid learning skills. This approach avoids the immediate expansion of all dimensions simultaneously.
According to Yao et al. 2023 growing the layers and heads in later stages and having larger hidden dimension in earlier stages can lead better model performance. Based on this finding, the E is formulated as follows, where K set to 5 in this situation. We present an empirical optimiza- tion approach with ablation studies illustrated in Subsection 3.3, deferring the determination of a theoretically optimal growth schedule to future research.
E == (2; 01), M2 02,, Ma 0s), Ma (x; 04), (z; 0s)} 81 > @2lihidden (0 + 0.] 0a (0,) 0g] + 0a(Ppna (0)) 0s) + 0s (Ptaave (0,)) + 0s)
(14)
2.6 ITERATIVE DISTILLATION WARMUP
LOIRE also designed re- re-training process after each growth stage that used an iterative distillation warmup strategy to prevent catastrophic forgetting of previously learned knowledge and overfitting new data. According to previous study, the difference in model size between teacher and a student can affect the distillation performance (Mirzadeh et al., 2020). A similar phenomenon also holds for our model growth scenario. We propose allowing intermediary models generated throughout the growth stages switch between student and teacher roles during the iterative distillation process.
For the current interation output model (t there exists a teacher model set Mteacher consisting of previously generated models Mteacher = (M(1) (() It should be noted that each M consists of K intermediate growth models. Distillation aims minimize the weighted sum of the difference between the distributions of each model in Mteacher and the current student model M() at the token level through Kullback-Leibler divergence. For each M E Mteacher (j S - 1) and an input sequence X {æ1, ..., Im), the knowledge distillation loss between (t) and M)) is
calculated as:
P(u.|Riem 0z) P(un|atemm 0z)log Pleulsiem: 0,) Stt-1
Lacarsii =
Σ
Σ V.VV
(15)
where Fiem is from the ground truth sequence. V denotes the vocabulary set and en is the n nith word in V. By and 9r are parameters of the teacher M E MMeocker and student model M() respectively.
The final loss function Liinal is a combination of normal language modeling and distillation loss: Liinnt- = LLM + Σ BBL where 3, E 3 and B = {,,1..., 3 is set of hyper-
j5(-1
parameters to control the contribution of each teacher model to the final loss. After the warmup, we obtain the M (0)+, which successfully inherits the knowledge from D = (D..... De} and is also well-prepared be the teacher for the next model growth stage.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Pre-training Dataset. We use five different domain datasets for growth pre-training, including the combination of Wikipedia & Book Corpus (Zy/Zhu et al. 2015), Realnevs(NEWS)Jeellen al, 2019) Amazon Reviews (REV)(He & McAuley 2016), Biomedical papers (BIO) et al. 2019)

---

Published as a conference paper at ICLR 2025
and Computer science papers (CS) et al. 2019), publicly available HuggingFace Hut In each domain, we sample out 10 GB of data and divide it into pre-training and recovery memory data for distillation in a 9:1 ratio. Fig. 2 shows the correlations among these five datasets. We also use Redpajama Weber et al. 2025) to generate initial version models for further evaluation, and then the five domain datasets proceed with training during lifelong learning.
Baseline models. We compare different kinds of baselines for both of the commonly utilized GPT-style (decoder-only) and BERT-style (encoder- decoder) PLM architectures. GPPTs baselines in- clude GPT S and GPT L that we train from scratch using the same domain pretraining dataset mentioned before with different parameters (small: 27.59M, large: 104.78M). GPT shares settings with GPT.L incorporating recovering period via retraining re- served data for each model growth stage. We compare with other lifelong learning methods ELLE QQin et al. 2022), distillation methods Token- Caapee-kDCaappelaazzo al. 2023) and ER (Chaudhry al.) 2019), which are also GPT-based. 2)BERR--style baselines include model growth methods LiGO (Wang al. 2023).
g
No Reeiee Nax
100.00
14:35
8.25
12.90
14.25
100.00
8.13
7:25
8:16
0.25
6:13
100.00
13.20
9:17
9:16
7:29
25
100.00
A0R
CA
Figure 2: Vocabulary overlap (%) between five domains. Vocabularies for each do- main are created by sampling the top 10K most frequent words from each domain.
Variations of LOIRE. There are three variations of LOIRE. LOIRE-GPTI and LOIRE-GPT2 are both GPT-based models. We multi-stage train LOIRE-GPTI using our suggested model operators and schedules on five domains of pre-training data. The difference between LOIRE-GPTI and LOIRE-GPT2 is that for the initial stage, M(1 of LOIRE-GPTI is trained from scratch, (1) of LOIRE-GPT2 is initialized from a PLM starting point, which train using Redpajama Weber et al., 2025) following the data mixing strategy of LLAMA2 Touvron et al. 2023b). LOIRE-Bert shares the same initialization strategy as LOIRE- GPT2, just with a different architecture. The English wiki dataset is then utilized for growth training to ensure compliance with LIGO.
More detailed settings for both the baselines and LOIRE variations provided in Appendix D
Model growth schedule setting: Based on Subsection 2.2 we illustrate Table I the growth schedule for both GPT and BERT-based models.
Table 1: The initial PLM Mj is around 27.59M parameters training from scratch. (384,1024.6.5 corresponds to the following 4 dimensions: hidden dimension, ffn dimension, head number, and layer number. Growth schedules expand dimension at time, illustrated in red.
Mi
M2
Ma
M4
Mr
Model size
27.59M
62.25M
71.69M
71.69M
104. 7M.78M
Growth schedule
1384,1024,666)
(768, 1024.6.6)
(768.2048.6.6)
(768,2048.1266)
(768.2048. 12,112
Growth operator
(Phideen
40/5n
Gomha $
Player
3.2 RESULTS AND ANALYSIS
We design a set of experiments to validate LOIRE To begin, we evaluate LOIRE's performance on the pre-training and downstream tasks. We also evaluate the amnction-presevving effect using the pre-training findings. Next, we compare FLOPS and wall time costs to determine training effi- ciency. Finally, we conducted ablation studies to investigate the separate impact of growth operators, schedules, and distillation.
Performance on the pre-training. We adopt AP(Aver Perplexity) and AP+(Averege Increased Perplexity) following (Chaudhry et al. 2019) to evaluate the pre-training performance, the lower the better. The details of metric implementations are listed in Appendix D.3
The results are illustrated in Table 2 and Figure The following insights can be drawn from Table compared to all baselines, LOIRE considerabl bly surpasses others by achieving the lowest AP+,
https//tuuggnnffacccoddaaa coolatasssss

---

Published as a conference paper at ICLR 2025
Table AP represents the model's ability to learn from emerging data. AP+ represents the model's capacity to maintain old knowledge, i.e. not forget it. M1-M5 are models trained at multiple stages with the five-domain pre-training data. Inat denotes the initial ppl after growth, while Final means the ppl before the next growth.
Schedule
Mi
M2
M3
M4
Mr
Metrics
AP
AP+
AP
AP+
AP
AP+
AP
AP+
AP
AP+
Growing M1 to Ms. from scratch
GPT.S
38.69
48.83
23.44
82.15
96.61
82.30
117.04
59.59
83.25
GPT.I. 22.43
27.69
12.86
48.87
55.50
47.75
64.3
36.01
GPT R
22.43
23.78
3.42
25.92
6.37
22.61
9.14
20.66
8.58
Token KD
38.69
48.48
22.93
56.94
90.15
80.53
112.19
56.77
78.35
38.69
42.28
9.87
45.30
14.45
40.24
21.67
35.94
19.22
ELLE 38.69
34.33
-0.79
31.72
1.21
25.595
4.3
21.75
LOIRE-GPTI
38.69
31.72
-5.32
29.18
-2.68
24.63
-0.94
19.19
-3.84
Growing Mi to Ms from loading PLM
LOIRE-GPT2 32.39
26.60
-5.79
25.55
-3.95
25.28
-2.90
23.24
-4.2
Growing M1 to Ms from loading an PLM
PPL
Init
Final
Init
Final 5.68
Init 5.9
Final 4.98
Init 5.18
Final 4.7
Init 4.79
Final 3.91
LOIRE-Bert 457.41
6.72
7.48
suggesting that LOIRE retains prior knowledge while acquiring knowledge through lifelong learning. Specifically, the AP+ score decreases by 6.97 when compared to the Sota lifelong learning method, ELLEE Figure 3 provides additional evidence that LOIRE shows the most rapid decline in AP+, suggesting superior model performance under comparable training settings. It is noticed that baselines with more parameters (GPT_L and GPT_R) have a smaller AP the early stages of training. This shows that big PLMs are extremely effective at generalization. As the training data grows, LOIRE gradually shows that it is better keeping information from being forgotten and ultimately outperforming other baselines.
(a) AP
(b) AP+
Figure 3: The multi-stage pre-training trends curve over AP&AP+ on GPT from scratch.
LOIRE-GPT2 and LOIRE-Bert load the checkpoint trained from scratch on Redpajama continu- ally grow training. As shown in Table experiments prove that LOIRE works both on pre-training from scratch and continually training from loading the checkpoint.
To get closer to the parameter scale of LLM models now in use in industry, particularly on the ter- minal side, we expand the model parameters from 177.45 to I.I on the GPT structure known as LOIRE-1.IB. Specifically, we use the CC datase growth. Then use the same five domain datasets mentioned above to proceed with the lifelong learn- ing. In contrast, train 1. IB GPT structure model (GPT-1.1B) from scratch with growth. The other settings GPT-1.IB are the same as LOIRE-I.IB Table shows the AP and AP+ corre- sponding each expansion of LOIRE-1 10REE-1.1B In the last round of expansion, the AP of LOIRE- decrease by about 6.22% compared to GPT-1.1B (same configuration as M2). Experimental results show that our method is still applicable for larger models and can effectively reduce catastrophic forgetting even as the model grows.
2 to train initial 177.45 M base model for
hhups:/commmnnrawllorg
8

---

Published as a conference paper at ICLR 2025
Table 3: AP& AP+ LLLIR--I.IB. compared to the baseline with fixed I.IB parameters.
M1
M2
M3
M4
M5
Size
177.45M
455.52M
606.57M
606.57M
1. BB
Schedule
110243072.16123
122,6.072))6111
(2018.614,16612)
(2088611432.1))
(2,35.6
Metric
AP
AP+
AP
AP+
AP
AP+
AP
AP+
AP
AP+
GPT-1.IB
14.06
18.06
9.73
27.74
21.50
34.70
57.38
20.51
19.52
LOIRE-1.1B
16.76
21.68
12.93
33.33
32.62
30.59
36.97
14.29
3.49
Empirical evaluation for function preservation. We empirically evaluate the function preservation effect over ppl of LOIRE-Bert. To the best of our knowledge, other works only prove function preservation theoretically without experiments. For LOIRE-Bert, which is growth trained using the English wiki data for all the expansion stages, we listed the ppl trends the bottom of Table 2 The M1 to M2 stage implements the growth operator Phidden, resulting in 0.76 ppl difference between the initial M2 and final M,. Similarly, our proposed novel growth operator Player has the best function preservation with only 0.09 ppl difference between the initial Ms and final M4- This observation reveals that, while all of the growth operators have been theoretically proven to be stringently function-preserving preserving, there are still deviations in the outcomes of the experiment.
Table 4: For the GPT style models, we compare the ratio of the computational costs (FLOPs) of LOIRE-GPTI to GPT_L and GPT_R on the 5-domain dataset's pre-training. For the BERT structure, we compare the train wall time with LIGO.
GPT base
BERT base
FLOPs %)
Method
Wall Time
Schedule
M1
M2
Ms
M4
AVG
LOTRE
21.20
78.42
85.69
85,69
76.20
LIGO
48h.25min
EoTRE
19.08
70.58
77.12
77.12
70.78
LOIRE-Bert
28h,48min
GPT.R
Training efficiency. From computational cost viewpoint GPT structure, as seen Table 4 LOIRE achieves an average decrease of 29.22% compared to GPT R. This leads to substantial sav- ings in computational resources the train wall times of LIGO and LOIRE. LOIRE saves around 40% wall time during training. The results demonstrate that LOIRE can effectively save training time and improve training efficiency.
greatly enhances efficiency. For the BERT structure, list
Table 5: The final PLMs performance after multistage model growth pre-training on representa- tive downstream tasks including MNLI & QNLI, Hyperpartisan& Ag news. HELPNESS & IMDB, CHEMPROT & RCT, ACL-ARC & SCIERC for domains WB, NEWS, REV, BIO, and CS respec- tivelyGGuruaanga et (2020).
Domain
WB
NEWS
REV
B1O
CS
Avg
Task
MNLI
QNLI
Hyper
Ag news
HELPNESS
IMD8
CHEM
RCT
ACL-ACC
SCIERC
GPT
73.39
80.79
77.08
92.51
86.16
91.58
78.01
86.99
69.53
80.00
81.61
GPT L
GPT R
79.53
82.35
76.38
93,33
86.93
93.08
80.57
87.36
69.53
82.70
83.18
Token KD
75.58
80.61
75.16
92.67
86.32
91.55
76.12
86.91
69.53
78.54
81.30
ER
77.05
92.61
91.52
77.98
70.31
82.70
ELLE 78.12
78.75
86.59
79.98
87.00
73.43
79.79
83.35
LOIRE-GPTI
79.60
84.34
81.68
93.12
87.16
93.57
81.27
87.40
78.13
82.08
84.84
Performance on the downstream tasks. As shown Table we present the final performance of PLMSs on representative downstream tasks. It can be observed that LOIRE achieves the highest performance across almost all downstream tasks, suggesting that the knowledge acquired during model growth can be effectively utilized and leveraged for downstream tasks. To further validate the capabilities of LOIRE, we proceed with fine-tuning BERT-style models. In Table 6 we list downstream tasks of GLUE and SAQuAD according to LIGO(Wang et al.
the performance 2023). To be noticed that the LIGO's performance is from their paper. Although we recreated LIGO using the same training data LOIRE, the results not comparable to their original

---

Published as a conference paper at ICLR 2025
work. Performance demonstrates that our method achieves comparable performance while saving computational costs.
Table 6: Downstream tasks performance on GLUE and SQuAD.
GLUE tasks
SQuAD tasks
Avg.
SST-2
MNLI
MRPC
CoLA
QNLI
QQP
v2.0
Method
GLUE
Acc.
FI/EM
LIGO
88,42
79.29
84.31
62.09
88.07 90.03
88.81
71.24/67.17
81.83
LOIRE-Bert
92.07
82.88
86.7
81.22
76.98
79.08/75.61
86.60
3.3
ABLATION STUDIES
Growth schedule: To validate our method's growth schedule, we reverse the partial expansion order. This involved converting the order of the dimensions from thidden=>fhn>hhea--llaaa (laye==>fhn-hhed--hddd))) We refer this growth schedule as SCHL-Reverse and compare the performance between SCHL-Reverse and LOIRE after training on five domains in sequence. As shown in Table 7 our model, outperforms the schedule of SCHL-Reverse with growth in all dimen- sions, supporting the optimum sequence schedule in our design.
to
Table 7: The pretraining and downstream performance of SCHL-Reverse after 5 domains' multi- stage growth training, compared to our method.
Schedule
M
M2
M4
Ms
AF
AFe
AP
AP+
Pre-training
LOIRE-GPH
38.69
31.72
-332
29.18
-2.88
24.63
-0.94
19.19
-3.4
Metric
SCIERC
Downstream
7688
9127
77.34
83.12
LOIRE GPTI
79.00
8434
81.58
93.12
87.16
93.37
81.27
87.40
76.13
82.08
Growth operators: We present the impact of growth operators, replacing MHA,FFN,and hidden state growth operators the below two methods: Random and Zero. Random refers randomly initializing the extended portion of the parameters while keeping the remaining settings the same. Zero identical to Random, except that it uses zero initializa- tion rather than random initialization. From Figure 4 after initial loading, we can see that LOIRE'S AP and AP+ are significantly lower than those of zero and random, proving that our operators are superior to zero random in terms of function preservation. We leave the analysis of scheduling and distillation to separate section in Appendix(F] F22
(a) AP
(b) AP+
Figure 4: AP and AP+ at the initial loading on the validation set for each seen domain after expansion.
CONCLUSION
In this study, we investigated methods of lifelong learning that incorporate current advances in the model growth field. We provide a novel plug-in layer growth operator with residual connections that skips the newly added layer during initial training, ensuring function preservation. We also introduced an iterative distillation strategy that allows a model in the growth stages to switch between being student and teacher, hence reducing catastrophic forgetting during dynamic learning after growth. Experiments show that LOIRE reduces computational costs by an average 29.22% while retaining equivalent superior downstream performance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020
Umberto Cappellazzo, Muqiao Yang, Daniele Falavigna, and Alessio Brutti. Sequence-leve knowl- edge distillation for caaseinncremetal end-to-end spoken language understanding. arXiv preprint arXiv:2305. 13899, 2023.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokamia, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 119902.04440 10486, 2019.
Cheng Chen, Yichun Yin, Lifeng Shang. Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint arXiv:2110.071433 2021.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Ne(2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: 1.1.0661,, 2015.
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with ceurrbttion-spccialiae experts. In International Conference on Machine Learning, pp. 5383-5395, PMLR, 2023.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-333) 2021.
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint 2rXiv:2309.1066 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00 2018.
Andrea Gesmundo and Kaitlin Maile. Composable function- prrettooo-reeriinggggg expansions for transformer architectures. arXiv preprint arXiv:2308.061103 2023.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pp. 2337-2346. PMLR. 2019.
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han, On the transformer growth for progressive bert training. arXiv preprint arXiv:2010.1 12562, 2020.
Xu Guo and Han Yu. On the domain adaptation and generalization of pretrained language models: A survey. arXiv preprint moXiv:2211.03/55 2022.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. arXiv preprint arXiv:2004.1 109444 2020.
Ruining He and Julian McAuley. Ups and downs:
Modeling the visual evolution of fashion trends with one-class collaborative filtering. proceedings of the 25th international conference on world wide web, pp. 507-517, 2016.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.

---

Published as a conference paper at ICLR 2025
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Wen Li, Xiaokai Wei, Andrew Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08534, 2021.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. Adapting a language model while preserving its general knowledge. arXiv preprint S.Ki::2301009988 2023.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1et.. 11692, 2019.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quanti- zation for vision transformer. Advances in Neural Information Processing Systems, 34:28092- 28103, 2021.
Kyle Lo, Lucy Lu Wang. Mark Neumann, Rodney Kinney, and Dan Weld. S2ore: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02700 2019.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investi- gation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24 1214):1-50, 2023.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv: 1904.000300 2019.
Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, and Sanjiv Kumar. Efficient stagewise pretraining via progressive subnetworks. arXiv preprint arXiv:2402.059/3. 2024.
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Elle: Efficient lifelong pre-training for emerging data. arXiv preprint arXiv:2203.063: 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1rXiv:1600.052500 2016.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 27Xiv:2204.06//5, 1(2):3, 2022.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference on Machine Learning, pp. 19893- 19908. PMLR, 2022.
Hugo Touvron, Thibaut Lavri Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1377 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. arXiv preprint arXi:
2023.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: open dataset for training large language models. Advances Neural Information Processing Systems, 37: 116462-116492, 2025.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pre- trained language model in continual learning: A comparative study. In International conference on learning representations, 2021.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407 10671, 2024.
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. faster language model pre-training masked structural growth. arXiv preprini arXiv:2305.02869 2023.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Yukum Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedingy of the IEEE international conference on computer vision, pp. 19-27, 2015.
A LIMITATIONS
First, wish to explore the potential for large-scale PLMs, although the largest models in experiments have 114 million parameters because of limited computing capacity and budget, which has gap with existing PLMs with parameters up to 65 billion, such Ilama2(Touvro) et al., 2023b According to our knowledge, this constraint exists in the vast majority of research projects.
Second, although method theoretically proves strict function preservation, there are still some deviations from the theory in the actual experimental results. Additionally, the existing research lacks empirical verification of function preserving theory, leaving us to explore and analyze this issue in the future.
Third, LOIRE performs worse few downstream tasks than other baselines, necessitating more analysis from the perspective of training data sample methodologies. In addition, we repeat the baselines in our experiment settings to confirm that the training procedures and data are consistent. Some of the work. such as MSG Yao et al. 2023), has not been properly reproduced, leaving room for future research.
A further limitation is that we focus mainly in pre-training language models. Investigating tech- niques for other data-driven modalities, such as images and video, is also attractive.
B RELATED WORKS
Efficient Pretraining attempts to reduce FLOPs while pretraining. A few recent works focus on stagewise efficient pretraining (Panigrahi et al. 2024). progressive pretraining, model reusing (Chen et al.) 2015 2021 Wang et al. 2023) Yao et al. 2023) by keeping the function of a pre- trained model while growing the model size. This gives compatible larger model starting state
13

---

Published as a conference paper at ICLR 2025
with great performance. Net2Net (Chen et al. 2015) is the first to propose the concept of function- preserving transforms by expanding the width dimension by splitting neurons and growing depth by adding identity layers. ber12BERT (Chen et al. 202T) applies function lunction-preserving concepts to the Transformer, extending Net2Net. LiGO (Wang et al. 2023) recently employs a trainable linear operator acquire an efficient expansion strategy. Unlike other studies, our method attempts to build full-dimension transfer that makes use of the entire smaller model.
Lifelong learning, also known as continuing learning(De Lange et al. 2021), has gained massive popularity in recent years. According to Gururangan et al. (2020). second phase of in-domain pre- training comain-adaptivv pretraining) improves performance in both high and low-resource con- texts. The studies conducted by Jin et al. (2021) and Wu et al. (2021) evaluate several continual learning algorithms on large PL.Ms using standard parameters. These studies also track the perfor- mance of these algorithms on downstream tasks. Another potential approach for continuing learning the utilization of key-value methods derived from computer vision Van Den Oord et al., 2017 Liu et al. 202 Ramesh et al. 2022). The most similar work to ours is ELLE (Qin al. (022), which uses cuunction-preservee
model expansion and pre-trained domain prompts to efficiently pre-train for emerging data over time. The main difference is that growth method for all feasible domains, whereas ELLE only grows in limited domains with non- strict function preservation.
strict function preservation model
The model growth schedule and the growth operator are two crucial areas of research for model reuse progressive pre-training. The growth schedule tells the model when and where add more parameters, and the growth operator describes the steps that are taken during growth to get knowl- edge from the prior model. Goog et al. (2019) proposes growth schedules that stack the Transformer layers to transfer knowledge from a shallow model to a deep model. The setups that are most similar to ours are msg Yao et al., (023) and the work conducted by GoesgeGGsmmnno & Maile (2023). MSG incorporates growth schedules that encompass all possible dimensions and growth operators while strictly preserving their functional integrity. Google proposes six composable transformations for gradually increasing the scale of Transformer-basee neural networks while preserving function- ality. However, MSG does not provide a uniform framework for the optimal integration of opera- tors and schedules, and Google's 's work is lacking in schedule studies, allowing more space for our method's innovation in the field of lifelong learning.
C ALGORITHM
Algorithm D summarizes LOIRE for growing Transformer in lifelong learning.
D EXPERIMENTAL SETTINGS
D.1 IMPLEMENTATION DETAILS
For GPT-based model growth, we begin by training a starting point model for 20,000 steps in the WB domain. During each model growth stage, we train the expanded model for 20,000 steps using incremental domain data. For the Bert-based structure, we train for 20,000 steps in each expansion stage, totaling 100,000 training steps on the Wiki Dataset. The initial models of both GPT and Bert trained for 20,000 steps the Redpajama dataset. Prior to each model growth stage, we employ a warmup strategy to retrain the model in 5,000 steps. For Bert-style structure, we train for 20,000 steps in each expansion stage, totaling 100,000 training steps the RoBERTa structure (Liu al., 2019) model.
For hyper- parameters, linearly increase the A Ā value in the layer growth operator from 0 to in 5000 steps per growth stage. During the iterative distillation in the warmup period, the best 3 B is set to 0.1 and vanishes after 1,000 steps. Adam is chosen as the optimizer. Our setup consists of four-core CPU and eight NVIDIA Tesla 1100 GPUs.
We list the seper-parameters used in the GPT architecture's 's domain downstream experiments in Table 8 The GLUE benchmarks for downstream tasks, such as MNLI and QNLI, are based on Ott ct al. 2019). SQuAD benchmarks are based on Rajpurkar et al. (2016). More downstream tasks are implemented, as detailed in Gururangan et al. (2020)
14

---

Published as a conference paper at ICLR 2025
Algorithm 1 LOIRE on Transformer with Growth Operators and Schedule.
Input: An initial Transformer M) trained on D:-1, with Multi head attention number being 01, FFN's hidden dimension being fI. Hidden dimension of input being hi, and number of layers L1. Output: An larger Transformer ( with Multi-head attention number being a2, FEN's hidden dimension being f2, Hidden dimension of input being hu, and number of layers L2.
1: Operators: き [Phidden; Pffn, Pmhaa Player
(Mx(Ox)]k=L
2: Schedule:
0. (((t) ) +8x-11 11.11
3: for i to k (M to M do
4:
& == Phoddeen then
6:
Hidden dimension growth: (Phiddon tu + h22
7:
end if
8:
A == Pffn then
10:
FFN hidden dimension growth: 8pssn : fi +
11:
end if
12:
13:
if 4 Pmha then
14:
Multi-head attention number growth: (emha : 01 + 02
15:
end if
16:
17:
if
Player then
18: Layer number growth: Gmhha I1 - L2
19: end if
20:
end for
21:
M (0,-)) (0.)
22:
Train the larger model ( on new Data Dr
23:
Utilize the iterative distillation generate (++ in the warmup period
Table 8: Hyper-parametent used in the GPT architecture's downstream experiments.
Tasks
MNLI
Hyper
HELPNESS
CHEM
ACL-ARC
Learning Rate
le 5
2e 5
2
2e re-5 5
2e -
Batch Size
32
256
256
256
256
Weight Decay
0.1
0.1
0.1
0.1
0.1
Max Epochs
10
10
10
10
Learning Rate Decay
Linear
Linear
Linear
Linear
Linear
Warmup Ratio
0.06
0.06
0.06
0.06
0.06
D.2 DETAILED SETTINGS FOR BASELINE MODELS AND LOIRE VARIATIONS
We conduct the validation to show the training acceleration both GPT-style (decoder-only) and BERT-style (encoder-decoden model structures. It should noticed that all the experimental results including baselines are executed by our work.
GPT-style baselines:
GPT-S takes parameters of 27.59 million and then pretrains each domain data D, to learn new knowledge while maintaining the parameters invariant without growth schedule.
GPTL is similar to GPT-S but shares the same scale of parameters which 104.78 million parameters and trains from scratch over the same 5-domain pre-training dataset.
final extended model,
GPT R the same as GPT L, but recovering the PLM using the subset previously conserved corpora.
15

---

Published as a conference paper at ICLR 2025
Token-KD is knowledge distillation method implemented according (Cappellazzo et al. 2023) and continually train it while preventing forgetting from the previous model using the data from prior stages.
ER (Chaudhry et al. 2019) alleviates forgetting by recovering the PLM on samples from the previ- training data Dz-1 after training the PLM on the domain data D,. We set the sample ratio of D2 and D to 9:1, which is consistent with our approach.
ELLE (Qin et al. 2022) is derived from bertBERT, and flexibly expands existing PLM's width and depth to improve the efficiency of knowledge acquisition using a function-preservinn method, while proposing domain prompt to stimulate the needed knowledge for downstream tasks. Differ- ent from ELLE, LOIRE employs a different function preserving method, further adopting iterative distillation consolidate knowledge.
LOIRE-GPTI is trained from scratch using our suggested model operators and schedules on five domains of pre-training data.
LOIRE-GPT2: We initialed and pre-trained GPT-based small PLM with 27.59M parameters from scratch using around 24GB of data sampled from Redpajamat Weber al. (025). The sam- ple weights for different domains follow LLAMA2(Touvro) et al. 2023b)'s data mixing strategy. LOIRE-GPT2's (1) is initiated using the above-trained PLM.
BERT-style baselines:
LiGO Wang et al. 2023 is an efficient data-driven method to map the weights between small and large models, which improves the training dynamics by learning initialize. We re-produce this work leveraging the LIGO open-source codo to first train from scratch using the Redpajama dataset.
LOIRE-Bert shares the same training strategy LOIRE-GPT2, just with a different architecture. LOIRE-bert is growth trained using the English Wikipedia corpus, to keep aligned with the LIGO experiment setting.
D.3 METRICS
We utilize two metrics to evaluate how PLMs perform on the learned domains for the pre-training period, following (Chaudhry al.] 2019). Average Perplexity (AP) used to measure the av- erage perplexity of the current checkpoint M(0 the validation set for each seen domain data D < J). Average Increased Perplexity (AP+) measures the influence of current data Domain, on previous learned knowledge. Lower AP represents the model's ability learn from emerging data. Lower AP+ represents the model's 's capacity to maintain old knowledge,i.c.no forget
For an intermediate PLM model (1) during the growth stages, when learning the j th domain, we measure the (s perplexity PLL()), on the validation set of each seen Domainj. Let PPLj. be the perplexity on the j_th domain when the PLM finishes training on the juth domains, the above metrics are calculated as follows:
AP = =P=EXP(
log POPPLLy PPL().))
/=1
(16)
J-1
1
AP+
-PLLys-PPM -
1=1
D.4 MODEL GROWTH SETTINGS
All of the models in our studies use vannila decode-only Transformer architectures. During model growth, we adhere to few simple constraints contained in the existing LLM structure, as detailed in their published technical report, such as llama Touvron et al. 2023a). qwen (Yang et al. 2024).
baichuan (Yang al. 2023 and mistral (Jiang et al. 2023 The constraints include: 1) The hidden
https//viaaggruupggihuuiioo/
16

---

Published as a conference paper at ICLR 2025
dimension size is multiple of 128. 2) The hidden dimension is either 8/3 or 4 times the fin dimen- sion. 3) The number of attention heads should divisible by the hidden dimension; nevertheless, this has effect on the model's size. With these constraints, set the initial beginning point model with the structure (384, 1024,6 6), and the final model to (768, 2048, 12, 12) in the main experiments.
E SUPPLEMENTARY EXPERIMENTS
E. VISUALIZATION OF THE ATTENTION PATTERNS DURING THE MODEL GROWTH
We visualize the evolving attention patterns of stream of interminate PLMs that are trained during the multi-stage model growth procedure. take intermediate checkpoints after completing a phase domain data into these checkpoints to
of model growth in each domain. Next, input the deduce the respective attention patterns. F shows that a descendant PLM's attention patterns closely resemble those of its "ancestors," despite the model's growth and continuous training data. The picture's s similarity demonstrates that our method successfully allows interminate PLMs inherit and preserve the knowledge of their "ancestors."
LayerD Heado
LayarT
Layer2 Head!
Leverr3
Layaná Head?
Laverd Headd
M
Leyerd Head0
Leyert Head]
Layer2 Head!
Layer3 Head5
Layard Head?
LayarS Hisac4
M
LayerO Heado
Layerf Head3
Layer2 Heedf
Layer3 Head2
Layer5
M
Layarö HeadO
Head3
Layer2 Head?
Layer3 Heads
Layord Head2
Layerd Heads
M
Layort Heado
Layor3 Head?
Layer5 Head1
Leyer7 Heads
Layert Head?
Layerff Headd
M
Figure 5: The visualization of the attention patterns of different attention heads in M2 Ms after finishing training on the corresponding corpus.
E.2 TRAINING EFFICIENCY OF LARGER PARAMETERS
We conduct experiments on the efficiency of LOIRE-1.1B mentioned in Table by measuring its FLOPs and training time, and compare with GPT-I.IB and ELLE-1.1B. As shown in the Table 9 with the increase of model parameters in lifelong stages, LOIRE-1. reduces FLOPs by 36.9% compared to GPT-1.1B specifically. Additionally, as illustrated in Table B LOIRE- does not hurt the model's performance. Therefore, with the increase of model parameters in lifelong stages, the proposed method demonstrates superiority in saving computational resources and greatly en- hancing efficiency when scaling up varying sizes.
17

---

Published as a conference paper at ICLR 2025
Table For the I.IB GPT style models, we compare the FLOPs and train wall time of LOIRE-I.IB to GPT-1.1B and ELLE-1.1B on the 5-domain dataset's pre-training.
M1
M2
M1
M4
M1
Avg.
Metrics
FLOPs(e18//wal
time(h)
GPT-1.IB
(20486114,12222
(2044.6144,52224)
(004.6114,82.24)
(2048.114.12.24)
(2448.6144322224
34.07/23.66
34.07/23.66 34.07/23.66
34.07/23.66
34.07/23.66
34.07/23.66
ELLE-1.1B
(122,,072_.6.122)
(1230,38:0220.15)
(1538.6608.24188)
(1792.5376,26.21)
(2088.6164322299)
21.56/14.97
8.68/6.03 13.66/9.48
20.03/13.91
27.93/19.39
37.47/26.03
LOIRE-I.IB
(024.3072..001222
(2048.3072.06.12)
(2048,6144.16.12)
(20886114,32,12)
(2048,0164.22.29)
21.48/14.92
8.68/6.03
20.18/14.02
20.53/14.26
20.53/14.26
37.48/26.03
E.3
INDIVIDUAL PPL OF LOIRE-GPTI FOR EACH DOMAIN
Table 10
which lists the individual PPL of LOIRE-GPTI for each domain the model grows, better provides clearer insights into any knowledge degradation specific to earlier domains, and illustrates the effectiveness of our proposed lifelong method in terms of knowledge preservation.
Table 10: The individual PPL of LOIRE-GPTI for each domain the model grows.
WB
NEWS
REV
BIO
CS
M1
38.69
M₂
33.37
30.16
M,
32.03
31.45
24.67
M4
33.01
30.55
27.13
13.45
M,
28.71
27.60
24.27
11.03
12.28
F ABLATION STUDY
The models that appeared in this section are all trained from scratch, employing GPT structure.
F.1 LAYER OPERATOR
We propose a novel approach that increases the layer dimension by introducing a plug-in Trans- former layer growth operator. This operator designed using residual connections and is specif- ically optimized to maintain function-preeerving quality. In order to assess its efficacy, instead of placing the duplicated layer between original layer and the subsequent one, we replace it with layer-stack method, which involves immediately stacking the duplicated layers on top of the existing layers, as depicted in Figure O Other components remain the same.
added Transformer layers
existing
Transformer
layers
Figure 6: Layer-stack: an alternative way to replicate the existing layers.
As shown in Table I1 the AP and AP+ of our layer operator have significantly decreased, while the results of our downstream tasks are also significantly better than those of the layer-stack method. These results clearly demonstrate the ability and novelty of our proposed plug-in Transformer layer depth growth operator to preserve old knowledge.
18

---

Published as a conference paper at ICLR 2025
Table 11: The pretraining performance before each expansion and the downstream performance after 5 domains' training of Layer-stack, compared to our method
Schedule
AP
AP+
Pre-training
KAIRE:CIPT
Lager-inck
38,09
18.00
31.72
31.72
-3.2
-332
29.18
18
2278
-288
24.83
24.53
-0.94
-034
62.64
19.19
157.39
-.34
s0c.
occ.
a0c.
Downstream
Taas
MNLI
ONLI
Hyper
Ag nems
HELP
IMDB
CHEM
RCT
ACL-ARC
SCIERC
Layersiick LIINECPP
82.4
1920
12.87
55.20
82.31
81.18
N7.36
15.78
84.79
93.57
82.08
F.2 EFFECT OF RECOVERING WARMUP VIA ITERATIVE DISTILLATION
In order to investigate the efficiency of the iterative distillation component, we design two compara- tive models. 1) Memory: takes out the knowledge distillation part of the model architecture directly and uses the subset of corpora D to get back the old knowledge. 2) Single KD: In contrast to the iterative distillation used in LOIRE, single distillation is performed using only one previous model, while the remaining components are consistent with LOIRE
Table 12: AP and AP+ of Single KD, Memory and LOIRE with the same train wall time, we evaluate the performance after completing multi- stage training on each domain.
Schedule
Mi
M2
Ma
M4
Mp
AP
AP+
AP
AP+
AP
AP+
AP
AP+
AP+
Memory
38.69
33.83
-1.37
33.46
3.78
29.95
11.34
22.85
5.35
Single
38.69
32.81
-2.67
32.50
3.02
29.41
11.07
22.19
4.86
LOIRE-GPTI
38.69
31.72
-5.32
29.18
-2.68
24.63
-0.94
19.19
-3.84
As listed in Table 12 the performance of LOIRE surpasses Single KD and Memory. These results indicate that without the interactive distillation-based recovery warmup period, LOIRE's ability to retain old knowledge diminishes. Furthermore, the innovative iterative distillation method has the superior ability to incorporate previous knowledge compared to a single distillation procedure.
G FUNCTION PRESERVING PROOFS
The following proofs are sourced from (Gesmundo & Maile 2023). Our work differs from (Ges-) mundo & Maile 2023) that we proposes an extra-optimized layer growth operator, which prove in Section In addition, we design a growth schedule and perform experiments to verify the function preservation empirically, which (Gesmundo Maile 2023) only has theories.
G.1
MHA GROWTHI OPERATOR
The proof for the function-preserving transformation of head addition is follows:
Hhead,
H Hheadag) exd
(WO)' (a2 xxxxxxx
sxd
sx(a2xa) Xd)
wo
=[heead ...
Hheaday>
..ng Hheadoy exd
] x
xd)x))
N ((o2-a1)xd)xh
exd
axd
(17)
ax xd)
=x(((==01)xd)
[[headd mny Hhoaday J
1.
N
sxd
sxd
(ar xaix sxh
=[head
H Hheaday
m (as (d)xd)xh
exd
axd
sx(a; xd)
19

---

Published as a conference paper at ICLR 2025
Based the settings of the above parameters, have:
[hedd sxd
Hhedday exd
x (WO)'
[[heady "" Hheada, WO
(a2 xxxxdxxh
axd
exd
(ai (dixd)xh
(18)
xd)
(x(a1xd)
G.2
FFN GROWTH OPERATOR
The proof for the function-preservvi transformation of FFN expansion is follows:
HH MHA
x (Wh)) + +(6))) x (Wk) +
6/2 faxh 8xh
sxh
hxf2 5x
Wi
GELU
[hh
RW 11)
] +
6/
R
fixh N (12-
+ b/2 sxh
hxfi
sx ex(f2-fu)
axh
(19)
W fi xh N
HMHA W +
HMHA RWi, + R
GELU
)x
+ b/2
8xf1
axif2-hi)
(fa-fi)xh
sxh
HHHH X Wh 8/1
Wh
+ 8/2 HFPN
=
axh
bxfi
sxfi
81xh
sxh
Based on the initialization of the above parameters, we have:
MHA X (Wh) +( ))) (W'z)' + b/2
1x
8x
f2
(20)
GE X Wh + b ) x W
+ 6/2 = FFFN
sxh
Axci sxh
fixh
sxh
G.3
HIDDEN DIMENSION GROWTH OPERATOR
The proof for the dunction-preserviig transformation of hidden dimension expansion for the MHA module is as follows:
WK/Q/V hi xd
H' x (WKjQ/V)
H 18h1
NH ex(h
x R
H
X
R/QNV
sxba
haxd
(bg-t1))xd
sxh1
hixd
[head > .... Hheadel x
WO (axd)xhi
N (oaa)x(
axd
exd
(21)
sx(axd)
Hheadd .... Hhead,]
wo (axdixh,
N
ex()
sx(axa)
HHMHA
N
exhi
sx(hg-hi)
20

---

Published as a conference paper at ICLR 2025
The proof for the dunction-preservvi transformation for the FFN module is as follows:
GELU(
(Wl))' hax,
+ b4 exs
x (W42) fxh2
+ +(b2) 6xb2
8Xha
W
-GELU(
IMMAA 88h1
N ex(hg-hi)
hyxf R
+ bl xxf
x (W'2)' fxh2
+ (bb2)
sxh2
(ha-h1)xf]
HHELU Wh + b/i
W
bb sxcs
8Xh1
hi hixs sxf
fxhy fx(b2-b1)]
ax (x2--bb1)
(22)
SX
GEAA Wh + bl1)
x W
+ 8 sxhy
exhi
bixf exf
fxhi
xx 2x h1)
axhi
HFPN
N 8x A1)
exh,
In the Layer normalization layer, when the hidden dimension expands, we have:
(HNDRMMA -Layer norm( H
+ (HMHA)))
exh2
exh2
Layer norm(
H
N sx(k2-ka)
HMHA exhs
ax(ha-hi) b
N
(23)
Layer norm(
H
HMHHA) sxhn
sxhn
sx (h2--h
HNNRMMMA
N
sx(h2-h1))
Based on the settings of the above parameters, we have:
(H (HMMA/FFN)]
MHA/FFN N -
sxh2
H HNNRMMHAAFEN
MAAMME
(24)
sxh
xx (bg h1)
8Xha
The normalization of FFN layer is in similar way. Thus, LOIRE is strictly function-preserving with LN layers.
21


Reviews:
[{"review_id": "CWUbJeIO5X", "reviewer": "Reviewer_NUWX", "summary": "The paper introduces LOIRE, a lifelong learning framework for pre-trained language models that enables efficient adaptation to incremental data without full retraining. LOIRE tackles catastrophic forgetting and model growth challenges using a plug-in layer growth operator with residual connections for function preservation, a multi-dimensional growth schedule for optimal model expansion, and an iterative distillation strategy to retain prior knowledge. Tested on multiple benchmarks, LOIRE shows strong performance in reducing computational costs and maintaining task accuracy across evolving domains.", "questions": "- Have you experimented with different initialization methods for the extended parts of the model, such as the Hidden Dimension, FFN, or MHA, beyond the current method you’re using? If so, what impact did these alternatives have on model performance, function preservation, and adaptation to new data?\n- Did you calculate perplexity (PPL) separately for each previous domain as the model grows, rather than averaging across them? Measuring PPL individually for each domain could provide clearer insights into any knowledge degradation specific to earlier domains.\n- Did you measure LOIRE’s efficiency—such as FLOPs, training time, or memory usage—compared specifically to continual learning baselines like ELLE? Since efficiency is a key claim, it would be helpful to understand how LOIRE performs against ELLE and other lifelong learning frameworks on these metrics.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Rigorous Experimental Validation: The paper describes extensive experiments conducted across various domains using multiple datasets. It also includes ablation studies that examine the effects of growth operators and schedules.\n- Comprehensive Methodology: LOIRE uses a multi-dimensional growth schedule (covering dimensions like hidden size, FFN, MHA, and layers) combined with iterative distillation. This methodology is applied to manage catastrophic forgetting and model adaptation across evolving data.", "weaknesses": "- Limited Scalability Testing: The experiments are limited to relatively small models (up to 114M parameters), which may not fully demonstrate LOIRE’s scalability to larger PLMs, commonly used in industry. Further exploration with larger models could strengthen confidence in its applicability for high-parameter models.\n- Although LOIRE’s function preservation claims are theoretically sound, empirical validation shows minor deviations. Further exploration into these deviations might strengthen the reliability of the proposed growth operators, especially in dynamic adaptation scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t6MHclEnWI", "reviewer": "Reviewer_Z3XF", "summary": "This paper introduces LOIRE, a new framework, introduces comprehensive growth operators and a strategic schedule to expand PLMs effectively while preserving function. It also uses iterative distillation, allowing the model to alternate between teacher and student roles to reduce forgetting. LOIRE demonstrates a 29.22% reduction in computational costs with maintained or improved performance.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method seems novel and sound, and the extensive experiments and ablation studies demonstrate the efficacy of the method from multiple dimensions.", "weaknesses": "I don't see any major weakness of this paper, but that may be due to my minimal knowledge of this topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFFfI4c32V", "reviewer": "Reviewer_x51u", "summary": "This paper proposes a framework called LOIRE to address the challenges of lifelong learning for pretrained language models (PLMs). The motivation comes from the need to improve the adaptability of PLMs to new, emerging data without requiring complete retraining. In real-world applications, data often evolves over time, with new domains and tasks emerge that were not part of the original pretraining data.  PLMs generally struggle with this scenario because they are typically trained once on a large dataset and then fine-tuned for specific tasks and this leads to two major problems: 1) Catastrophic Forgetting where the models are fine-tuned on new data, they tend to forget previously learned information and 2) Computational Inefficiency where retraining LLMs from scratch every time when new data emerges is computationally expensive and impractical. LOIRE addresses these challenges in lifelong learning by introducing growth operators, schedules, and distillation strategies.", "questions": "As I mentioned in weaknesses, I have concerns about the scalability of the approach. Can you please explain how scalable the proposed method? \n\nHow well LOIRE handles significant shifts in data distribution? -- In real-world applications, new data often comes from domains that are very different from what was seen during pretraining or earlier stages of fine-tuning. It is unclear how well LOIRE would perform in such scenarios where there are drastic shifts in data distribution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "LOIRE proposes several approaches to overcome the limitations given in the summary. These approaches are the strength of this work IMHO. \n\nFirst one is layer growth operator that replicates selected layers and inserts them between existing layers with residual connections. The residual connections allow the newly added layers to be skipped during initial training, ensuring that the model's function is preserved. By ensuring that new layers do not interfere with the model's initial behavior, LOIRE enables smoother transitions when expanding model capacity. This is particularly important in lifelong learning scenarios where models must adapt to new data without forgetting previously acquired knowledge.\n\nSecond one is multi-dimensional growth operators that expands model capacity across multiple dimensions, such as hidden states, feed-forward networks, multi-head attention, and layers. By considering multiple dimensions, LOIRE can better adapt its expansion strategy to the specific needs of different tasks or datasets.\n\nThe main proposal is the growth schedule that determines when and where to expand the model structure across multiple stages. The schedule is designed to optimize the sequence of growth operations, ensuring that the model grows efficiently while minimizing computational costs. The results shows the significant computational savings -- an average reduction of 29.22% in expenses while maintaining or improving task performance compared to baseline methods.\n\nAnd finally the iterative distillation warmup to mitigate catastrophic forgetting during model growth where intermediate models generated during growth stages switch between being students and teachers. The iterative distillation warmup strategy helps retain knowledge from previous stages while adapting to new data, ensuring that the model remains effective across both old and new tasks. This technique enhances LOIRE's ability to handle incremental data without sacrificing performance on previously learned tasks.", "weaknesses": "- Although these approaches are individually strong, all these process is pretty complex and i have doubts how practical is the proposed method overall in real world applications. Implementing multi-dimensional growth operators and optimizing growth schedules requires careful tuning and may be challenging for practitioners who are not familiar with these techniques. The added complexity could limit the accessibility of LOIRE for users who need simpler or more straightforward solutions for lifelong learning. It would greatly help if the authors can address these concerns. \n\n- My main concern is the scalability of the proposed method. While LOIRE improves computational efficiency by reducing expenses by an average of 29.22%, scaling up this approach to very large models or datasets might still pose challenges. Lifelong learning frameworks need to be scalable to handle ever-growing datasets and increasingly large models. LOIRE's reliance on iterative distillation and multi-stage growth could still become computationally expensive as models grow larger. Further research may be needed to ensure that LOIRE can scale effectively without resulting in excessive costs. If the authors can provide computational complexity analyses for larger models, or conduct experiments with models of varying sizes to show how performance and efficiency scale, that would help to address this concern.\n\n- I also have concerns about the increasing model parameters. Table-1 shows that model parameters increases significantly for each task --> M1:27.59M, M2:62.25M, M3: 71.69M, M4: 71:69M, M5: 104.78M. In reality we may observe hundreds or thousands of new tasks and if we want our model to perform well on all these how big the model will end up? That seems like a bottleneck for scalability of the approach. Can you explain what is the behaviour of parameter growth?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d9jMtXLJo9", "reviewer": "Reviewer_MJ86", "summary": "This paper presents a method for lifelong learning of foundation models that relies on a growth strategy to allow updating of multiple dimensions of a Transformer architecture (multi-head attention, FFN, layer and hidden dimensions). Growth operators are proposed for each of these dimensions. A growth schedule, which is controlled by a hyper-parameter, is also proposed based on previous findings that \"growing the layers and heads in later stages and having a larger hidden dimension in earlier stages can lead to better model performance\". Experiments are run over 5 benchmark datasets, on both GPT and BERT-like architectures with increasing number of parameters. Experimental results include analyses on computation cost.", "questions": "- It was not entirely clear to me why there are GPT-like baselines but not BERT-like baselines to be compared against the proposed method (specifically in Table 1). Please clarify.\n\n- In the experiments, it was not entirely clear to me how are datasets divided into n number of tasks. From Table 1, it seems that each \"x\" is a growth operator, so how is this related with grouping datasets into tasks? Or are all datasets used all-together from the beginning?\n\n- Did you test all possible schedules beyond the growth schedule presented in Table 1? What are the results like?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper is original in the sense that it proposes a new growth strategy for foundation models which consists of growing multiple dimensions of these Transformer-based models. It presents \n\nQuality: The quality of the paper is good. There is explanation of existing concepts of new concepts, using tools such as text, diagrams and equations. There are no major points where quality needs to be \n\nClarity: The paper is written clearly, the concepts are presented in a reasonable flow. Experiments are also mostly clear.\n\nSignificance: There seems to be some significance of the paper from the experimental results presented against other architectures such as GPT.", "weaknesses": "- Significance of experiments: According to Table 1, each of the M_x steps is related to applying one of the growth operators proposed in the paper. However, the common setting in lifelong learning is grouping data(sets) into \"tasks\", which are learned sequentially. How is this lifelong learning idea applied in this paper?\n\n- Significance of experiments: Similar to the previous point, in lifelong/continual learning typically measuring \"forgetting\" gives useful information about the \"interference\" experienced by the system because of learning sequentially. I do not see this reflected in this lifelong learning paper at all. \n\n- Significance of experiments: There is no further justification as to why the particular growth schedule presented in Table 1 was selected. Are there other possible schedules? What would the results be under these schedules?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)