# Deep Lifelong Learning Optimization Algorithm in Dense Region Fusion
## 3 Materials and Methods
### Algorithm Principle in DLLO-DRF
#### 3.2.2 Dense Region Fusion
The calculation of \(X_{dense}\) is carried out using Equation (3). \[X_{dense}= \begin{cases}X_{i}&s^{2}<V\\ \arg\max_{X_{ij}\in X_{i}}|X_{ij}|&s^{2}\geq V,\end{cases}  \tag{3}\]  
where \(|X|\) represents the number of elements of \(X\), and \(V\) is the set threshold of variance. ```
Input:\(X\): Parameter set obtained from multiple models. \(j\): The number of regions that should be divided. Output: The subset of \(X\) with size \(j\): \(X_{1},X_{2},\ldots,X_{j}\). 1 Initialize the empty set: \(X_{1},X_{2},\ldots,X_{j}\);
2 Compute the size of \(X\): \(n\), and the maximum value and minimum value of \(X\) (\(x_{max},x_{min}\));
3 Compute the size of divided region according to Equation (2): size;
4 for\(i\gets 1\) to \(n\)do
5 for\(k\gets 1\) to \(j\)do
6 if\(x_{i}\geq(x_{min}+k\times size)\) and\(x_{i}<(x_{min}+(k+1)\times size)\)then
7 Put \(x_{i}\) into \(X_{k}\)
8 end if
9
10 end for
```  
**Algorithm 2**Region Division  
Finally, the mean value of the parameters in \(X_{dense}\) is used as the parameter at the \(i\)th position in the fusion model \(x_{i}^{fusion}\), and it can be calculated using Equation (4). \[x_{i}^{fusion}=\frac{1}{|X_{dense}|}\sum_{x\in X_{dense}}x, \tag{4}\]  
where \(|X|\) represents the number of elements in the set \(X\). We also offer an approach for directly identifying dense regions. This approach involves constructing a network model that employs short- and long-term neural networks to learn the boundaries of dense regions within a given parameter space, with the model being referred to as a dense region divider. The structure of the region divider is illustrated in Figure 3.  
The input for the network consists of the parameters of the \(n^{th}\) stage model, followed by a layer of LSTM units with input and output dimensions of \((n,100)\). Subsequently, another layer of LSTM units with input and output dimensions of \((100,100)\) is employed. Finally, a fully connected layer with input and output dimensions of \((100,2)\) is added. This connected layer's two outputs represent the upper and lower bounds of the learned dense region. By utilizing this region divider, dense regions can be obtained as an alternative to employing Algorithm 2 and Equation (3). Based on the principles and steps of the above algorithms, an algorithm for dense region fusion has been developed, which is presented in Algorithm 3. ```
Input:\(Model=\{model^{(1)},model^{(2)},\ldots,model^{(n)}\}\): Trained model set. Output:\(model^{fusion}\): Fusion model. 1 Map all model parameters in \(Model\) to one dimension according to Algorithm 1;
2 Obtain the parameter dimension, \(shape\), of the original model based on Algorithm 1;
3 Calculate the parameter length, \(length\), of \(Model\);
4 for\(i\gets 1\) to lengthdo
5 Initialize \(X_{i}\);
6 for\(k\gets 1\) to \(n\)do
7 Obtain the parameter of \(model^{(k)}\) in the \(i\)th position and put into \(X_{i}\);
8 end for
9 Obtain the \(j\) regional subset of \(X_{i}\)\(X_{i1}\), \(X_{i2},\ldots,X_{ij}\) according to Algorithm 2;
10 Obtain the dense region \(X_{dense}\) according to Equation (3);
11 Obtain the parameter of the fusion model in the \(i\)th position \(x_{i}^{fusion}\);
12 end for
13 Restore \(model^{fusion}\) to its original dimension according to \(shape\). ```  
**Algorithm 3**Dense Region Fusion