esa-sceva/satcom-chunk-collection huggingface dataset:
# Intelligent End-Edge Computation Offloading Based on Lyapunov-Guided Deep Reinforcement Learning
###### Abstract  
To address the end-edge computation offloading challenge in the multi-terminal and multi-server environment, this paper proposes an intelligent computation offloading algorithm based on Lyapunov optimization and deep reinforcement learning. We formulate a network computation rate maximization problem while balancing constraints including offloading time, CPU frequency, energy consumption, transmission power, and data queue stability. Due to the fact that the problem is mixed integer nonlinear programming, we transform it into a deterministic problem based on Lyapunov optimization theory, and then model it as a Markov decision process. Then, we employ deep reinforcement learning algorithm, i.e., asynchronous advantage actor-critic (A3C), and propose Lyapunov-guided A3C algorithm named LyA3C to approximate the optimal computation offloading policy. Experiments show that the LyA3C algorithm can converge stably and effectively improve the long-term network computation rate by 2.8% and 5.7% in comparison to the A2C-based and TD3-based algorithms.  
Keywords:edge computing; computation offloading; Lyapunov optimization; deep reinforcement learning +
Footnote †: journal: Applied sciences
## 1 Introduction
However, activated carbon, which is the most largely used adsorbent for industrial wastewater treatment, has a quite high price for its implementation at the majority of municipal WWTPs. In this case, and as above highlighted, the utilization of microalgae as biosorbent in wastewater treatment could be an environmentally friendly and economically sustainable option. Sorption by microalgae has been pointed out as a key process in the removal of antibiotics in high-rate algal ponds (HRAPs) [16; 17]. On the other hand, when studying the removal of 28 different ECs in HRAPs, [PERSON] et al. [18] found that sorption was relevant for the removal of hydrophobic compounds, while other ECs, such as paracetamol, were mostly removed by biodegradation. The removal of pharmaceuticals, including paracetamol, by the microalgae _Chlorella sorokiniana_ was studied by [PERSON] et al. [19] in batch experiments, who observed that sorption to algal biomass accounted for less than 20% of the removal. However, to the best of our knowledge the biosorption of pharmaceuticals, as the passive uptake of such pollutants by non-living microalgal biomass, has not been assessed in the literature. Therefore, the aim of this work was to study the potential of a blue-green microalgae, _Synechocystis_ sp., to be used as a biosorbent for the removal of pharmaceuticals from contaminated water. Paracetamol has been selected as target EC in this study, since it belongs to the group of anti-inflammatory analgesics, and is one of the most frequently found pharmaceuticals in wastewater.
## 4 Proposed Algorithm  
The overall algorithm contains three main steps illustrated as follows.  
Figure 3: The layer-divided model of the 3-D network.  
Figure 2: The FFT domain signal of temperature data.  
Firstly, we use PSO algorithm to choose \(M\) CHs for each segment and divide the nodes in each segment into \(M\) clusters. The second step is to select one routing path for each cluster using PSO algorithm, hence we obtain \(M\) routs to compose an integral routing matrix \(\mathbf{\Phi}\in\mathbb{R}^{M\times N}\). Figure 4 presents the flowchart of the whole procedure. In the flowchart, \(E\) represents the remaining energy of the whole network and \(f_{1}\), \(f_{2}\) are the fitness functions of clustering and routing steps respectively. \(f_{1}\) considers energy consumption and uniformity of cluster head distribution while \(f_{2}\) focuses on energy consumption state and BCRB.  
The detailed description about how the CHs and routing paths, i.e., the measurement matrix are chosen will be presented in the next subsections. After we get the measurement matrix, the sink node receives a \(M\) dimensional measurement \(\mathbf{y}\) and its measurement matrix \(\mathbf{\Phi}\). Finally, at the sink node, we use BCS algorithm to reconstruct the sparse signal \(\mathbf{x}\) and then transform it to the original temperature signal \(\mathbf{z}\). After obtaining signals for all the segments one by one, the base station merges them together to recover the complete original signal.  
To design the final object, there are three important factors to be taken into consideration: the signal reconstruction accuracy, the remaining energy of the network, and the balance state of the network.  
Usually, mean squared error (MSE) is used to measure the reconstruction quality of the signals. If \(\hat{\mathbf{x}}(\mathbf{y})\) denotes the recovered signal based on noisy measurements \(\mathbf{y}\) represented by Equation (2). The MSE in estimation of the vector \(\mathbf{x}\) is given by  
\[\text{MSE}=E\{\|\mathbf{x}-\hat{\mathbf{x}}\|^{2}\}=Tr\{E[\left(\mathbf{x}- \hat{\mathbf{x}}(\mathbf{y})\right)\left(\mathbf{x}-\hat{\mathbf{x}}(\mathbf{ y})\right)^{T}]\} \tag{11}\]  
However, the calculation of MSE requires the actual estimator which is impossible for us to get before choosing the measurement matrix. To solve this problem, we introduce Bayesian Cramer-Rao Bound (BCRB) to measure the lower bound of MSE; that is, \(\text{MSE}\geq\text{BCRB}\). Hence, adding BCRB into the fitness function is beneficial for us to find the solution with lower MSE, since minimizing BCRB is equal to obtaining minimum acceptable MSE. Subsequently, we will derive the formula of BCRB for our problem.
## 2 Results and Discussion
### Optical Properties of Printed SANP-S and SANP-T Films  
Since both dye loading and light scattering affect the LHE, these properties were quantified for SANP-S and SANP-T films, including dye loading after sensitisation with either D149 or N719 [31]. Figure 4a,b (and Table 1) show the results of the dye desorption experiments, with SANP-T films hosting slightly more dye compared to SANP-S films of the same thickness. This is roughly in line with the differences in surface area reported above. Meanwhile, undyed SANP-S films (10 \(\upmu\)m) showed a considerably higher diffuse reflection (35-55%) (with lower direct transmittance) than undyed SANP-T (5-10%), as shown in Figure 4c and (Figure S2, Supplementary Materials) and in the visual images of photoanode films (Figure 5).  
\begin{table}
\begin{tabular}{c c c} \hline
**Photoanode** & **D149 Loading (nmol-cm\({}^{-2}\))** & **D149 Loading (umol-cm\({}^{-3}\))** \\ \hline SANP-S \(\mid\)D149 & 72 \(\pm\) 2 & 111 \(\pm\) 2 \\ SANP-T \(\mid\)D149 & 81 \(\pm\) 2 & 124 \(\pm\) 3 \\ SANP-S \(\mid\)N719 & 68 \(\pm\) 2 & 104 \(\pm\) 3 \\ SANP-T \(\mid\)/N719 & 74 \(\pm\) 2 & 113 \(\pm\) 3 \\ \hline \end{tabular}
\end{table}
Table 1: The amount of D149 dye loading on SANP-S and SANP-T films (thickness = 5.0 \(\pm\) 0.3 \(\upmu\)m; area = 1 cm\({}^{2}\)) calculated using the desorption method [31].  
Figure 4: (**a**) and (**b**) Absorbance spectra of dye (D149 or N719) desorbed from either SANP-S or SANP-T films (5.0 \(\pm\) 0.3 \(\upmu\)m thick, area = 1 cm\({}^{2}\)); (**c**) Diffuse reflections of SANP-S and SANP-T films (10.0 \(\pm\) 0.5 \(\upmu\)m thick).  
Figure 5: Visual images taken through 10 \(\upmu\)m-thick photoanode films of (**a**–**c**) SANP-S and (**d**–**f**) SANP-T, with (**a**,**d**) no sensitier or (**b**,**e**) sensitised with N719 or (**c**,**f**) D149.
## 5 Numerical Experiments
### Comparison of FG and AG Algorithms  
The numerical experiments involved production lines with ten and twenty machines (i.e., \(K=10\) and \(K=20\)). For 10-machine lines, the total buffer capacity \(N\) is set to 500 and 1000, respectively, and for 20-machine lines, \(N\) is set to 1000 and 2000, respectively. Hence, there are four sets of BAPs. Each set is denoted as \((K,N)\) and the four sets are (10, 500), (10, 1000), (20, 1000), and (20, 2000). For each set, twenty BAPs are studied and the total number of experiments is 80. Machine parameters are randomly generated from the following sets for each experiment:  
\[\mu_{i}\in\{1,1.1,1.2,1.3,1.4,1.5\},i=1,\ldots,K, \tag{8}\]  
\[MTTR_{i}\in\{100,\ldots,300\},i=1,\ldots,K, \tag{9}\]  
\[e_{i}=\frac{MTTF_{i}}{MTTF_{i}+MTTR_{i}}\in\{0.85,\ldots,0.95\},i=1,\ldots,K, \tag{10}\]  
where \(e_{i}\) is the efficiency of machine \(M_{i}\). The failure times are randomly set to Weibull or gamma distribution, whereas the repair times are set to log-normal distribution. In the case of non-exponential distributions, the coefficients of variation (CVs) of failure and repair times are selected randomly from the set \([0.1,\ldots,2]\).  
Table 1 presents the experimental results. For each BAP, the percent error of each algorithm is calculated as follows:  
\[\epsilon=\left|\frac{f(\mathbf{b})-f(\mathbf{b^{*}})}{f(\mathbf{b^{*}})}\right|\times 100\%, \tag{11}\]  
where \(\mathbf{b}\) is the solution obtained by the algorithm under consideration and \(\mathbf{b^{*}}\) is the best solution obtained by the two algorithms. For each problem set, the average percent error is listed in the third column of Table 2. As can be seen, the average error of the AG algorithm is within 0.05%, which is very small. Theoretically, the FG algorithm should always yield a better solution than the AG algorithm. However, when the throughput obtained by the two algorithms is very close, it is difficult to distinguish which algorithm is really better due to the stochastic errors in the simulation. Therefore, the FG algorithm also leads to small errors. Considering the fact that the machine reliability data are seldom within 5% of their actual values [41], the AG algorithm is accurate enough for practical problems. In terms of computation time, the AG algorithm is much more efficient than the FG algorithm, with the former taking about 1/10 to 1/20 the time of the latter. Note that the computation time of the AG algorithm increases only slightly when the total buffer capacity \(N\) is doubled in cases of both \(K=10\) and \(K=20\). This indicates that the AG algorithm can adapt well to the scale of BAPs.
## 6 System Evaluation
### Search Efficiency  
#### 6.4.1 System Setup  
To setup the system, the data owner first employs a SURF algorithm in order to extract a descriptor vector for search, each of which, in turn, requires the computation of a \(O(m)(p)\)-dimensional. Then, the data owner encrypts all descriptor vectors used for the search process, and each requires the computation of \(O(m)(p)\)-dimensional secure dot products. Following this step, the data owner proceeds to construct the index tree _TreeIndex_ with \(\sum_{r=0}^{logR(m)-1}R^{r}\)\(k\)-means\((\dfrac{m}{R^{r}},R)\) operations. We evaluate the building cost of _TreeIndex_ within the framework of lowering the dimension using the PCA method and the dataset size. Figure 4 shows that lowering dimension can lower the building cost of _TreeIndex_, since a majority of functions in the \(k\)-means algorithm are \(L_{2}\) distance computation, and, therefore, with linear cost to the vector's dimension. Regarding the size of the data set, it is clear that the expansion of dataset size will necessitate a higher computational cost for building a _TreeIndex_, (see Figure 5) because the owner requires undertaking more \(k\)-means clustering on an increased amount of vectors.  
Note that the initial setup for the system is a one-time function, which does not affect the effectiveness of the search process in real-time. In addition, enhancements to the implementation may be made through customizing it towards the procedure of tree construction through the employment of the parallel \(k\)-means clustering library [38], in which one billion 128-dimensional vectors might be processed within a 50 min period.
# Numerical Simulation-Based Study on the Arching Effect in Subsequent Backfill
## 3 Numerical Simulation
### Research and Analysis on the Influence of Backfill Elastic Modulus on Stress Distribution Patterns  
Figure 8 shows the evolution law of the stress at the vertical centerline of the filling body varying with the elastic modulus (100-1000 mpa). The research results show that when the elastic modulus increases from 100 MPa to 1000 MPa, the maximum vertical stress at the bottom of the filling body only slightly increases from 530 kPa to 540 kPa, and the horizontal stress only slightly increases from 140 kPa to 144 kPa. This stress variation amplitude is highly disproportionate to the increase in elastic modulus, fully indicating that the stress of the filling body has a distinct weak sensitivity characteristic to the change in elastic modulus.  
Combined with the previous research results, it can be concluded that the stress transfer mechanism of the filling body is mainly controlled by the interfacial friction force and cohesion force, rather than the stiffness characteristics of the filling body itself. This important discovery provides a new theoretical basis for the design of filling engineering: in practical engineering applications, simply increasing the elastic modulus of the filling body cannot effectively improve the arching effect or optimize the stress distribution. Therefore, the focus of engineering design should be placed on reasonably regulating the stiffness of the contact unit to ensure the integrity of the filling body structure. By scientifically matching the elastic modulus with the stiffness of the contact element, it is possible to ensure the safety of the project while achieving efficient utilization of resources and reaching a balance between economic benefits and project quality.  
Figure 7: Comparative analysis of the influence of cohesion on backfill stress distribution.
## 2 Study Area and Data  
### Study Area  
Zhengzhou City is located in the central north of Henan Province (Figure 1a), at 113 deg27 deg-113 deg51' E, 34 deg36'-35 deg00' N between the urban area north of the Yellow River, west of the Mount Song, the topographic trend from southwest to northeast gradually low, a step decline, an area of about 1010 km2, and the strata are mainly composed of Quaternary loose silt, silty clay, and gravel layers. Adverse rock and soil conditions such as collapsible loess are mainly distributed in the western part of Zhengzhou, as shown in Figure 1b; the climate conditions belong to a temperate continental monsoon climate, with an average annual precipitation of approximately 632 mm, and the precipitation time in the year is mainly concentrated in June-August [18]. The Jalu River, Dongfeng Canal, Jinshui River, Xionger River and Soxu River in the city undertake the task of precipitation and wastewater discharge. Further, 77% of the city's features is covered by buildings and 13% is cultivated land.  
Figure 1: (**a**) Location of the study area, (**b**) shows the distribution of the study soil types, (**c**) distribution of feature distribution type distribution.  
From 18:00 on 18 July to 0:00 on 21 July 2021, there was a rare continuous heavy rainfall weather process in Zhengzhou. The maximum hourly precipitation observed by Zhengzhou Station from 08:00 on 20 July to 08:00 on 21 July reached 201.9 mm/h (16:00-17:00), breaking through the historical extreme value on the Chinese Mainland. During the 720 extremely heavy rainstorm disaster in Zhengzhou (hereinafter referred to as the disaster), the city's cumulative average precipitation was 449 mm, but a single day of precipitation broke through after 1951. In Zhengzhou, since the establishment of the station, with 60 years of historical records, the city's drainage system has been seriously overloaded, the city flooding has been serious, and 380 people died due to the disaster, accounting for 95.5% of the province; the direct economic loss was CNY 40.9 billion, accounting for 34.1% of the province, and multiple urban roads and buildings were damaged to varying degrees. In this paper, the main urban areas of Zhengzhou City (Jinshui District, Huiji District, Zhongyuan District, Ecri District, Guancheng District) were selected for monitoring.
## 1 Actinobacteria and Their Primary Metabolism as Basis for Secondary Metabolism Precursor Supply
### Primary Metabolism in Actinobacteria: Nitrogen Metabolism
#### 1.2.2 Regulation of Nitrogen Metabolism in Actinobacteria  
The regulation of nitrogen metabolism takes place at the transcriptional and the post-translational levels. This regulation in Actinomycetes is distinct from that in Enterobacteriaceae.  
Several transcriptional regulatory components known from Enterobacteriaceae, such as GlnK (=Pi), GlnD, GlnE and Crp have counterparts in Actinomycetes, but their individual function and interplay are different. Regulators with functionality specific to Actinobacteria like GlnR, GlnRII and NnaR are only present in some actinomycetes like _Streptomyces_. The complex nitrogen metabolism of _S. coelicolor_ requires additional control by other transcriptional regulators: ArgR, AfsR, AfsQ1, DasR and PhoP [44; 45; 46] (Figure 1). Interestingly, it has been demonstrated that \(glnA2\), \(glnA3\) and \(glnA4\) in _S. coelicolor_ are regulated differently at the transcriptional level. A strong interaction between acetylated GlnR and the \(glnA2\) promoter region has been revealed. \(glnA3\) and \(glnA4\) promoter regions appeared to interact with specialized regulators EpuRII and EpuRI, respectively [39].  
At the post-translational level, GSI is controlled by GlnE via adenylation and dead-enylation [33] (Figure 3). Interestingly, GSI lacks post-translational regulation through adenylation [36]. Also, the GS-like enzymes GlnA2, GlnA3 and GlnA4 lack an adenylation motif and cannot be modulated post-translationally by adenylation like GSs.
## 2 Battery Modeling Methods
### Battery Model and Model-Based SOC Estimation  
The driving conditions of EVs are significantly different in different countries and regions [16], which means that the estimation methods should be able to accurately estimate SOC under diverse driving conditions [17]. Therefore, advanced SOC estimation methods with good robustness are needed for EVs. Model-based methods have frequently been used for the SOC estimation area due to their good accuracy and robustness. The structure of the model-based SOC estimation method is illustrated in Figure 1.  
Figure 1: Framework of the model-based State of Charge (SOC) estimation methods.  
We see from Figure 1 that the model-based estimation consists of two major parts: a battery model and an estimation algorithm [14; 15; 18; 19; 20; 21]. In the model-based estimation structure, a battery model is established to predict the terminal voltage, and the current, SOC, and temperature are the usual inputs. In order to clearly explain the process of model-based estimation, it is assumed that the battery model is exactly equivalent to the real battery and the measurement from sensors does not contain any noise. Therefore, if the real SOC is the input of the battery model, the predicted voltage of the battery model is expected to be the same as the measured terminal voltage. However, errors in SOC always exist in real applications. The deviation between the output of the battery model and the terminal voltage can be used to correct the SOC in each sampling time. Afterwards, the corrected SOC acts as the input of the battery model for the calculation of the terminal voltage in the next cycle. As discussed in [22], the SOC estimation error contains dynamic transient error and steady state error. The dynamic transient error can be small when using a large gain \(L\) in the estimation algorithm, while the errors related to the modeling accuracy are independent of \(L\)[22]. This means that the gain \(L\) cannot correct battery modeling errors. Therefore, an accurate battery model surely improves the accuracy of SOC estimation. The model-based SOC estimation methods are insensitive to the initial SOC due to the fact that it has a closed loop structure [10].  
After pointing out the main function of the battery model in the model-based estimation structure, different kind of battery modeling methods are detailed in this section. According to their modeling principles, modeling methods are mainly divided into four categories: empirical models, ECMs, electrochemical models, and data-driven models.
# Modeling and Profiling of Aggregated Industrial Network Traffic
## 3 Modelling Aggregated Traffic Break-Down
### Objective 2: State Modelling and Profiling
#### 3.2.2 State Profiling  
After identifying the states, the second objective is achieved by profiling distinguishable transmission states in a work-cycle. A set of labels was produced in the modeling process to identify the data points belonging to each state. The summary statistics of the identified states are presented in Table 1. Since the transition between the states is time-varying, each state is profiled by the summary statistics and the communication intensity parameters, i.e., transition volatility and transmission volatility. Each varying length state can be profiled as  
\[\{(t_{1:k},s_{i})|s_{i}=\{\mu_{i},\sigma_{i},M_{i},max_{i},uts_{i},tx_{i}\},i=1,2\}.\]  
Figure 4: The criterion values, y-axis, suggest a second order HMM for the dataset consisting of 5 work-cycles (**a**); the identified states by the second-order model are color-coded in (**b**); (**c**) shows distribution of the data in each state.  
At this point, the profiles can be used as the basis for identifying the states, given a sequence of observations in a work cycle. The steps taken to achieve the second objective are described in Algorithm 2.  
```
1:Work-cycles dataset, \(wcd\).
2:Profiled states.
3:Initialize model parameters:
4:Model order: \(model\_k,k\gets 1:10\),
5:Sample size: \(n\leftarrow|wcd|\),
6:Model evaluation criterion: \(model\_ec=[AIC,BIC,EDC]\).
7:for\(model\_k\&n\)do
8:Fit and predict probability: \(model\_k(wcd,n)\).
9:Calculate likelihood: \(L(wcd)\).
10:Calculate model accuracy score: \(model\_score_{k}(wcd)\).
11:for\(model\_ec\)do
12:Calculate evaluation criterion: \(model\_ec(n,k,model\_score_{k}(wcd))\).
13:Append to \(EC\_List\): \(EC\_List[k,AIC,BIC,EDC]\gets model\_ec()\).
14:endfor
15:endfor
16:Select \(model\_k\) where \(EC\_List[k,AIC_{init},BIC_{init},EDC_{init}]\).
17:\(model=HMM(k,wcd)\).
18:\(state_{1k}=model\_predict(wcd)\).
19:for\(state_{i}[items],i\gets 1:k\) & \(items=wcd[state_{i}]\)do.
20:Calculate \(n_{i},\sigma_{i},M_{i},max_{i},uts_{i},tx_{i}\).
21: Profile \(s_{i}=\{n_{i},\sigma_{i},M_{i},max_{i},uts_{i},tx_{i}\}\).
22:endfor
```  
**Algorithm 2** State modeling and profiling.  
To provide support for the effectiveness of the proposed approach, including the introduced parameters and deployed learning model, a set of experiments were carried out where the results are presented in the following section.