Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
Authors: ShaoQi Wang
Chunjie Yang
Siwei Lou
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural networks
network's structure design
minimum variance estimation
online learning
training stability
natural gradient
soft sensor
Primary area: deep_learning_architectures
TL;DR: A novel network structure that efficiently approximates  minimum variance estimation and computes natural gradient achieving stable convergence and superior performance in industrial setting.

Abstract: Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their  feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation in NN, wherein the truncated gradient approximates the natural gradient. Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
Siwei Lou t Zhejiang University, China, {sq_ cjyang999, swlou] edu.
Shaoqi Wang
Chunjie Yang
Abstract
Neural networks (NN) are extensively studied in cutting- soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accu- racy. Notably, industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires clearer un- derstanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.
Introduction
Deep learning methods have achieved recent success many regression areas such as natural language processing, protein structure prediction, and building energy consumption forecasting. However, for these methods to be useful in the industrial soft sensor field, which demands higher immediacy and stability, further research into model structure and the stability of the training process is necessary I 31. The safety and economic impact of factory impose stringent requirements on soft sensor models deployed online [ (5) For example, each mini-batch update must not cause significant performance fluctuations to ensure that downstream controllers and monitors do not execute erroneous actions; soft sensor models must be deployed online to avoid fluctuations due to changes in operating conditions and model switching [6]. Common network's 's training tricks not suitable for soft sensor contexts. For example, it's feasible use checkpoints for early stopping but to always use the latest updated checkpoint; there wouldn's 't be adaptive learning rate changes but rather constant learning rate maintained throughout. Experimental results demonstrate that such differences lead to a substantial decline in performance. These constraints necessitate the development of better-suited network architectures for regression task that ensure more stable optimization during training 6
MVE is the best unbiased estimator under the Mean Squared Error (MSH) criterion, essentially representing the performance ceiling for regression models g Unfortunately, directly applying MVE to NN is challenging due to the difficulty in obtaining the likelihood distribution p(y|æ) of inputs & and outputs [9]. Traditional research on MVE has focused on techniques like Kalman
filtering [10 []]. algorithms based on Ordinary Least Squares (OLS) [12 [ and other system identification research [14] 15 16] which operate under linear and convex conditions. These methods,
while effective within their scope, have limited expressive power
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Earadobbee
Traduable
AE
(A)
(D)
Figure Trackable parameters and Untrackable parameters. Solid green lines represent model parameters, and orange curves represent non con-arammttic operations. (a) Conventional deep NN framework. (b) Typical broad learning system framework through data enhancement.
Many studies have explored NN optimization from various perspectives such adaptive learning rates [18], momentum updates [99] and customized loss functions [20]. Compared to the first-order optimization methods, second-order optimization algorithms based Natural Gradient Descent (NGD) 21 (2) can achieve faster and more stable convergence (33 This is because NGD considers the manifold distribution of model parameters by computing the Fisher Information Matrix (FIM) during the gradient update process. However, calculating FIM introduces significant computational overhead, making NGD challenging implement in most machine learning models [24]. Much research is focused on adapting model structures for NGD [25] 26 27 and reducing its computational
costs 28 29 30 yet applying NGD to NN optimization remains an unsolved issue 31 32 33
Some studies have approached the regression task from the perspective of targeted modular design 36], emphasizing the construction of local neuron-level rules to assist models in learning
34
35
conducive features exemplified by SL.TTM [37). SIAE 38). MIF-Autoformer [39], and CBMP
40 Some endeavors have yielded results with solid theoretical underpinnings 41
42 such as
$4 43 VAF, VIOG These examples, integrate with other great work [27 45
46 jointly
demonstrate excellent integrations of NN with theoretical basis and also have stronger expressivity
compared to the identification algorithms [14] (7) However, even though these networks possess certain interpretabilityy is still unclear whether these prior biases beneficial for regression task 48 Furthermore, their mathematical foundations are not rooted in soft sensor tasks, which do not guarantee stable performance regression [5
AOPU differs from conventional studies by focusing on training optimization and the overall input- output relationships. Assuming there is robust feature extraction module (augmentation block), AOPU pays specific attention on better optimization and more stable convergence. AOPU innovatively introduces trackable and dual parameters, enhancing structure approximation of MVE. The dual parameters are injective representations of trackable parameters, mainly aiding in truncating the gradient backpropagation process. This truncation will be validated as an effective approximation to NGD. The augmentation module boosts AOPU's performance and also acts as an extension interface, making AOPU more versatile. Rank Ratio (RR) is introduced an interpretability index provide deep and comprehensive insights into the network dynamics. RR quantifies the ratio of linearly independent samples within batch, providing measure of the heterogeneity and diversity of information. By harnessing RR value, we can roughly foresee the performance in advance of the training. A high RR suggests the model output more closely approximates the MVE, and optimization is more in line with NGD, leading to superior performance. Conversely, a low RR implies that the precision of computation compromised, resulting in inferior performance.
2
AOPU:Methodology
2.1 Trackable vs. Untrackable
Parameters within NN that can be decoupled from input data and computed through an inner product are defined as trackable parameters. Couversely, parameters that cannot be decoupled are classified as untrackable parameters.
f(x) W x (W,z)
g(x) M(x) (M()),
(I)

---

0004
(a) gradient decline ignores the manifold (radical)
(b) natural gradient recognizes the manifold (conservative)
Figure 2: Comparison between NGD and GD. Direction matters more than step size (learning rate) in stable convergence.
where W represents parameter matrix and M() denotes operator dependent æ. According the definition, W is identified as a trackable parameter, whereas Mxx) is considered untrackable.
A significant proportion of parameters in NN are untrackable depicted in Fig. | This predominance attributable to the networks' reliance on stacking activation functions to bolster their nonlinear modeling capabilities. Proposition I indicates that any parameter influenced by activation function thus rendering it untrackable.
transitions to being
Proposition 1. There does not exist an transition operator T independent of such that for given parameter matrix W, and Vx1, T2, the following equations hold,
acti [ x1) = TWW)a1, acti( = =T(W)a2
Proof Appendix
2.2 Natural Gradient Gradient
Fig. 2 vividly presents the major difference between NGD and GD using a simple GPR in experiment. This GPR had only two parameters, the bias of the mean and the coefficient of the kernel matrix, both constant values. We sampled 100 instances from this GPR and updated these two parameters 100 times using these samples. It was observed that NG require higher learning rate, while conventional gradients only need a smaller one. The major difference between NG and conventional gradients lies their directions. Conventional gradients ignore the parameter manifold and treat every parameter equally. NG, by dividing the gradient by its second derivative, treat sensitive parameters cautiously (low gradient) and non-sensitive parameters boldly (high gradient). This adjustment results in different gradient directions and contributes to better convergence.
Nevertheless, the calculation of NG involves considering the inverse of the FIM, thereby introducing computational complexity cubic to the number of parameters, making it infeasible for neural networks. Existing research on NG focuses on conventional machine learning, e.g., considering more complex distributions (such as the product of multiple exponential family) for computing NG. Research in NN domain NG mostly centers on second-order optimizers (such AdamW), which are merely first-order approximations of second-order NG.
2.3 Network's structure
We wish to emphasize that within AOPU framework, the truncated gradient of the dual parameters serves as an approximation of the NG of the trackable parameters, while AOPU's structured output approximates the MVE. The fidelity of these approximations is measured by the RR, the closer RR to I, the more precise the approximation; conversely the closer RR is to O, the more precision loss occurs. Furthermore, it can be demonstrated that the output of AOPU fundamentally differs from traditional neural networks: instead of explicitly modeling a mapping from to y. implicitly models mapping from z to Ty. To ensure that y can be recovered from Ty. is imperative that RR equals 1. AOPU also guarantees the convergence of the dual parameters the input-outpus relationship can be characterized by specific system. The proof of above is intricate and comprehensive, and one may refer to Appendix A. A and [ for more detailed information. This section focuses the implementation of AOPU.
AOPU utilize data augmentation to replace stacked activation structures to enhance the nonlinear modeling capabilitie. In such designs, the choice of the data augmentation module forms a crucial
3

---

Trackuble Pirmmeter
Inference
Fenction
Gradient Update
Injection
Back Propugation
luput Dara
Output Datn
Dual
Objectivve Function
Purameter
AOPU's data flow schematic, The gradient is backpropagated but truncated at the dual parameter, and this gradient is then used to update the trackable parameter.
Figure 3:
model prior. For ease of implementation, AOPU adopts random weight matrix approach for its data augmentation module 52). Specifically, suppose the original feature dimension of the input data is d, and the defined model hidden dimension size is h. Let G be fixed Gaussian initialized random weight matrix, G' Rd.4 and for input data x E 12466 where b is the mini-bated size, the data augmentation process is as follows,
I concat(acti (GT x),
(3)
Subsequently, for the augmented ă, the output processed using the trackable parameter W (dd+A),, where o represents the output dimension and, for simplicity, we assume o indicating research into univariate output. The output function is then,
( = . W
The optimization of parameter W in AOPU differs from other networks. We introduce dual parameter D(W) to describe this process,
D(W) 7 W
(5)
and the loss computed using the following objective function,
b C
[(=" -
D(W))
(6)
AOPU innovatively introduces trackable and dual parameters. The dual parameters are injective representations of trackable parameters, mainly aiding in truncating the gradient backpropagation process. During the training process, gradient backpropagation is truncated at D(W), and the gradient of the dual parameter updates the original trackable parameter demonstrated in Fig. 3 thus completing the training of the model.
It is important to note that the training process involves the inversion of matrix F 2. which is not always invertible, thus introducing numerical computation issue. We employ the reciprocals of the positive singular values circumvent the solvability issues that arise when inverse does not exist. However, this approach introduces significant computational precision loss, which in turn prompts a thorough analysis of RR.
We define the metric RR to represent the ratio of the rank of z to its batch size. Clearly, RR a value between [0, 1). and as RR approaches 1, the process of approximating the inverse of a 5 through cigenvalue decomposition becomes more accurate (owing to the presence of more reciprocals of cigenvalues). When RR equals I, - 2 is an invertible matrix; conversely, the smaller the RR, the less stable the model's numerical computations and likely poorer performance.
AOPU utilizes parameters' trackability nature to accelearte the NG computation. The key to AOPU's capability to rapidly approximate the NG is its ability bypass the FIM computation and its inverse. The key to the capability to skip the FIM lies in our utilization of Eq. [O]] which separates the model parameters from the variables and data. this context, we can use gradient matrix of the expectation parameter with respect to natural parameter to replace the FIM (without actually performing this calculation), thereby substituting the original complicated NG computation with an equivalent conventional gradient computation, i.c., Vam F()) Vn (F( so

---

that (F V C. This allows us to use the automatic differentiation toolbox for rapid calculations. Otherwise we must explicitly compute the inverse of the network's FIM, which is very time-consuming and memory-intensive. For instance, a network with 20kB of 32-bit parameters, which equates to 5120 trainable parameters, requires inverting 5120-dimensional matrix with each training iteration. This requirement grows with model size and can easily lead to GPU memory shortages. More critically, such large matrix inversions often lead to significant numerical precision loss, severely impairing model performance.
3 Experiments and Analysis
In this section, we detail the experimental results of the AOPU model, analyze the impact of hyperparameters on AOPU, its robustness regarding changes in hyperparameters, its advantages over other comparative algorithms, and some inherent limitations of the model. Comprehensive and detailed experiments and comparisons have been conducted on two publicly available chemical process datasets, Debutanizer and Sulfur Recovery Unit (SRU). For more information of the dataset please refer to Appendix D
3.1 Baselines
We choose seven different NN models as baselines: Autoformer, Informer, DNN, SDAE, SVAE, LSTM, and RVFLNN, covering four major domains including RNN-based networks, auto-encoder- based networks, attention- based networks, and MLP-based networks. Notably, all baseline models except RVFLNN and AOPU operate solely within the latent space, meaning that there are linear transformations mapping the input data to the latent space and from the latent space to the output space before and after the baseline models. This approach is designed better control the model size.
3.2 Experiment Implementation
Apart from AOPU, which is trained using the approximated minimum variance estimation loss function as previously described, all other deep learning algorithms are trained using the Mean Squared Error (MSE) loss. AOPU's learning rate for gradient updates is set at 1.0, while for all other deep learning algorithms, it is set at 0.005, with the Adam optimizer used for gradient updates. The learning rates of all models remain static throughout the training process. The experimental setup differs based on the requirements of various models regarding input dimensions. Models such as Autoformer, Informer, and LSTTM necessitate an input that includes an additional dimension for "sequence length'. This dimension is preserved as part of the input structure for these models. Conversely, models like DNN, SDAE, SVAE, AOPU, and RVFLNN do not require this additional dimension. For these models, the sequence length and input dimensions are combined and flattened to serve the feature dimensions in the input. AOPU's latent space size is set at 2048. Autoformer, Informer, SDAE, and SVAE utilize two layers each for their encoder and decoder layers; LSTM uses two layers of LSTM layers; RVFLNN and AOPU share identical settings. All models except AOPU and RVFLNN have their latent space sizes set at 16 to ensure the trainable parameters size across all models comparable.
3.3
Main Result
3.3.1 How certain we are about the inverse
According the previous discussion, the existence of the inverse of T i is crucial as does not only impact the numerical stability of the model but also directly determines whether it is possible to recover y from the approximated mapping relationship ă - žy. Clearly, the input feature dimensions d+ h and the batch size b significantly affect whether the inverse of 2" f exists. Specifically, the larger the batch size, the more columns ã has, and the less likely ã is to be column-full-rank: conversely, the longer the sequence length and the larger the input feature dimensions, the more likely T is to be linearly independent and thus column-full-rank, From the following experimental results, it will be clearly observed the impact of batch size and sequence length on RR.
5

---

EE MEDE
LA0
B0
19922
27v
Figure 4: Histogram of the frequency distribution of RR on SRU dataset under varying batch sizes and sequence length settings.
94
b4332
50
Figure 5: Curve of the mean of RR distribution on SRU dataset under varying batch sizes and sequence length settings.
Fig. 4 shows the distribution of the RR for the AOPU model across various batch sizes and sequence length combinations on the SRU dataset, where bs stands for batch size and seq for sequence length. The experimental results align with the previous analysis: increasing the batch size with a fixed sequence length significantly decreases the RR distribution, whereas increasing the sequence length with fixed batch size significantly increases
Fig. 5) shows the mean values of RR distribution changing with sequence length under different batch size settings, marked by red circles every ten data points. Clearly, the experimental results shown in the figure corroborate our analysis that with increasing batch size, the curve's 's slope becomes flatter, indicating the model's 's decreasing sensitivity to changes in sequence length. Compared Fig. Fig. provides additional insights into the mean values of the RR distribution relative to sequence length and batch size, offering more comprehensive insight for subsequent experimental interpretations. Results of the RR study on the Debutanizer are listed in Appendix F
3.3.2 Is the training stable
Stability is erucial characteristic for the online deployment of deep learning models in actual produc- tion processes. Specifically, the incremental updates to model parameters following the observation of new mini-batch data should have a smooth impact on model performance. However, experimental results indicate that most networks in the soft sensor field fail to achieve stable convergence. Fig, 6 provides a detailed display of how the MSE metrics for different networks change with training itera- tions on the SRU validation dataset, with blue solid circles marked every 50 iterations. It is evident that all models, except Autoformer, Informer, LSTM, and AOPU, exhibit significant performance fluctuations training iterations progress. The density of the blue solid circles to some extent represent the likelihood of corresponding performance fluctuations.
It can be observed that the SDAE and SVAE networks, despite experiencing significant fluctuations in validation performance (indicated by large fluctuations in the curve), are mostly stable (as shown by the blue circles concentrated below the curve). In contrast, the DNN and RVFLNN networks

---

Figure 6: Curves of SRU validation loss changes with training iteration for different models with a fixed batch size of 64 at different sequence length settings. The curves are shown in translucent blue, with a solid blue circle labeled on the curve every 50 iterations.
have relatively unstable convergence (indicated by blue circles evenly distributed above and below the curve). Although Autoformer and Informer have relatively stable convergence dynamics, their performance is relatively poor. Specifically, Autoformer consistently converges to a bad output, whereas Informer can effectively learn under identical settings but is sensitive to changes in seq, which can lead to model performance collapse. The convergence process of LSTM is relatively stable, partly explaining why it is a widely adopted baseline in the field of time series analysis; however, LSTM is significantly prone to overfitting and its performance is not outstanding.
In contrast to all other network, AOPU exhibits exceptionally impressive performance. AOPU demonstrates very stable and rapid convergence, with almost no fluctuations in performance as training iterations progress. Furthermore, AOPU is less sensitive to changes in hyperparameters and does not exhibit significant overfitting, making it truly reliable and deployable NN model in production processes. We also notice that with bigger batch size setting, the fluctuations will be mitigated. For comparative training dynamics under other batch size settings, refer to Appendix F
3.3.3 Quantitative analysis
To verify the reliability of the AOPU model's performance and to quantify its comparison with other methods, we implemented two different training strategies. Strategy one involved an early stopping trick and used the best checkpoint to validate the model's performance on the test dataset. Strategy two involved training all models for 40 epochs and using the final checkpoint to test the model performance. All following experiments has batch size set to 64. The outputs of strategy one are presented Table [ while the results of strategy two presented in Table All NN configurations were subjected to 20 independent repeat experiments, with the mean of the experiments represented by uppercase numbers on the left of the table and the standard deviation by lowercase subscript numbers on the right.
From Table we can intuitively compare the optimal performance among all models. Overall, there is not much difference in final performance among the various networks. Notably, almost all MAPE sample in the test dataset where the butane
metrics the Debutanizer dataset exceed 100 due content is nearly zero, which significantly distorts the MAPE calculation. While AOPU performs comparably to other network models in terms of the R metric, its stability is significantly superior, as indicated by much lower standard deviations in the R² values compared to all other models.
Further, to more closely align with real industrial application scenarios. if we do not record the optimal checkpoint but instead complete training for 40 epochs as shown in Table the performance of the models significantly declines. Despite this, AOPU continues to provide stable and reliable performance. In Table it is noted that using the R² metric, AOPU consistently performs best with the sequence length set at 48, and the performance drop from the optimal results calculated using
7

---

Table 1: The evaluation metrics for different models under different seq settings on Debutanizer and SRU dataset with bs set to 64 and best training epoch recorded
Model
Dataset & Metric
Debutanizer MAPE
SRU MАPE
Seq
Name
MSE
R²
MSE
R²
Autoformer
0,0314
0017
177.2 56
0.1588 20.0481
0,00312ga
00003
0.3295 0310
0.2079.:0 0957
Informer
0.0165 20.0025
130.9
0.5574
0.00090
000099=0.0000 20.00011
0.1579
0.7715
DNN
0.0146
0016
121.9 82
0.6076-10.0440
0.00081
20.0017
0.1645.1..00 0248
0.7964220.449
SDAE
0.0134
136.1
0.6406.10.00
0.00080 c00000
0.1453
16
SVAR
0.0138 -0.0018
128.8++30.74
0.6294=0.0862
0000078+0.00004
0.1214+0.0002
0000
0.7992++0.0102
LSTM
0.0259 20.00000002
161.2++
0.3058-2o.0r3
0,00088 =0.0001
0,1445 0000000
0.7750 0292
AOPU
0.0149 22149900.000 0009
200.5 277
0.5992-10.0243
0.0007220.00007
0.16942...011
0.8154-1..00
RVFLNN
0.0114=0.0011
78.88 11
0,69361000664
0.00074g0.00000
0.15842.0.01.
0.8128 0..002
Autoformrr
0.0245 2000200000020
153.1 os
0.209110.000
0.022660.00000
0.2685 1:0 0202
0.40144:000 0774
Informer
0.0154_0.000 00:42
139.334:10.30
0.5864.2..114
0.00110g0.0000
0.1670 0233
0.7210 0072
DNN
0.0122 -00027
124.7++48.75
0.6717=+0.078
0.00091 +0.0014
0. 0,1671 40.0192
0. 0.7684 0861
SDAE
0.0120 20.002
|44.94;2b.14
0.6786-20.06ss
0.00076=0.00009
0,1489 0,1489+0..0.38
0. 0,8070
24
SVAE
U.0127 - 0000
143.0::333.0 10
0.6599_20.0422
0.000722000000
0.128911 1.0.0.0000000 01.4
0.8149 22
LSTM
0.0262 00
192.2:144
0,2961
0.00091
0,1420 1142010.0130
0. 0.7672
AOPU
0.0133 000000
178.3++11.16
0,,421=000181
0.00080=0.00004
0.1868=#0.0002 0000
0.7965=0.0110
RVFLNK
2 0.0109
67.40 oi
0.7070
0.00079
0.16381..00 01:6
0.7986 0287
Autoformer
0.0307
199.34
0,1770
0.00222
0.2730
0.4366
Informer
0.0237=0.004 0.0099
119.++18.44 42
0,3648 18225
00.0000
0.1849 +0.044
0.6679 +..0222
DNN
0.0120 00007
93.9611-5..0 53
0.6779.10.0730
0.00095
0.1693
0.7589 0374
SDAE
0.0111 _0 0020
131.9 131.9127..0
0.686_0.058 0.0039
0.00080 0.00005
0.1534 0124
0. 0.7959 4:0
32
SVAE
0.0133 00.4
143.9+
0.6419=100046
0.00078
0.1300++..01 0118
0. 0,8010 0182
LSTM
0.0257 -00009
180.1 +H0.1++448.20
0,3097 +0,0785
0.00094
0,1488 0084
0. 0,7603 +-3
ADPU
0.0119 00000
155.6 827
10.011
0.00076=00.00008
0.1847.400.00 0048
0,8063 0078
RYFLNN
0.0121
57.82+33.15
0,6751
0.00091+0.00m
0.1725+0.0120
0,7679
Autofonner
0.0307
199.41:21.0
L1177110.0042
0.00245
0.2879
0.3790
Informer
D.0237 -. 0059
119.4 86
0.3648 20. 1597
0.00168 0.00168_0.00004
0.2205 0334
0.5723
13360 1369
DNN
0.0120 20120=0.0020
93.97.+42..04
0.678_0.0702
0.00099
0,1784
0.7489
40
SDAE
0.0111 -0.025
134.4++30.79
0..009:000618
0.00081+0.c0001
0.1546++0.0110
0. 0.7938+0.00.5
SVAE
0.0116 0.022
127.2:1:11.0 .68
0.686822.00m
0.00082
0,138311.001
0. 787888 0310
LSTM
0.0261 -0.0089
197.2++50.01
0.2904+0.11500
0.00093
0. 0,1479 11499+0.0001
0. 7.3688 +-8 0284
ADPU
0.0110 0..000
143.4:2..977 9777
0.7030:10.0121
0.0007110.00001
0.1785 0000
0.8193 0048
RVFLNN
0.0148
53.59
0.6013 20.0773
0.00093
0,1761 10.0010
0,7644 0033
Autoformer
0,0321 =0.002
189.4
0.1390
0.00249 20.0001
0.2879 +0. 0220
0. 0.3676 0.000
Informer
0.0247
000000
122.6+36.11
0,3385 1865
0.00190+0.00033
0,2270+ 0388
0,5186 1092
DNN
0 20144000000
110.0::42.75
0,6941 20.0732
0.00105
0. 0,1796.go.00
0.7332
48
SDAE
0.0105 -0.011
128.5+11.98
0.7182+0.0400
0.00083
+0.0000
0066440.110
0. 0.7890+0.01.8
SVAE
0.0105 00.007
116.5.4334.52
0.7180.20.0477
0.00087
20.00100
0,1389 009e
0.7789 025e
LSTM
0.0268 _ 00066
185.61.60.4 66
0,2813 20.0882
0.00091
000000
0.1471 1 0098
0.7691 1:0 0312
AOPU
0.0107 0 0003
144.94
0.7118 0087
0.00070
0000001
0,1796 0018
0.8209 0037
RVFLNN
0.0157 (0457700.000
60. 11 +1 00
0.5792+0.0711
0.00100
0. 0.1833+0.0020 0200
0. 0.7454
strategy one is minimal. For instance, the Debutanizer dataset, AOPU's final optimal performance is 0.6054, which is a 14.9% decrease from the original 0.7118; on the SRU dataset, its best final performance 0.8069, only a 1.7% decrease from 0.8209.
Compared to the results on the Debutanizer and SRU datasets, other models show more significant declines: Autoformer dropped from 0.2091 and 0.4366 to -0.2396 and 0. respectively, decline of 123.9% and 64.8%; Informer decreased from 0.5864 and 0.7715 to 0.4047 and 0.6897 respectively, declines of 30.9% and 10.6%; DNN dropped from 0.6941 and 0.7944 to 0.5109 and 0.5639 respec- tively, declines of 26.3% and 29.0%; SDAE decreased from 0.7182 and 0,8070 to 0.4470 and 0.7209 respectively, declines of 37.7% and 10.7%; SVAE dropped from 0.7180 and 0.8149 to 0.4315 and 0.7397 respectively, declines of 39.99 99.9% and 9.2%; LSTM went from 0.3097 and 0.7750 to -0.3903 and 0.6280 respectively, declines of 226.0% and 19.0%; RVFLNN decreased from 0.7070 and 0.8 to 0.2844 and 0.5652 respectively, declines of 59.8% and 30.5%.
In terms of robustness, AOPU shows significant improvement. It is observed that AOPU's model performance steadily improves as sequence length increases, in contrast to the other comparison networks, which exhibit large fluctuations. For example, RVFLNN's R² the SRU dataset drastically drops from 0.5652 at seq 16 to -0.8785 at seq 32, indicating extreme robustness.
In terms of stability, AOPU not only demonstrates an outstanding advantage in terms of average R² values but also in standard deviation. From Table it can be seen that the standard deviation for AOPU between the two training strategies changes only slightly, for instance, from 0.0087 to
8

---

Table 2: The evaluation metrics for different models under different seq settings on Debutanizer and SRU dataset with bs set to 64 and no best training epoch recorded
Model
Dataset & Metric
Seq
Name
MSE
Debutamizer MAPE
R²
MSE
SRU MАPE
R²
Autoformer
0.0942.1 0370
217.0
-1.5240.10.pm
0.00836 004426
0,536211 1755
-1.11811.081
Informer
0.0283
140.8
0.2414:0.1164
0.00122 20.000s
0.1750
0.6897
20.0118
0,021510.0022
161.81-66.14
0.4233 07700
0.0017220.0007z
0.22.38
101000
0.5639 190100
SDAE
0.0217
144.61-43.c0
0.4184..
0.00113 000011
0.1559 11155940.00
0.7114
16
SVAE
0.0217+0.0000
161.6 551.6+-3..4
0.4166+0.1144
0.00113+0.c0044
1469 +-00320
=0.181
LSTM
0.0521
215.5 102.7
40.3956:20.400
0.00251+0.oom
0.2055
033.22
0,3634 20.2844
ADPU
0.021522.0007
206.61.9.000 9.90000
0.4239 2.2.9.0...000 01211
000009500000013
0.1963 01:352 01.52
0,751820.0030
RVFLNN
0.0529 107.1
0.1171 1.4770
0.00171g0.0013
0.2540 0807
0.5652.0.2505
Autoformer
0.996910.0200
113.7
-1.5980_10.7860
0.004522000014
0.3901 10-0761
40.15901 3692
Informer
0.022220.0005 000003
134.9429..e
0.4047 0.11422
0.00146.0000047
0.18788 0280
0.6280 1201
DNN
0.0204+0.0000
159.7+27.221
0.4518
0.00281+0.00126
0.2760+0.0600
0,2879 +0.3206
SDAE
0.0206=0.00m2
152.2++44.45
0.4470
0.00118:0.00023
0,1688 0,1688++0.0218
0.6998 20.0500
24
0.0217.20.0018
152.61.33.14
0.4165.0.1300
0.0010620.000511
0.1582 0248
0.7286 0.0073
LSTM
0.0544
226.9
-0.4574
0.0031310.c0144
0.1878 1:
0.6280
ADPU
0.0185=00.000
193.5+5.6199
1.5040+0.01.0
0.00093+0.00000
0.2022 000022
0.7628 00.0232
RVFLNN
0.033810.0470
135.0.18.11
0.0945 1 2610
0.00507
0.3854 2346
40,2499 136660
Autoformer
0.0583
217.3
-0.5629 #0.6210
0.00334
0.3279
0.1536
Informer
0.0359 00.134
135.4-t-3..8
0.0371 0.3612
0.00193+0.00064
0.2163 0.0.3..
0,5091 1837
DNN
0,0206
173.21444.6
0,4459 1-1316
0.00278
0.2780 0858
0.2949 20.3895
SDAE
0.021120.000
179.3
0.4342 g 2228
0.00110
0.1812 0253
0,7209 050024
32
SVAE
0.0239+0.0005
151.3 .ss
0.3598 0..02...2210 2234
0.00102:00.0040
0.1669+ 0861
0.7397 :0
LSTM
0.0519+
243.74-76.24
-0,3903
5..5471
0.002844+0.002
0.2186++0.
0,2803 -3.3448
AOPU
0.0003
191.0 493.0
0.5396 0.0000
0.00085550.00004
0.1945 0070
0.7833=0.011
RVFLNN
0.0305+0.0340
86. 36.84.+
0.1809
0.00741+0.01854
0,401
-0,8785
4-7855+4.0044 004022
Autoformer
0.046220.0136
213.2
40,2396110.3m44
0.00341
0.3528
0.1343
Informer
0.0194.0.00 0059
119.5 91
-0.1124 20.12200.220 5220
0.00201
20.00011000004
0.2283 0285
0.4911 1351
DNN
0.0182
166.4433.o0
0.5109 08.2
0.00422
0,33764
0.0703++.0133
40
SDAE
0.0223++0.0053
158.9+37.77
0.4028+0. 0.1100
0.00140+0..c009
0.1987++0.0422
0.6455 +0.247
SVAE
0.021420.00m
147.5
0.4259 0.1008
0.00118:
1699 099910.0290
0.7001 1001
LSTM
0.0534+00.021
219.6
-0 00,407 40.5600
0.00391
0,2460
0.0090 0.0090+0.615) +0.6161
AOPU
0.015620.0000
17..0.g4.715
0.5804_20.0110
0.0007620.o000
0.184320.0002 0052
0.8069 20.0101
RVFLNN
0.0267 10-030
88.96 88.961.30.0
0.28444 11344110.000
0,00338 0003388..00070
0.3133
0,1442 0.1442_0.7003
Autoformer
0.0492
226.2 44.66 .55
.0.3187 #0.3287
0.00338
0.3655
40.55400.000
0,1440 20.122
Informer
0.0440+0.0015
123.7 124.74-40.20
-0.1798-+00 +0.4700
0.00248+0..00777
0,2389+ 00.89++00000
0.3
DNN
0.0194
167.9 167.91:37.16
0.4799
0.00393
0,3284
0.0046 20.5541
SDAB
200002+0.00
153.0+444.14
1.4116+0.1772
0.00120 +0.0022
0.1831 +0.0272
0.6953 +0.0723
48
SVAE
0.0212
161.04:39.55
0,4315 0.11000
0.00109
20.000s
0,1811 20.077
0.7223 20.084
LSTM
LL0682200031
271.5
-0.8286
182886...8550
0.00539
1 00057
0,2784 0844
40,3652 9060
AOPU 0.014720.000
177.444.002
0.6054
0064
0.00076
0.0000
0,1862 0053
0,8069
10.0002
RVFLNN
0.0479+0.0585
106.6474.60
-0.2843++1448
0.00898 +-0 000208
0.4238+0.3722
-1.2760+5,288
0.0094 on the Debutanizer dataset, an increase of about 8.0%. In contrast, the standard deviation for
other networks often increases several-fold, such as Autoformer on the Debutanizer dataset, which increases from an optimal 0.0481 to 0.3287, an increase of about 583.4%; similar trends are observed with other models.
3.4 Ablation Study
In this section, we further investigate the effects of structural designs for augmentation through some ablation studies, examining the impacts of the ReLU piccewise activation function, the Tanh smooth activation function, and normalization
AOPU's performance. It is important to note that if AOPU is trained using direct gradient descent without dual parameter updates, actually degenerates to an RVFLNN model, and this part of the ablation study has been detailed in section 3.3.3
From Table we can draw two conclusions: The first normalization significantly impairs AOPU's model performance. The second it ReLU piecewise non-lineat activation function suits worse for AOPU than the Tanh activation function. As previously analyzed in A where both the input data and y should to be zero mean, hence reducing the covariance operator R to inner product operator. However, piecewise linear functions like ReL.U and LeakyReI.U are not zero-mean mean, which violates such assumptions.

---

Table 3: AOPU evaluation metrics the Debutanizer and SRU datasets under various combinations of activation functions and layer normalization settings.
Stracture
Dataset & Metric
Acti
Norm LaNorm
Dehutanizer MAPE
SRU
Relu
Tanh
MSE
R²
MSE
MAPE
R²
0
0.0162 00030
183. 2 e6es
0.5654 0082
0.00074.0.
10000002
0,1838 10-0000
0.8115
0.0103 20.00021
148.6
0.7216 27216:0000008
0.00077 10000003
0.1902 =0.0022
0.8026 0.806-00.000-
0.0633 12.02113
117.2
-0.6961 50.5000
0.58831
17910
4.5829 0.06003
-147.98 422
0.0189
157.64-628822
0.4930
0.00205
0.3149 0.3149=0.0619
0,4803
Table 4: Comprehensive experiments on activation function influence to AOPU
Structure
Dataset & Metric
Dehulamizer MAPE
Class
Acti
MSR
R²
MSF
MAPF
R²
Shrink
20010320.0000
151
0.7236
0081
0.00076
00002
0.1882.1o o037
0.8069 0059
Tanh
0.0104
152.0:16.25
0.7202
0.00076
10.00004
0.1894
0. 0.8056
zero-maeun
Tanh Skrink
0.0103 20.0002
152.7.5.803
0,7221 1. 0008
0,00077
40.00003 00000003
0.1891 4. 0056
0,8043 0093
Soft Siyn
0.0103-20.000
152.846.300
0.7225 200 0061
0.00077
00000
0.1905
0,8040 0068
Soft Shrink
0.0104=0.0002
152.6444.58
0.7201
+0.0060
0.00076+0.0c00s
0. 0.1872++0.0050
0.8074 0.0081
Sigmoid
0.0057220.0004
192.3.26.336
0,5777 20.0124
0.00080
0, 0,1925 10-004
0,7955 0089
Relu6
0.0171
187.915.502
0.5416.1.. 0100
0.00074 80003
0,183611 0051
0.811 10 0076
non-2cro-mean
RRclu
0,0165 20163=0.0002
18..341..24
0.5582 00.82220.000 0074
0.00076 00004
0. 0.1858-+0. 00047
0,8072= 007722=0.00 0105
Hard Swish
10000320.o002
149.2.1..300
0.7252 20 20.077 0078
0.00077
1:. 00003
0.1900 ocan
0.8026 0076
Mish
0.010320.0003
151.4
0.7229
0.00079
00008
0. 0.1927
0.7999
To validate the analysis regarding the effects of zero-mean and non-zero-mean activation func- tions on AOPU's performance, an additional comparative experiment was conducted. This experi- ment included 20 independent repetitions for activation functions classified into zero-mean, Hard Shrink, Tanh, Tanh Shrink, Soft Sign, and Soft Shrink, and non-zero-mean, Sigmoid, Relu6, RRelu, Hardswish, and Mish. The results are listed in Table 4
The experimental results largely confirmed the hypotheses outlined previously. In the zero-mean group, whether on the Debutanizer or SRU dataset, fluctuations in MSE, MAPE, and R² metrics were consistently controlled within 1% (with the maximum R² fluctuation being 0.48%, from 0.7236 to 0.7201). Conversely, in the non-zero-mean group, Sigmoid, Relu6, and RRelu all demonstrated notable performance declines on the Debutanizer dataset. Notably, although Hard Swish and Mish are classified as non-zero-mean activation functions, they did not negatively impact AOPU's performance. This could likely be attributed to the fact that Hard Swish and Mish are approximately zero-mean near the zero index, unlike Sigmoid, Relu6, and RRelu, which are non-zero-meat across any arbitrary small neighborhoods.
4 Conclusion and Limitation
This paper introduces novel NN regression model, AOPU, which is grounded in solid mathematics basis and validated through extensive experiments. The results demonstrate its superior performance, robustness, and training stability. The development of AOPU lays the foundation for the practical im- plementation of deep learning soft sensor techniques in industrial processes and provides guidance for subsequent control, monitoring, and optimization management of these processes. The introduction of RR also illuminates a promising and valuable direction for exploring the design of augmentation models. Such prospective topics of value encompass how to reduce the sensitivity of AOPU to batch size and sequence length, how to derive the NG optimization of the augmentation model, and how to bolster the nonlinear modeling capability of the augmentation model.
We note that AOPU is not "plug-and-play" model; it requires adjustments based actual data conditions. AOPU necessitates clear understanding the RR distribution of data intended for application to guide the selection of batch size and sequence length hyperparameters. This requirement stems from the inherent matrix inversion operations in AOPU, When the RR value too low, noise during the AOPU training process greatly exceed the effective information, potentially leading to model divergence as Appendix]i discusses.
FidpendiiFiissuussss..
10

---

References
[1] Xinyu, Z., G. Zhiqiang. Automatic deep extraction of robust dynamic features for industrial big data modeling and soft sensor application. IEEE Transactions on Industrial Informatics, 16(7):445-4467, 2020.
[2] Xiaoxia, C., S. Xuhua, T. Chudong. Multi-time-scale tfe prediction for iron ore sintering process with complex time delay. Control Engineering Practice, 89:84 0984-99,20 2019.
[3] Xiaofeng. Y., Z. Jiao, H. Biao, et al. Hierarchical quality-relevant feature representation for soft sensor modeling: A novel deep learning strategy. IEEE Transactions Industrial Informatics, 16(6):3721-37.66 2020.
[4] Xiaofeng, Y., H. Biao, W. Yalin, et al. Deep soaring--based feature representation and its application for soft sensor modcling with variable-wise weighted sac. IEEE Transactions Industrial Informatics, ((7))3335-32453 2018.
[5] Zhiqiang. G. Process data analytics via probabilistic latent variable models: A tutorial re- view. Industrial and Engineering Chemistry Research, 10.1 1a.10//accieerr8b02999
2018. Doi:
[6] Zhang, Y., Q. Wen, X. wang, et al. Onenet: Enhancing time series forecasting models under concept drift by online ensembling. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine, eds., Advances in Neural Information Processing Systems, vol. 36, pages 69949- 69980. Curran Associates, Inc., 2023.
[7] Siwei, L., Y. Chunjie, Xujie, et al. Blast furnace ironmaking process monitoring with time- constrained global and local nonlinear analytic stationary subspace analysis. IEEE Transactions on Industrial Informatics, pages 1-14, 2023.
[8] Daniel, S., U. Elisabeth. On multilevel best linear unbiased estimators. SIAM/ASA Journal on Uncertainty Quantification, 8(2):601-635, 2020.
[9] Peter, G., J. Ramesh, R. Mohammad. Adaptive experimental design with temporal interference: maximum likelihood approach. Advances in Neural Information Processing Systems, vol. 33, pages 15054-15066 Curran Associates, Inc., 2020.
[10] Pinheiro, T. F., A. d. S. Silveira. Stochastic model predictive control using laguerre function with minimum variance kalman filter estimation. International Journal of Dynamics and Control, 133):1330113500 2023.
[11] Yuming, C., S.-A. Daniel, W. Rebecca. Autodifferentialle ensemble kalman filters. SIAM Journal on Mathematics of Data Science, 4(2):801-833, 2022.
[12] Chong, Y., Y. Chunjie. L. Junfang, et al. Forecasting of iron ore sintering quality index: A latent variable method with deep inner structure. Computers in Industry, 141:103713, 2022.
[13] Yong, L., W. Haixu, W. Jianmin, et al. Non-stationary transformers: Exploring the stationarity in time series forecasting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh, eds., Advances in Neural Information Processing Systems, vol. 35, pages 9881-9893. Curran Associates, Inc., 2022.
[14] Elad, H., L. Holden, S. Karan, et al. Spectral filtering for general linear dynamical systems. In Advances Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018.
[15] Elad, H., S. Karan, Z. Cyril. Learning linear dynamical systems via spectral filtering. In Advances in Neural Information Processing Systems, vol. 30. 2017.
[16] Peter, O., D. M. Bart. N4sid: Subspace algorithms for the identification of combined deterministic-stochale
systems. Automatica, 30(1):75-93, 1994, Special issue on statistical signal processing and control.
[17] Gianluigi, P., A. Aleksandr, Daniel, et Deep networks for system identification: A survey. arXiv preprint arXiv:2301. 12832, 2023,
[18] Ilya, L., H. Frank. Decoupled weight decay regularization. In International Conference on Learning Representations. 2019.
[19] Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv.1412.6980, 2017.

---

[20] James, H., R. Magnus, L. Neil. Fast variational inference in the conjugate exponential family. In Advances in Neural Information Processing Systems, vol. 25. Curran Associates, Inc., 2012.
[21] Amari, Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276. 1998. Yv501 Times Cited: Cited References Count:41.
[22] James, M. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21, 2020. Np3tx Times Cited:69 Cited References Count:73.
[23] Matthew, D. H., M. B. David, W. Chong, et al. Stochastic variational inference. Journal of Machine Learning Research, 14:1303- 1347, 2013. 168md Times Cited: 1 175 Cited References Count:
[24] Hensman, J.. N. Fusi, N. D. Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6835, 2013.
[25] Matthias, B., v. d. W. Mark, R. Carl Edward. Understanding probabilistic sparse gaussian process approximations. In Advances in Neural Information Processing Systems, vol. 29. Curran Associates, Inc., 2016.
[26] Trong Nghia, H., H. Quang Minh, L. Bryan Kian Hsiang. A unifying framework of anytime sparse gaussian process regression models with stochastic variational inference for big data. In Bach, D. Blei, eds., Proceedings of the 32nd International Conference on Machine Learning, vol. 37 of Proceedings of Machine Learning Research, pages 569-578. PMLR, Lille, France, 2015.
[27] Dutordoir, V., J. Hensman, M. van der Wilk, et al. Deep neural networks as point estimates for deep gaussian processes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang. J. W. Vaughan, eds., Advances in Neural Information Processing Systems, vol. 34, pages 9443-9455, Curran Associates, Inc., 2021.
[28] Wu, L., K. Mohammad Emtiyaz, S. Mark. Fast and simple natural natural-gradicn gradient variational inference with mixture of exponential-famin approximations. In K. Chaudhuri, R. Salakhutdinov, eds., Proceedings of the 36th International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research, pages 3992-40022. PMLR, 2019.
[29] Wu, I.,N. N. Frank, F. Khan Mohammad, et al. Tractable structured aatural-gradien descent using local parameterizations.. In M. Meila, T. Zhang, eds., Proceedings of the 38th International Conference on Machine Learning, vol. 139 of Proceedings of Machine Learning Research, pages 6680-6691. PMLR, 2021.
[30] Mohammad, K., L. Wu. inference in non-conjugate models to inferences in conjugate models. In A. Singh, J. Zhu, eds., Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol. 54 of Proceedings of Machine Learning Research, pages 878-887, PMLR, 2017.
variational inference : Converting variational
[31] James, M., Roger. Optimizing neural networks with ffatereer-factored approximate curva- ture. In F. Bach, D. Blei, eds., Proceedings of the 32nd International Conference on Machine Learning, vol. 37 of Proceedings of Machine Learning Research, pages 2408-2417 PMLR. Lille. France, 2015.
[32] Wu, I.., D. Felix, F. Runa, et al. Can we remove the square-root in adaptive gradient methods?
a second-order perspective. arXiv preprint vxXiv:2402.03499 2024.
[33]
Structured inverse-free natural gradient: Memory-efficient and numerically-stee kfac for large neural nets. arXiv preprint arXiv:2312.05700 2023.
[34] Qingqiang, S., G. Zhiqiang. A survey on deep learning for data-driven soft sensors. IEEE Transactions on Industrial Informatics, 17(9)5853-558) 5866, 2021.
[35] Luigi F,, G. Salvatore, R. Alessandro, et al. Soft Sensors for Monitoring and Control of Industrial Processes, vol. 22. Springer, 2007. Springer.
[36] Zhiqiang, G., H. Biao, S. Zhihuan. Mixture semisupervised principal component regression model and soft sensor application. AIChE Journal, 60(2):533-545, 2014.
[37] Yuan, X., L. Li, Y. Wang. Nonlinear dynamic soft sensor modeling with supervised long short-term memory network, IEEE Trunsactions on Industrial Informatics, 16(5):3168-3177 2020.
12

---

[38] Yuan, X. Y. Wang, C. Yang, et al. Stacked isomorphic autoencoder based soft analyzer and its application to sulfur recovery unit. Information Sciences, 534:72-84, 2020.
[39] Chong, Y. Y. Chunjie, Z. Xinmin, et al. Multisource information fusion for autoformer: Soft sensor modeling of feo content in iron ore sintering process. IEEE Transactions on Industrial Informatics, (9(1)) 1584-1155544 2023.
[40] Feng, Y. Y. Chunjie, Z. Xinmin, et al. A 3-d convolution-based bum-through point multistep prediction model for sintering process. IEEE Transactions on Industrial Electronics, 71(4):2219- 4229, 2024.
[41] Zhichao, C., Zhiqiang. Knowledge automation through graph mining, convolution, and explanation framework: A soft sensor practice. IEEE Transactions on Industrial Informatics, 18(9):6068-61)) 607606 2022.
[42] Haoyi, Z., Z. Shanghang, P. Jieqi, et al. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):1 (112))11106-11111 2021.
[43] Albert, G., G. Karan, R. Christopher. Efficiently modeling long sequences with structured state spaces, arXiv preprint arXiv:2111.00300 2021.
[44] Zhichao, C., S. Zhihuam, G. Zhiqiang. Variational inference over graph: Knowledge representa- tion for deep process data analytics. IEEE Transactions on Knowledge and Data Engineering. pages 1-16. 2023.
[45] Yi, T., B. Dara, M. Donald, et al. Synthesizer: Rethinking self-attention for transformer models. In M. Meila, T. Zhang. eds., Proceedings of the 38th International Conference on Machine Learning, vol. 139 of Proceedings of Machine Learning Research, pages 10183-10192. PMLR, 2021.
[46] James, W. B. Viacheslav, T. Alexander, et al. Efficiently sampling functions from gaussian process posteriors. In H. D. III, A. Singh, eds., Proceedings of the 37th International Conference on Machine Learning, vol. 119 of Proceedings of Machine Learning Research, pages 10292- 10302. PMLR, 2020.
[47] Zhiqiang, G., S. Zhihuan. Nonlinear soft sensor development based on relevance vector machine. Industrial and Engineering Chemistry Research, 49(18))8685-8693, 2010. Doi: 10.1021/ie (0021/ie111146 146d.
[48] Lim, S. H. Understanding recurrent neural networks using nonequilibrium response theory. Journal of Machine Learning Research, 22(47): 2021.
[49] Zachary C., L., B. John, E. Charles. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.0000 2015.
[50] Ailing, Z., C. Muxi, Z. Lei, et al. Are transformers effective for time series forecasting? Proceedings of the AAAI Conference on Artificial Intelligence, 1121-111,, 2023.
[51] Chen, C. L. P., Z. Liu. Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Transactions Neural Networks and Learning Systems, 29(1):10- 12.10-2-22222 2018
[52] Malik, A., R. Gao, M. Ganaie, et al. Random vector functional link network: Recent develop- ments, applications, and future directions. Applied Soft Computing, 10377, 2023.
[53] Zhengyuan, Z., M. Panayotis, B. Nicholas, et al. Stochastic mirror descent in variationally coherent optimization problems. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett, eds., Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017.
[54]
On the convergence of mirror descent beyond stochastic convex programming. SIAM Journal Optimization, 30(1):687-716, 2020.
[55] Chaobing, S., Z. Zhengyuan, Z. Yichao, et al. Optimistic dual extrapolation for coherent non-monotone variational inequalities. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., Advances in Neural Information Processing Systems, vol. 33, pages 14303-14314. Curran Associates, Inc.. 2020.
13

---

[56] Bo, D., H. Niao, D. Hanjun, et al. Provable bayesian inference via particle mirror descent. In A. Gretton, C. C. Robert, eds.. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, vol. 51 of Proceedings of Machine Learning Research, pages 985- 994. PMLR, Cadiz, Spain, 2016.
[57] Luigi, F., G. Salvatore, R. Alessandro, et al. Soft sensors for monitoring and control of industrial processes. Springer, London, UK, 2007.
14

---

A Network's mechanism
In this section, we first demonstrate that AOPU implements a MVE through NN, explain the relation- ship between data augmentation, minimum variance, and orthogonal projection. We then discuss the physical significance and necessity of the dual parameter from NGD perspective.
A.1 From MVE perspective
We start by giving the definition of the MVE,
Definition 1. Given the independent variable x and the dependent variable (y(2)) is said to be the MVE for y if the the following hold,
B - (y(x))) & B[(y- [(]l)))
(7)
where E[]] denotes the expectation operator, and f(y] x)) represents any unbiased arbitrary estimation function for given x.
According to definition T it is straightforward that the MVE is the optimal unbiased estimator under the Mean Squared Error loss metric, providing a performance boundary for all regression networks. However, the solution to the MVE, detailed in Appendix C.I represented as f yp(yl x x)dy where p(-) denotes the probability operator, is challenging determine. Since the prior knowledge of the likelihood distribution p(y[z) is not accessible, this integral is difficult to solve. Instead delving into modeling the likelihood, AOPU turns to referencing solvable linear MVE operators for network design. Specifically, when the function form of [y/[z) is constrained to be linear with respect to æ, it can be set as fyy(x) = W m.e. + b, with the solution Woon = Ryz MyzR7Z R 7 and b = Elyl W E[r)
where Rab represents the covariance matrix E [( - Ela)) - =[ For clarity, the proof is listed in Appendix C.2
Given that variables and y have been normalized to have zero mean, i.e., E[x] and Efy] it follows that b = 0, and the covariance operator R degenerates into inner product operation. Revisiting the loss function of AOPU, the parameters Wmye of the linear MVE. Here the (5²*)- is aligned with the R and .. D(W) ought correspond to the estimation of the input-output covariance matrix. It is noteworthy that the parameter W Weeve is not the estimator's output. Therefore, by approximating Wmee in the loss function, AOPU implies an important assumption: unlike other regression algorithms that explicitly model the mapping relationship from ã to y, i.c., F + y, AOPU implicitly models the relationship f + Zy.. Given that d and RR.,1 the key to deriving y from known F and ły lies in the requirement that ş must be column-fuu cauumn-full-ree full-rank. This requirement aligns with the numerical stability needs during the computation process of AOPU, thereby establishing self-consistent mathematical framework for the unit.
evident from Eq. that AOPU essentially estimates
The geometric interpretation of the linear minimum variance estimator as orthogonal projection underpins the naming of the AOPU. AOPU differs from the orthogonal projection in threefold: (1) Orthogonal projection is non- non- parametric batch algorithm, whereas AOPU operates as parametric, gradient- baseed mini-batch optimization algorithm. (2) Orthogonal projection requires the covari- ance matrix's 's inverse to exist definitively, whereas AOPU can employ approximate inverses for its computations. (3) Orthogonal projection strictly adheres to linear minimum variance estimation, but AOPU introduces non-lineed -ineariyy through data augmentation, allowing it to serve versatile minimum variance estimator. The data augmentation techniques illustrated in Fig. Mare critical for enhancing the expressive capabilities of AOPU. We initially improve model expressiveness through fixed, randomly initialized Gaussian matrix G. However, this approach remains confined within the linear transformation. Consequently, in subsequent experiments, the inputs are further augmented using LeakyRelu(G"" z), pushing the model beyond linear transformations.
A.2 From NGD perspective
In this section, we introduce NGD, including the computation of FIM, how to reduce the computa- tional complexity of NGD through EM, and ultimately demonstrate that the truncated gradient of AOPU is an approximated NG.
15

---

We begin with an introduction to the most basic optimization algorithm used in neural network training, Gradient Descent (GD). Assuming the network parameters are represented (A represents W in AOPU), and V denotes the gradient, GD can be defined by the following equation:
(8) where a > represents the network's learning rate. Many optimization algorithms assist the network in escaping local optima and accelerating convergence by incorporating momentum gradients and adaptive learning rates. However, fundamentally, these are first-order optimization methods (considering only first-order derivatives) and typically exhibit suboptimal performance in practice. NG, by calculating the FIM, exploits the information geometry of the model's s output distribution to accelerate convergence more effectively. NGD can be defined by the following equation:
GD:
A++1 = A a x,
NGD:
Ae+1 = de- - (BF(A.))-V...
where 3, often set to I, is a scale factor. F(A,) is the negative expectation of the second-order derivative regarding the network parameter λe. expressed as F(A,) -E((g V log p(ğ Ae)] The essence of the FIM is to act as a preconditioning matrix that properly scales the gradient on the manifold of the parameter space. The primary difference between NGD and GD is that FIM accounts for dependencies among parameters, unlike the assumption of independent gradients for each parameter as in Eq. 8 with a. NGD generally exhibits better performance compared to GD when FIM is well-defined.
The computational cost of calculating the FIM and its inverse is substantial, i.e., cubic in terms of time complexity and quadratic terms of space complexity. However, for specific model structures, the computation of NGD can be straightforward as that of GD. We provide the definition of the exponential family as follows,
p(=|A)= =- h(2) exp (o(=), X)- - A())]
(10)
where p(2[A) is defined as an exponential family function with the structure given on the right-hand- side, h22) is the base measure, 6(2) is the sufficient statistics, À is the natural parameter, and A(A) is the log-partition function. The expectation
is defined as the expected value of the sufficient statistics, expressed as m() (o(=))))(((2)) For exponential family distributions, we propose the following proposition,
The FIM of p(=AA) with respect to X is equivalent to the gradient of the expectation parameter m(X) with respect to
Proposition 2.
Proof. In Appendix C.4
Therefore, the NGD can be simply expressed as follows:
NGD:
det1 \ 8V (m)
(11)
By utilizing the MSE loss function, we can establish a connection between AOPU and NGD. From Eq. 6 it is known that the output of AOPU is the inner product of the model parameter W and the augmented variable ă. Supposed the output is viewd as an Gaussian distribution characterized by mean x" and covariance I, where following optimization objective can be proven to be equivalent to MSE:
an appropriately dimensioned identity matrix. The
L(W) = - Ky 0)" 5)
= - 2) 9) + Tr(I)
(12)
MSE(U.
where Σ denotes the equivalence up to constants.
Proposition 3. The NG of ((W) with respect to W is approximately equivalent to the gradient of L(D(W)) with respect to dual parameter D(W).
Proof. In Appendix C.S
Proposition 3 valids the effectiveness of using truncated gradients for parameter updates in AOPU. Fact that ((D()) is actuallt the aforementioned loss function L, and it can only be an approximation of L(W) because the inverse of jf 7 may not be well-defined as discussed sections 2.3 and A. Therefore, RR not only serves measure of AOPU's numerical stability and its approximation degree to MVE but also employed to quantify how closely truncated gradient approximates the NG.
16

---

B Convergence Analysis
In this section, we are going to analyze the convergence of AOPU referencing the conclusions from 53 54 We eventually demonstrate that under the condition i is column-full-rand, AOPU converges to the optimal solution almost surely. Firstly, thanks to the trackability of parameters, AOPU is capable of being proven coherent optimization problem under strict assumption, which agrees on previous analysis, made about the distribution of the observed samples y during the proof; AOPU's truncated gradient is then proven to structurally ensure consistency with the Stochastic Mirror Descent (SMD), specifically that the dual parameters correspond directly to the mirror map; the assumptions in [53] about regularity (assumption 3), differentiability (assumption 1), and bounded second moments with Lipschitz continuity (assumption 2) are also proven to be met under the condition ž is column- cull--fulllaaiil Finally, by referencing theorem 3,4 from [53] prove that AOPU enter arbitrarily small neighborhood of the optimal parameter solution W*
Definition 2. The optimization problem min Z(W) is said to be coherent if
(VE[C()), W - -W*) 2 for all W E W.W. E W
(13)
with equality holds if and only W E W", where W the feasible parameter space and W is additionally constrainted by W arg min
Proposition 4. If the observed sample is characterized by an underlying D(WY* e D(W)", for short, AOPU's training objective is coherent with respect to D.
Proof. In Appendix C.6
Definition 2 introduces the concept coherent which involves the anal- ysis of the first-order derivatives of the loss function. For conventional NN, due to the untrackability of parameters, analyzing parameter gradients is exceedingly challenging. Although some studies [56 32 3T have explored the parameters' local characteristics, analyzing their global properties remains difficult. Proposition Arrisittonnsuunne summarizes the properties of coherence of AOPU briefly.
Referring the work of 53
55
Proposition 5. If the regularizer in Q is characterized by square Mahalanobis distance with covari- matrix Σ = 3.5 instead of Euclidean distance Σ I, AOPU's training strategy is identical to SMD.
Proof. In Appendix C.7
In SMD, each iteration involves calculating the stochastic gradient from the model's current state, updating within the dual space, and then mapping back to the parameter space. This process is outlined in Algorithm [ where X and Y represent the parameters and dual parameters, respectively, and subscripts denote the iteration number. Q represents the mirror map, which is defined as follows,
Q(Y) = arg max (X, Y) h(X)
(14)
where h acts as a regularizer. Proposition 5 Summarizes the connection between AOPU's training and SMD by carefully selecting regularizer.
Algorithm 1 Stochastic mirror descent
Require: Initial Yo
1: n +- 0
2: repeat
3: X, = =((Y))
4: Yn+1 = Yn an-1 v((Xa)
5: n 1+ n +1
6:
until end
7:
return solution candidate X₂
The assumption about regularity fundamentally guarantees the continuity of the Fenchel coupling at the point where Y equals the subgradient of la with respect to X, and can typically be considered trivially satisfied. Regarding the properties of differentiability and bounded second moments and
17

---

Lipschitz continuity, the objective function of AOPU is typically quadratic, thus both its first and second derivatives exist and are linearly related (+* -1 These assumptions are only satisfied when @ is column-full-rank, i.e., when the inverse is well-defined.
Combining the conclusions from the proofs discussed above with theorem 3.4 from 5 it can be determined that AOPU's dual parameters D(W) always converge to the optimal values under the constraints, Essentially, the primary difference between using W for inference and D(W) for training is term (x" ( ž)ー Therefore, when I is column-full-ramt full-rank, the convergence of the dual parameters D(W) is equivalent to the convergence of the parameters W. This conclusion is consistent with the results from the earlier MVE analysis and highlights the important role of RR.
The proof of convergence complements the final piece of the puzzle for the deployment of AOPU in advanced industrial applications. It provides solid theoretical support in the aspects of derivation processes, optimization procedures, state monitoring, and convergence assurance. It can be confi- dently stated that AOPU is ready be applied in industrial soft sensing, having established robust foundations for operational reliability and efficacy.
C
Mathematic Proof
C.1 Solution to General Minimum Variance Estimator
In this subsection we about prove that given æ. the solution to the general minimum variance estimator of y is f yp(u]æ x) byy, i.e., Eylz Since it is the expectation of likelihood, this result is intuitive to prove. Rewrite the covariance calculation in the following,
E [y - f(yl=)) - f(vl=))²]
-E (y + Engnaol - E - f (v) Eyjall f(ul=))
(15)
=E [(v (2) [u) + E [(Ey] (E, f(ll)))
[(y -
- [[ + E [(Eg) (ognaalw-(((ll)(((
(wlx)) - Eyl [w)]
Noting that [y](x) is not a conditional probabilistic distribution representation, it denotes function that takes x as input, and the output of such function is regarded as an estima- tor of y. In conclusion, f(y|z) is fundamentally independent of y, therefore, for the term [Eygely) - f(ylx))(y [y))
we can rewrite it into,
E [By h [u] f(yl=)) Fyle(v))²]
/ / (Bylalu) (( F. [v)) p(x, v) drdy
/ (Eyti - f(ul=)) (/ (v - Bylelv) p( p(x)dox
p(ul=)
(16)
(Ey)=ll ((yl=)) Enja p(=)dr
The conclusion also applies to the term E - Consequently, the last
two terms in Eq. 16 consistently equal zero. Given that E [F [v] -1ly)))))(FFF.)..)))))))) (F - f(y(=))² semi- comitpositive definite, it follows that E [[v - (v)) - [y) establishes lower bound for E [ (y)l))) f()(=))²] Equality holds if and only if fy]rr) = Ey|z which completes
the proof.
C.2 Solution to Linear Minimum Variance Estimator
In this subsection, we are about to prove that the solution to the linear minimum variance estimator is W Wmee- Ry and = Flyy Ejr). Initially, it straightforward to see that the value of renders the estimator unbiased. By simply taking the expectation, can complete the proof. Again
18

---

we rewrite the covariance calculation in the following.
E [y (1 [((l=)))]
(17)
E (vv Ely] + W Ela] - W Wmee Ely] + W weve - W muve 2 "]
Incorporating the covariance matrices Ry [x [xx]) - ([)])=-)=>>]]] and Rzy = E [( - (V - - We can reformulate Eq. Upon simplification, this equation
E [ly El Fly)) (y- Ely))²), R
17 as Ry + W weee Rex WT mue Ryz mee W Wmve Rzy.
transforms to,
Ryy + WaeeR W
mon RyzWer W
W WmeeRay Ray
moe
= -eeeeke Raw W more
W Rer R Ryz R 20 Rax W more + Ryv + Rya
Ry R. Ray
R Ro Rzy
-(W mom Ry R
REy moee- Ryt) Ry Ry R Ruy
(18)
Ran Wmne- - Ry is again semi-positive definite, indicating that the optimal W Wooe is identical to R R 22 which completes the proof.
Noting that (W mane- Ry 22
C.3 Proof to Proposition
Suppose there exists an operator T independent of æ such that for given W and any inputs æ and 22. the Eq. holds. From the linearity property of operators, it follows that,
acti(Wa1) + 2c)) T(W)(z) x22)
(19)
acti( xx) acti(Wax) acti(W((: x22)
Due to the nonlinearity of the activation function, is clear that Eq. doesn't hold, consequently completing the proof.
C.4 Proof to Proposition 2
We are about to give concise and precise proof in this section, starting by proving the connection between the expectation-paramert and the log-partition function.
Proposition 6. The ienetaatiio-prramtter equals to the gradient of log-partition function with respect natural parameter.
Proof. Since EM represents a probability distribution, the log-partition function acts as a normalizing factor, thus the following identity holds true,
/ h(2z) exp (()(),
A(A) log
(20)
Therefore, expectation-paraneete could be derived from differentiating A()) with respect X.
VxA(A) V log / h(x) ((d())))
VA / h1(2)) exp (()(=)
I (22) exp (()(2), A))dz
(21)
VA (0(z), A) (J (12)) exp A))d))
J h22) exp ())dz
[p(=)x)((z)
Clearly, A(X) is the only term that second-order derivable the score function log (=))). The FIM can then be intuitively derived from its definition.
F(A) EA(2IA) V [({A()
= EE((EAA) (
(22)
= Vm(AA)
19

---

C.5 Proof to Proposition 3
We first reiterate that treating the output ( ()) as a Gaussian distribution is merely a prior assumption and does not alter the structure or computation of AOPU. Representing this Gaussian distribution as the minimal EM can be expressed as follows,
1 (B- - F. - - ='
NO W, (2
I
exp
2
(23)
1 2/2 oT W WT (5). W.
(
exp
Under this representation, the sufficient statisties and the natural parameter are respectively zg and W. From this, by the definition of the expectation parameter, we calculate m mW) E N w [T]]] equals to er W which is exactly identical D(W). According to proposition 2 the FIM with respect W is equivalent to the gradient of D(W) with respect W. Thus, by introducing D(W), we can accelerate the NGD computation with respect to W shown below,
Note that L(W) is equivalent to MSE. To compute the gradient of È D(W) using automatic differentiation tools and avoid complex algebraic operations, we design L(D(W)) as,
((D(W) = =f 9) 0)]
(24)
Clearly, ((V( is identical the introduced in section 2.3
C.6 Proof to Proposition
We now assume the observed sample y fully characterized by dual parameter addition with zero-mean random variable Such constraint implies that there exists an optimal parameter set D which fully captures the mean trend of y, i.c., y = (5" ="" + The coherence definition could be rewritten as follows,
D D
=E D.
-E ((
(25)
=E ( - y)]
-E [A(D)] +B((('")) -
= [(D) + (
D*),
In Eq. 25 the first equation arises due to the linear invariance of the gradient with respect to expectation. The second equation is derived by expanding the objective function and calculating its gradient, followed by reorganization. The third equation results from adding and subtracting the same variable the objective function and cross-terms from the third equation. The fifth equation reconstructs the objective function under optimal parameter settings from the fourth equation.
the right- hand- side of the second equation. The fourth equation reconstructs
Note that the objective function under globally optimal parameter settings is necessarily less than or equal to objective function under any other parameter settings, thus term E ELĆ(D) - L(D*)) Z Given the previous assumption that each instance within the optimal parameter set perfectly captures the trend in y. the second term on the right-hand side of the fifth equation is equivalent to the previously defined zero-mean random variable e, and hence the second expected value is identically zero. Thus, it proven that AOPU's optimization is coherent.
C.7 Proof to Proposition
Using dást and distg represent Mahalanobis distance and Euclidean distance respectively we have dist (2,,g(z,, yi )) / / and distg(a, y) Vx y. The major difference between them is that
20

---

M63
320
Su
SW8
sate
MANIELLF
Figure Schematic diagram of two industrial process. (a) SRU. (b) Debutanizer.
the former adjusts for the distribution of data across different dimensions. Euclidean distance is essentially the Mahalanobis distance when the covariance matrix is (i.e., when dimensions independent and identically distributed). Both Mahalanobis and Euclidean distances are strictly convex functions with respect the input, making them suitable for mirror maps. Referring to Algorithm [ we have revised the training strategy for AOPU, presented in Algorithm
regularizer terms in
evident that both share consistent optimization structure, thus structurally ensuring that AOPU's optimization process aligns with SMD. The key to the proof lies in establishing the relationship between the mirror map in SMD and the dual parameter AOPU.
Wny Dn M(D.))
(Wi: Dn))
1
DT (7:5") )-ot")--n,n
(26)
Wn (i
Algorithm 2 SMD in AOPU
Require: Initial D(W),
1: -
2: repeat
3: D(W), = Q(W,)
4: Wn+1 = Wn. -VV((D()))))
5: n m+n+1
6:
until end
7:
return solution candidate W.,
Clearly, the mirror map is the solution where the gradient with respect to D,, is zero in (w,, Dr,) h(D.). Eq. 26 details this gradient computation process, where the first equation is obtained by incorporating the square Mahalanobis distance into the regularizer h, and the second equation is derived by differentiating with respect D,, The solution D,, 7. W precisely matches the definition of the dual parameter in AOPU, confirming the coherence of AOPU's optimization strategy with the principles SMD.
D Dataset Description
D.1 Debutanizer
The Debutanizer column part desulfuring and naphtha splitter plant. required to maximize the C5 (stabilized gasoline) content in the Debutanizer overheads(LP gas splitter feed), and minimize
21

---

Table 5: Variable Description
Debutamizer
Process Variables
Unit OC
Description
Process Variables
Unit
Descriptio flow GAS Air foow AIR MRM Sccondary air Bow AIR_MEA_2
U1
Top temperature Top pressure Refux flow
U2
kg.cm m² -
U2
m²
Us
Us
m² b
U2
m²
-h
Flow to next prozess
U4
m² h-1
Gas foo in SWS zome Air flow in SWS z0ne
Us
CC
6 temperature
Ug
m
Us
CC
Bottom temperature
Uy
"C
Bottom temperature B
the C4 (butane) content in the Debutanizer bottoms (Naphtha splitter feed) [57]. butane content is not directly measured on the bottom flow, but the overheads of the downstream deisopentanizer column by the gas chromatograph resulting in a large measuring delay, which the reason soft sensor steps in.
However, the
The dataset comprises 2,394 records, each featuring 7 relevant sensor measurements, The flowehart of the Debutanizer column, detailing the locations of these sensors and their respective descriptions, presented in Fig. (b). The corresponding details can also be found in Table
D.2 Sulfur Recovery Unit
The sulfur recovery unit (SRU) removes environmental pollutants from acid gas streams before they are released into the atmosphere. The main chamber is fed with MEA gas, and combustion is regulated, in air deficiency. by supplying an adequate airflow (AIR MEA). The secondary combustion chamber is mainly fed with SWS gas and a suitable air flow is provided (AIR SWS). The combustion of SWS gas occurs in a separate chamber with excess air, order to prevent the formation of ammonium salts in the equipment, thereby giving rise to the generation of nitrogen and nitrogen oxides. Air flows are controlled by plant operators to guarantee a correct stoichiometric ratio in the tail gas. Control is improved by closed-loop algorithm which regulates further airflow (AIR MEA_ 2) on the basis of analysis of the tail gas composition. On-line analyzers are used to measure the concentration of both hydrogen sulfide and sulfur dioxide in the tail gas of each sulfur line. Hydrogen sulfide and sulfur dioxide frequently cause damage to sensors, which often have to be removed for maintenance. The design of soft sensors able to predict H2S and SO2 concentrations is therefore required.
The dataset contains 10,080 records with 5 relevant sensor measurements. The flowchart for the SRU illustrated in Fig. (a), with the input descriptions provided Table
E
Hyperparameter Scanning
The hyperparameter selection is guided by two principles: first, to ensure the model size of various comparative methods remains comparable; second, to choose hyperparameters that optimize model performance. Fig. X shows the hyperparameter scanning result, where we can see for SRU the smaller setup is recommended, while for Debutanizer bigger model possibly leads to better but still limited performance. However, the model size the compared methods increases dramatically with layers and hidden dims, which means that the efficiency of parameters drops. Therefore, chose hyperparameter settings that keep the model size comparable to that of AOPU, maintaining a balance between performance and efficiency. We also study the influence of the learning rate the stable convergence. Fig. 9 shows that NGD outperforms the conventional gradient method on stable convergence under various learning rate setups, further validating AOPU's contributions.
F Supplementary Figure of Training Dynamics
In this section, we complete the comprehensive RR distribution experimental results and the stability of the training process of AOPU the SRU and Debutanizer datasets. We note that, with fixed sequence length and increasing batch size, the variance during training generally decreases for other models, illustrated in Fig. 10 and Fig. However, for AOPU, increasing bs actually diminishes
22

---

SEu
debutanizer
Figure 8: Hyper parameter scanning result on SRU and Debutanizer.
Figure 9: Training dynamics under various learning rate setups.
performance. As seen in Fig. 10 when bs is 128 and seq is 16, AOPU's performance has declined compared to bs of 64 and seq 16 Fig. characterized by increased fluctuations and an upward shift in the loss curve. Even more critical, Fig. 11 shows that when bs is 288 and seq is 16, AOPU encounters convergence issues.
This performance degradation is due to the gradual shift of the RR distribution toward zero as bs increases. In the AOPU structure design, RR serves not only during the matrix's 's inversion in forward computation but also as theoretical foundation for AOPU's recovery of y from Ty. Therefore, lower RR correlates with poorer AOPU performance. From Fig. 5), we observe that with bs at 128 and seq at 16, the mean RR is around 0.5, while at BS of 288 and sequence length of 16. the mean RR distribution is near 0.25. Given that the RR distribution highly concentrated, the mean value represents the statistical properties of the entire distribution, indicating that AOPU cannot converge when RR falls below 0.25.
indicator of numerical stability
Another interpretation of AOPU's lack of convergence is that when RR is too low (e.g., assuming RR zero), AOPU's pseudo-inverse (5 5) has been calculated (5 (). effectively converting the original normalization process (x" F) - Li ā into squared process of (5 a ā, significantly devi- ating from the initial computational assumptions and causing model collapse. Similar observations
can be found in the Debutanizer experiment results in Fig. 12 Fig.
Fig.
Fig. 15
and Fig. [6
23

---

by
20000
be 2e
by Lib
5400
X0
2011 100
be 2e
by
Figure 10: Curves of SRU validation loss changes with training iteration for different models with fixed batch size of 128 at different sequence length settings.
co0e
2000
10
de
ot
2000
Figure 11: Curves of SRU validation loss changes with training iteration for different models with fixed batch size of 288 at different sequence length settings.
24

---

64 seq:8
bs:64 Seq:16
bs: 64 seq:24
bs: b5:64 seq:32
bs:64 seq:40
10
10
Diarkuton
Ditrrutton
Disribulion
50
5
0.5
1.0
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
RR
RR
RR
RR
RR
6s:128 seq:8
bs:128 seq:16
0s:128
seq:24
bs:128 seq:32
seq:40
Diceibutton
Dicetbution
Diteibuckoo
Ostribution
Ditribacion
10
100
1.00
1.00
1,00
bs:192 seq:8
bs:192 seq:16
bs:192 seq:24
bs:192 seq:32
bs:192 sec:40
Disrrbutton
Dirrrbutonn
Dizrrbutttn
Damibbtto
Dittrbutton
100
0.5
1.0
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
0.50
0.75
1.00
AR
M
AR
Figure 12: Histogram of the frequency distribution of RR Debutanizer dataset under varying batch sizes and sequence length settings. In each subplot, the horizontal axis represents RR, while the vertical axis indicates frequency, with the distribution normalized. Subplots within the same column have the same sequence length, while subplots within the same row have the same batch size.
bs:32
bs:64
bs:96
bs:128
bs:160
5A moon
88 mean
RR mcsn
RM meam
100
50
100
50
100
50
100
50
100
1eq
seg
s0q
5eg
1.0
1.0
1.0
1.0
3.0
BA moan
88 me0n
88 mesh
N0 meen
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
100
100
50
bs:352
bs:384
bs:416
bs:448
bs:480
88 mean
RR mean
BR moan
RM mean
0.0
50
100
50
100
50
100
50
100
50
100
10q
seg
Figure 13: Curve of the mean of RR distribution Debutanizer dataset under varying batch sizes and sequence length settings. In each subplot, the horizontal axis represents sequence length, while the vertical axis indicates the mean of RR distribution. The batch size increases from left to right and from top to bottom.
25

---

30
361
20
340
2.0
seyls
Sepel
Iedi Maliiggl
do
361
66
29
100
1.0
19
40
Da
Figure 14: Curves of Debutanizer validation loss changes with training iteration for different models with fixed batch size of 64 at different sequence length settings.
1012
3ic
3:0
100
Pride
240
45
16
16
10
10
18
3.10
g10
3/1
$00
Figure 15: Curves of Debutanizer validation loss changes with training iteration for different models with fixed batch size of 128 at different sequence length settings.
26

---

10
601
94.88
14.08
2:40
0.00
100
22311
0:218 H4
[401
20 b1311
150
Figure 16: Curves of Debutanizer validation loss changes with training iteration for different models with fixed batch size of 288 at different sequence length settings.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main idea of AOPU is an industrially usable stabilized neural network, whose model structure, mathematical support and experimental results, ablation experiments have been included.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We integrate limitation and conclusion explore the limitations of AOPU compared to conventional neural networks in terms of batch size and sequence length selection constraint.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The proof have been given following each statement.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All experiment details have been disclosed in section 3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
29

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: The authors have developed their work within an roonmaaking facility where both the code and data are treated as confidential. Despite these restrictions, we anticipate that the AOPU can readily replicated by reviewers and future researchers. This due to the model's straightforward and uncomplicated design, NO complex modules or intricate information exchanges between them. This paper focus on establishing a comprehensive framework to support the AOPU's effectiveness. Notably, replicating the model simply involves applying gradient truncation at the appropriate stages (dual parameter stage), underscoring its simplicity and efficiency.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipsscccppullc//eeeeec gwwcc//gg//////egg httpa//ggiideee/eeSuuubiiiiiee www.SSuumiisiiooPooo ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experiment details have been disclosed in section 3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experimental results were obtained after 20 repetitions. The standard deviation is also analyzed in section to ensure that the results significant
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer information are disclosed in section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte:r://///isss//ee http:///urrrpsscccuulcleeeeeeeee http://blcccctccccc.c.el....eeee
Answer: [Yes]
Justification:
This paper conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiciio).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: AOPU is industrial-direct usable neural network, which we believe has potential societal impact.
31

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: AOPU is industrial- usable neural network, which we believe has no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper have used the existing code repository: Autoformer, Informer; and dataset: Debutanizer, SRU, and we have cite the original paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
32

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssii thcode. com con/dataset package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "4ZG0CjEThU", "reviewer": "Reviewer_8Lrh", "summary": "This paper introduces a new model for soft sensor tasks, the Approximated Orthogonal Projection Unit (AOPU), to enhance the stability and interpretability of regression networks. AOPU incorporates trackable and dual parameters, which are treated differently during the inference and training processes.  AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. The paper provides theoretical proof that AOPU is an approximation of both MVE and Natural Gradient Descent (NGD).Experimental results on two chemical process datasets demonstrat that AOPU outperforms other models in achieving stable convergence.", "questions": "I do not have specific questions about the paper.", "limitations": "The limitations of AOPU is discussed in the paper, specifically, the need for an understanding of the RR distribution to guide the selection of hyperparameters. I do not see potential negative societal impact of the work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is novel and has a strong theoretical basis. The authors provide detailed proofs of theorems in the appendix.\n2. If the contents in the appendix are considered, this paper analyses the proposed AOPU from many aspects, and provide sufficient experimental results and ablation study to validate the advantage of AOPU.", "weaknesses": "1. Due to the limitation of paper length, the contents in the formal paper is incomplete. Many important content like quantitative analysis and ablation study are put in the appendix. The formal contents also lacks a conclusion section. For the quality of publishing, I suggest submitting the paper to other platforms like a IEEE Transaction, where the paper length can be longer.\n\n2. The proposed method are not incorporated into DNN structures, therefore its expressive power is limited in more complicated tasks. Considering the requirements of industrial soft sensor tasks, this is not a critical flaw, but it still hinders AOPU from challenging AI applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to reviewer 8Lrh:**\n\nWe greatly appreciate the time and effort the reviewer has dedicated to reviewing our manuscript and thank you for recognizing our work. We will do our best to answer the weaknesses and hope that our answers will make you reconsider raising the AOPU rating.\n\n**Response to weaknesses 1:**\n\nDue to our focus on rigorous experimental validation and theoretical foundations, the manuscript may have become overly extensive. If possible, we will reduce the narrative and formulas in the main text and move quantitative analysis, experimental details, and conclusions into the main text to enhance the integrity of the paper. To be specific, we can \n\n1. We will merge the Introduction and Related Work sections to conserve space.\n2. We will briefly introduce the concepts of trackable and untrackable elements, deferring detailed discussion to the appendix.\n3. We will refine and concisely describe the network’s structure.\n\nBy reallocating space saved from these modifications, we can elaborate on crucial sections such as quantitative analysis and conclusions. This reorganization ensures that each critical part of the manuscript maintains readability and coherence.\n\nWe hope to be able to present our research at NIPS, being recognized by a top global AI conference would be immensely encouraging and affirming.\n\n**Response to weaknesses 2:**\n\n**We kindly clarify that this is a misunderstanding.** AOPU in fact can be incorporated into the DNN research framework, differentiating from traditional DNNs that use various stacked nonlinear structures for end-to-end modeling. AOPU separates nonlinearity and output modeling into two independent areas of study. \n\nSpecifically, AOPU’s augmentation projection can be any compatible DNN architecture, which enhances RR. The essence of RR is to measure the number of linearly independent items in a matrix; thus, if RR increases with the choice of augmentation projection, it indicates that the selected projection possesses stronger nonlinear modeling capabilities (transforming linearly dependent items into independent ones). Consequently, we can extensively test and measure the nonlinear modeling capabilities of various DNN frameworks, such as Transformers, Diffusions, GNNs, RNNs, CNNs, etc., as long as they are compatible with time series data. \n**From this perspective, AOPU essentially serves as a unified framework for constructing nonlinear modules using DNNs.**\n\nThe reason we did not conduct these DNN experiments in the paper is that we focused on proving **AOPU’s structural consistency with minimal variance estimation, parameter convergence, and gradient effectiveness, as well as its superior stable training qualities.** AOPU represents a significant contribution and influence in the field of soft sensor deep learning, focusing on research into frameworks that enhance optimization performance and stability. We look forward to in-depth discussion and research on RR and various aug modules with the deep learning community.\n\nWe are grateful that the reviewer recognizes the strengths of our work, such as its **strong theoretical basis**, **detailed proofs of theorems**, **sufficient experimental results**, and **comprehensive ablation study**. We believe we have addressed all concerns raised by the reviewer regarding the limited expressive capability of AOPU. We are confident in AOPU's significant practical value (stable training) and its potential for further research value (future work on RR augmentation). Hoping the reviewer can reconsider raising the AOPU rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZUAzeQEBS", "reviewer": "Reviewer_PyNt", "summary": "The paper introduces the Approximated Orthogonal Projection Unit, the basis for a new neural network, designed to enhance the stability and interpretability of regression models, particularly in industrial soft sensor applications. The primary aim is to address the need for stable and immediate optimization in online settings, where traditional NN training techniques fall short. The paper introduces the theoretical background and demonstrates the effectiveness on two tasks, while also introducing ablations and comparisons to several other techniques.", "questions": "- The resoning/phrasing of lines 25-29 is not clear to me. Could you please elaborate?\n- I am not sure, how the augmentation of Eq. (4) helps. Isnt it just a linear transformation, that introduces no new representation?\n- Could you please provide detaisl of how the hyperparameters (Section 4.3) were found and potentially how sensitive these methods are to changes? It appears that the true value of some of the algorithms might be obscured by improper hyperparameter settings. How many random seeds were used? It is unclear how robust the results are against random variations (see for example the second DNN plot of Figure 5, which appears to be somewhat of an outlier compared to the shorter and longer sequence length).\n- I noticed the following typos: Line 9: parameters'; Line 14: missing 'the', Line 106: integrated; Line 153; Line 156: a\n- Please keep heading capitalization consistent!", "limitations": "The limitations have been addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method appears novel and straightforward.\n- The paper provides a solid theoretical foundation.\n- The paper imrpoves interpretability of the neural network's behavior and training dynamics by differentiatiating between trackable and untrackable parameters, enhancing the interpretability.\n- The authors demonstrate superior performance of AOPU in experiments with two chemical process datasets, showcasing its practical effectiveness in achieving stable convergence compared to existing models.\n- Practical Relevance: Tailors the AOPU framework specifically for industrial soft sensor applications, addressing the need for immediate optimization and stability in online settings.\n- Limitations, such as numerical stability issues during matrix inversion in the training process, are discussed.", "weaknesses": "- Code not published. The justification provided is somewhat questionable, since easy reproducibility should also enable the authors to provide code (possibly mirrored from the code implemented at the company).\n- While the page limit is formally met, the authors make extensive use of the Appendix, including core elements of the paper. The Conclusion and Limitations, for example, are in the appendix.\n- There is no mention of thorough hyperparametertuning and its results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to reviewer PyNt:**\n\nWe are deeply grateful for the time and effort the reviewer has invested in reviewing our manuscript. Your recognition and support are crucial to us. We take each of your concerns seriously and have addressed them thoroughly.\n\n**Response to weaknesses 1:**\n\nTo expand the influence of AOPU and promote community progress, we have proposed to the company that the code be desensitized and then made public. We will continue to advocate for this initiative.\n\n**Response to weaknesses 2:**\n\nThank you for your understanding. We also acknowledge that the organizational structure of the manuscript was not optimal. In order to clearly articulate our reasons for studying AOPU and how it is deployed, we devoted a significant portion of the manuscript to this discussion, which necessitated relegating many experimental results and conclusions to the appendix.\n\nHowever, we believe that this issue can be corrected. As we have planned in the response to reviewer 8Lrh, we are capable of optimizing the structure of the AOPU paper for better presentation.\n\n**Response to weaknesses 3:**\n\nWe acknowledge that two reviewers have expressed concerns regarding the selection of hyperparameters. We've presented comprehensive hyperparameter results in the **global response**, Fig. 2. We can see for SRU the smaller setup is recommended, while for Debutanizer bigger model possibly leads to better but still limited performance. However, the model size of the compared methods increases dramatically with layers and hidden dims, which means that the efficiency of parameters drops. Therefore, we chose hyperparameter settings that keep the model size comparable to that of AOPU, maintaining a balance between performance and efficiency.\n\n**Response to questions 1:**\n\nCertainly! Existing deep learning models are trained on offline datasets where they can employ numerous tricks using epoch information, such as:\n\n1.\tEarly Stopping: After each epoch, if the val loss increases over several consecutive epochs, it is assumed that the model has overfitted, and training is halted to select the model with the optimal val loss for testing.\n\n2.\tWarm Up: In the initial epochs, a smaller lr is used to avoid rapid convergence to local optima, later switching to a constant learning rate.\n\n3.\tLearning Rate Schedule: The lr is dynamically adjusted during training to help the model overcome local optima and fine-tune, using methods like step decay, exponential decay, cosine annealing, or cycle policy—all based on epochs information.\n\nFor the online streaming update processes in actual production, there is no concept of epochs, making these tricks inapplicable.\n\n**Response to questions 2:**\n\nWe thank the reviewer for pointing out this issue. This was indeed an oversight. The expression in Eq. (4) lacked the notation of activation function $\\text{acti}$, and it should be represented as $\\hat{x}=\\text{acti}(\\hat{G}^Tx)$. \n\n **Response to questions 3.1:**  \n\nRegarding the hyperparameter information, we have already conducted a comprehensive analysis in our previous **response to weakness 3**, and therefore, we will not reiterate it here.\n\n**Response to questions 3.2:**  \n\nWe fully understand the reviewers' perplexity regarding some unexpected experimental outcomes, specifically for Autoformer. We used the code provided in Autoformer's GitHub repository. We believe there are two plausible explanations for the outputs of Autoformer:  \n\n1. Time Embedding: The code in this repository includes embedding techniques for year, month, day, hour, and minute. Since our dataset does not contain these detailed timestamps, we replaced this with conventional position embedding.  \n\n2. Historical Label: The repository’s code uses historical labels to predict current labels, which is inappropriate for soft sensors, thus we removed this feature.  \n\nFor Informer, we replicated the experiments using the code from the same repository. As a comparison, Informer performs consistently well with shorter sequence lengths. The contrast between Autoformer and Informer not only verifies the correctness of our code deployment but also highlights several interesting observations:  \n\n1. Autoformer, compared to Informer, has a stronger dependency on label data, such that even after removing the historical labels, Informer still captures the input-output relationships correctly, while Autoformer collapses.\n\n2. In industrial time series data, Transformer models do not necessarily model long-term dependencies better. In fact, with larger batch sizes, LSTM models perform better.\n\n**Response to questions 3.3:**  \n\nAs stated in the **Quantitative Analysis** section of our manuscript, we conducted 20 independent repetitions for each model configuration without specifically restricting random seeds. The mean results are presented in regular font size in Tables 2 to 5, and the std. are presented as lowercase subscripts. Thus, the experimental results can be considered significant. For the visualization of iteration loss, we did not average the results of 20 experiments; instead, we displayed the outcome of a single experiment. While the dynamic changes in iteration loss may indeed be influenced by randomness, they are still representative as the result aligns with our analysis of std. detailed in the **Quantitative Analysis**, lines 827-833, thereby affirming the validity and significance of our findings regarding iteration loss.\n\nThere are also additional experiment results in the **global response**, Fig. 3, where we conducted experiments under different lr setups, we believe such results can also corroborate the aforementioned statement.\n\n**Response to questions 4:**  \n\nThank you for your meticulous review. We will promptly inspect the presentation of AOPU to avoid typos like that.\n\n**Response to questions 5:**  \n\nIndeed! We will ensure that all section and subsection titles are consistently capitalized. Thank you again for your reminder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0wOzk0Ncjx", "reviewer": "Reviewer_SnqU", "summary": "The paper proposes a novel training framework for regression tasks called the Approximated Orthogonal Projection Unit (AOPU), optimized using truncated natural gradients. The authors utilize the Rank Rate (RR) of the augmented data covariance matrix as a metric. They demonstrate that their method offers more stable training than existing architectures and optimizers, which is crucial for industrial applications requiring online training during production. Additionally, the authors provide a comprehensive analysis of their setup's convergence.", "questions": "1. Did the authors try to tune the learning rate or other hyperparameters carefully for all other methods? Those bad-performing/unstable training curves in Fig 5 might be because the authors used only one specific learning rate across all those settings.\n2. Could the authors offer more insights on the augmented data $\\tilde{x}$? Why did the authors choose a random Gaussian matrix as the togo augmentation?\n3. Could the authors explain how the data was fed into the model? Suppose we have sensors U1-U7 in Debutanizer, each with 2394 records. I assume those data are the inputs to the network, so what are the targets? Also, for sequence models, the way those data were fitted into sequence matters, so how did the authors implement that?\n\nI am willing to increase my rating if the authors could address my concerns.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Detailed introduction on the background and intuition.\n2. The method is very simple.\n3. A thorough theoretical analysis of the method was provided.", "weaknesses": "1. Poorly arranged paper; conclusions are at the end of the appendix.\n2. In the introduction, the authors claim that their methods improve interpretability, but they do not explain later why that matters. Also, many existing works explain the behavior at a neuron level; it is not clear why one has to track the parameter itself.\n3. Experimental qualities are not good; no hyperparameter search is mentioned in the paper, which is essential when the authors claim that their method improves training stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to reviewer SnqU:**\n\nWe are grateful for the time and effort you have invested in reviewing our manuscript. We take each of your concerns seriously and are confident that we can address all issues raised to your satisfaction.\n\n**Response to weaknesses 1:**\n\nDue to our focus on rigorous experimental validation and theoretical foundations, the manuscript may have become overly extensive. If possible, we will reduce the narrative and formulas in the main text and move quantitative analysis, experimental details, and conclusions into the main text to enhance the integrity of the paper.\n\n**Response to weaknesses 2:**\n\nThere are many neuro-level studies of NN focusing on mimicking biological features or mathematical patterns, presuming these structures are advantageous and subsequently validating them through experiments. However, such methodologies often lack practical guidance. For instance, the autocorrelation block introduced in Autoformer, which substitutes attention with autocorrelation, is presented as an effective enhancement. However, it is challenging to ascertain at a neurological level whether neurons adhere faithfully to the assumptions during deployment, i.e., capturing terms with the largest autocorrelation coefficients. Moreover, it is difficult to evaluate whether these assumptions positively or negatively impact NN. As a result, we are limited to analyzing outcomes without theoretical guidance, making such approaches novel yet lacking in interpretability.\n\nAOPU’s interpretability differs from conventional research by emphasizing a deep understanding of the overall behavior of networks through the analysis of RR. For example, if the RR is very low, it indicates that the mini-batch data is more homogeneous, potentially leading to performance decline due to accuracy loss in inverse calculations and homogenized data. Conversely, if the RR is high, it suggests that the data quality is good, and the truncated gradient is effective, approximating the natural gradient (Newton method). This situation is akin to achieving a minimum variance estimation, which typically results in good performance. \n\n**Response to weaknesses 3:**\n\nThe hyperparameter selection is guided by two principles: first, to ensure the model size of various comparative methods remains comparable; second, to choose hyperparameters that optimize model performance. We present the detailed hyperparameter information in our **global response**, Fig. 2, where we can see for SRU the smaller setup is recommended, while for Debutanizer bigger model possibly leads to better but still limited performance. However, the model size of the compared methods increases dramatically with layers and hidden dims, which means that the efficiency of parameters drops. Therefore, we chose hyperparameter settings that keep the model size comparable to that of AOPU, maintaining a balance between performance and efficiency.\n\n**Response to question 1:**\n\nWe fully understand the reviewer's perplexity. Noting in **Experiment Implementation** section, line 246 to line 247, the learning rate (lr) of AOPU is set 200 times larger than that for others yet even more stable. This is because the natural gradient is much more conservative than the conventional gradient. We conducted two experiments in our **global response**, Fig. 1 to vividly demonstrate the difference between two gradient methods explaining the superiority of our methods, and Fig. 3 to complement the experiments on different lr setups to address the reviewer's concerns.\n\nThe major difference between natural gradient and conventional gradient is not the step size but the direction. Conventional gradient ignores the parameter manifold and treats every parameter equally. Natural gradient divides the gradient by its second derivative thus it treats the sensitive parameters carefully (low gradient) and non-sensitive parameters boldly (high gradient). That adjustment results in different gradient directions and contributes to better convergence.\n\nDue to space limitations, we presented the results of different lr setups only for LSTM and AOPU. Our findings indicate that AOPU is significantly more robust than LSTM with respect to changes in lr.\n\n**Response to question 2:**\n\nWe find there is a misunderstanding regarding the construction of the aug model in AOPU. It is not necessary for AOPU to follow the specific method we used in the manuscript, which adheres to the traditions of the Broad Learning System and RVFLNN. AOPU’s aug projection can utilize any compatible DNN architecture to model nonlinearity.\n\nDifferentiating from traditional DNNs that use various stacked nonlinear structures for end-to-end modeling, AOPU separates nonlinearity and input-output modeling into two independent areas of study and focuses on the latter one. \n\nDue to space limitations, we defer detailed clarification to our **global response.**\n\n**Response to question 3:**\n\nWe thank the reviewer for emphasizing the problems.\n\nThe targets for the Debutanizer and the SRU (Sulfur Recovery Unit) are the butane content in the bottom flow and the SO2 concentration in the tail gas, respectively. We apologize for any inconvenience caused by the lack of detailed and clear descriptions of the datasets. These datasets are referenced in the book Soft Sensors for Monitoring and Control of Industrial Processes, Appendix A, on pages 231 and 234, respectively.\n\nDuring training, we first construct an input matrix with the shape [b, s, d], where b represents batch size, s represents sequence length, and d represents input dimensions. For models requiring sequential computation or global query operations, such as LSTM, Autoformer, and Informer, we feed the model with the matrix in this form. For other models that do not explicitly require sequential operations, the matrix is reshaped to [b, s*d] before being fed into the model. This process is described in detail in the **Experiment Implementation section**, lines 248 to 254.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers for dedicating their time and effort to scrutinizing the manuscript. We have noted the reviewers' have some concerns and misunderstandings regarding the manuscript's presentation. We wish to clarify the contributions and impact of AOPU on the soft sensor deep learning community more clearly and intuitively here, aiding the reviewers in better assessing the AOPU rating.

AI methods in the soft sensor can **be broadly divided into two steps:** initially extracting abstract features or latent variables from data, followed by modeling the input-output relationships based on these features or variables. Most NN studies **focus on the first step**, proposing a structure at the neuro level that mimics certain biomimetic features or mathematical patterns, assuming that this effectively extracts abstract features, and validating this assumption experimentally. The drawback of this approach is the inability to monitor whether the neurons adhere faithfully to such biomimetic features or mathematical patterns during implementation. Moreover, it is challenging to determine whether the feature extraction structure is conducive to regression problems, much less guide model improvement directions.

AOPU differs from these studies by **focusing on the second step**, namely how to model the input-output relationships. Assuming there is a robust feature extraction module (augmentation block), AOPU focuses on better optimization and more stable convergence. We introduced RR as an interpretability index to provide deep and comprehensive insights into the network dynamics. When RR is low, we expect bad performance because the mini-batch data has much-homogenized information. When RR is high, we expect good performance because AOPU's output approximates the minimum variance estimation, and AOPU's gradient approximates the natural gradient (NG). 

AOPU's research has significant contributions and impacts, which can be summarized as follows:
1. It is fully compatible with historical research results on feature extraction; any model compatible with time series can be attempted as an augmentation block to test model performance.
2. We proposed a unified interpretability index, RR, that guides model improvement directions—to enhance RR—regardless of the model used as the augmentation block.
3. We introduced a unified framework for optimizing soft sensor problems, enabling more stable training and providing solutions for the online deployment of neural networks.

However, combining different deep learning modules with AOPU is not simple. Due to the untrackable nature, we cannot efficiently compute natural gradients for the parameters of the augmentation module. According to the chain rule, the gradient of the augmentation module is a partial natural gradient, and existing optimizers cannot perform such calculations. Furthermore, the dynamic changes in RR for different augmentation blocks during training must also be studied. **These topics are too lengthy for the main purpose of this manuscript—introducing the AOPU framework and performing theoretical derivations and experimental validations—hence, corresponding experiments were not conducted**. We believe this content holds significant potential for future research and are eager to explore it with the community.

We also wish to demonstrate the advantages of natural gradients over conventional gradients through a simple experiment. In Fig. 1 of the global response, we conducted a simple GPR experiment. This GPR had only two parameters, the bias of the mean and the coefficient of the kernel matrix, both constant values. We sampled 100 instances from this GPR and updated these two parameters 100 times using these samples. It was observed that natural gradients require a higher learning rate, while conventional gradients only need a smaller one. The major difference between natural gradients and conventional gradients lies in their directions. Conventional gradients ignore the parameter manifold and treat every parameter equally. Natural gradients, by dividing the gradient by its second derivative, treat sensitive parameters cautiously (low gradient) and non-sensitive parameters boldly (high gradient). This adjustment results in different gradient directions and contributes to better convergence.

Nevertheless, the calculation of natural gradients involves considering the inverse of the Fisher Information matrix, thereby introducing computational complexity cubic to the number of parameters, making it entirely infeasible for neural networks. Existing research on natural gradients is almost entirely focused on conventional machine learning, e.g., considering more complex distributions (such as the product of multiple exponential family) for computing natural gradients. Research in the neural network domain on natural gradients mostly centers on second-order optimizers (such as AdamW), which are merely first-order approximations of second-order natural gradients.

AOPU efficiently computes truncated gradients and ensures they approximate natural gradients through many compromises, leading us to analyze RR. We demonstrate in the manuscript that when RR consistently equals 1, AOPU calculates the natural gradient, and the model outputs the minimum variance estimation. Hence, we could observe in experiments that AOPU's iteration loss surpasses all comparative models in stability and is less prone to overfitting. Conversely, when RR is too low, AOP’s output is bad, highlighting the compromises.

In summary, we reiterate and emphasize AOPU's significant contributions, impact, and potential for future research on soft sensor deep learning. We intuitively explain that AOPU's stable optimization capability stems from its consideration of manifold information. We trust this fully resolves the reviewers' concerns and clarifies all misunderstandings, and we hope the reviewers will consider increasing AOPU's rating.

Meta-review: The paper introduces a novel neural network structure called Approximated Orthogonal Projection Unit (AOPU) designed to enhance the stability and interpretability of regression networks, particularly in industrial soft sensor applications. AOPU truncates gradient backpropagation and optimizes trackable parameter updates, leading to improved stability and convergence. The paper provides a solid theoretical foundation for AOPU, demonstrating its approximation of minimum variance estimation and natural gradient descent. Experimental results on chemical process datasets show AOPU's superiority in achieving stable convergence compared to other models. The proposed method is novel and has a strong theoretical basis. The authors provide detailed proofs of theorems in the appendix. The reviewers have highlighted the practical relevance of the work and the thorough theoretical analysis of the method provided. Given all this I believe that the paper can be accepted provided that the authors incorporate the recommendations made by the reviewers in the final version of the paper.
Decision: Accept (poster)