Title: Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural networks

network's structure design

minimum variance estimation

online learning

training stability

natural gradient

soft sensor
Abstract: Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their  feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation in NN, wherein the truncated gradient approximates the natural gradient. Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel training framework for regression tasks called the Approximated Orthogonal Projection Unit (AOPU), optimized using truncated natural gradients. The authors utilize the Rank Rate (RR) of the augmented data covariance matrix as a metric. They demonstrate that their method offers more stable training than existing architectures and optimizers, which is crucial for industrial applications requiring online training during production. Additionally, the authors provide a comprehensive analysis of their setup's convergence.\n\nSTRENGTHS:\n1. Detailed introduction on the background and intuition.\n2. The method is very simple.\n3. A thorough theoretical analysis of the method was provided.\n\nWEAKNESSES:\n1. Poorly arranged paper; conclusions are at the end of the appendix.\n2. In the introduction, the authors claim that their methods improve interpretability, but they do not explain later why that matters. Also, many existing works explain the behavior at a neuron level; it is not clear why one has to track the parameter itself.\n3. Experimental qualities are not good; no hyperparameter search is mentioned in the paper, which is essential when the authors claim that their method improves training stability.\n\nQUESTIONS:\n1. Did the authors try to tune the learning rate or other hyperparameters carefully for all other methods? Those bad-performing/unstable training curves in Fig 5 might be because the authors used only one specific learning rate across all those settings.\n2. Could the authors offer more insights on the augmented data $\\tilde{x}$? Why did the authors choose a random Gaussian matrix as the togo augmentation?\n3. Could the authors explain how the data was fed into the model? Suppose we have sensors U1-U7 in Debutanizer, each with 2394 records. I assume those data are the inputs to the network, so what are the targets? Also, for sequence models, the way those data were fitted into sequence matters, so how did the authors implement that?\n\nI am willing to increase my rating if the authors could address my concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Approximated Orthogonal Projection Unit, the basis for a new neural network, designed to enhance the stability and interpretability of regression models, particularly in industrial soft sensor applications. The primary aim is to address the need for stable and immediate optimization in online settings, where traditional NN training techniques fall short. The paper introduces the theoretical background and demonstrates the effectiveness on two tasks, while also introducing ablations and comparisons to several other techniques.\n\nSTRENGTHS:\n- The proposed method appears novel and straightforward.\n- The paper provides a solid theoretical foundation.\n- The paper imrpoves interpretability of the neural network's behavior and training dynamics by differentiatiating between trackable and untrackable parameters, enhancing the interpretability.\n- The authors demonstrate superior performance of AOPU in experiments with two chemical process datasets, showcasing its practical effectiveness in achieving stable convergence compared to existing models.\n- Practical Relevance: Tailors the AOPU framework specifically for industrial soft sensor applications, addressing the need for immediate optimization and stability in online settings.\n- Limitations, such as numerical stability issues during matrix inversion in the training process, are discussed.\n\nWEAKNESSES:\n- Code not published. The justification provided is somewhat questionable, since easy reproducibility should also enable the authors to provide code (possibly mirrored from the code implemented at the company).\n- While the page limit is formally met, the authors make extensive use of the Appendix, including core elements of the paper. The Conclusion and Limitations, for example, are in the appendix.\n- There is no mention of thorough hyperparametertuning and its results.\n\nQUESTIONS:\n- The resoning/phrasing of lines 25-29 is not clear to me. Could you please elaborate?\n- I am not sure, how the augmentation of Eq. (4) helps. Isnt it just a linear transformation, that introduces no new representation?\n- Could you please provide detaisl of how the hyperparameters (Section 4.3) were found and potentially how sensitive these methods are to changes? It appears that the true value of some of the algorithms might be obscured by improper hyperparameter settings. How many random seeds were used? It is unclear how robust the results are against random variations (see for example the second DNN plot of Figure 5, which appears to be somewhat of an outlier compared to the shorter and longer sequence length).\n- I noticed the following typos: Line 9: parameters'; Line 14: missing 'the', Line 106: integrated; Line 153; Line 156: a\n- Please keep heading capitalization consistent!\n\nLIMITATIONS:\nThe limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new model for soft sensor tasks, the Approximated Orthogonal Projection Unit (AOPU), to enhance the stability and interpretability of regression networks. AOPU incorporates trackable and dual parameters, which are treated differently during the inference and training processes.  AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. The paper provides theoretical proof that AOPU is an approximation of both MVE and Natural Gradient Descent (NGD).Experimental results on two chemical process datasets demonstrat that AOPU outperforms other models in achieving stable convergence.\n\nSTRENGTHS:\n1. The proposed method is novel and has a strong theoretical basis. The authors provide detailed proofs of theorems in the appendix.\n2. If the contents in the appendix are considered, this paper analyses the proposed AOPU from many aspects, and provide sufficient experimental results and ablation study to validate the advantage of AOPU.\n\nWEAKNESSES:\n1. Due to the limitation of paper length, the contents in the formal paper is incomplete. Many important content like quantitative analysis and ablation study are put in the appendix. The formal contents also lacks a conclusion section. For the quality of publishing, I suggest submitting the paper to other platforms like a IEEE Transaction, where the paper length can be longer.\n\n2. The proposed method are not incorporated into DNN structures, therefore its expressive power is limited in more complicated tasks. Considering the requirements of industrial soft sensor tasks, this is not a critical flaw, but it still hinders AOPU from challenging AI applications.\n\nQUESTIONS:\nI do not have specific questions about the paper.\n\nLIMITATIONS:\nThe limitations of AOPU is discussed in the paper, specifically, the need for an understanding of the RR distribution to guide the selection of hyperparameters. I do not see potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}