Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis

## Abstract

Abstract As a neural network’s depth increases, it can achieve strong generalization performance. Training, however, becomes challenging due to gradient issues. Theoretical research and various methods have been introduced to address this issues. However, research on weight initialization methods that can be effectively applied to tanh neural networks of varying sizes still needs to be completed. This paper presents a novel weight initialization method for Feedforward Neural Networks with tanh activation function. Based on an analysis of the fixed points of the function tanh ⁡ ( a ⁢ x ) 𝑎 𝑥 \tanh(ax) roman_tanh ( italic_a italic_x ) , our proposed method aims to determine values of a 𝑎 a italic_a that prevent the saturation of activations. A series of experiments on various classification datasets demonstrate that the proposed method is more robust to network size variations than the existing method. Furthermore, when applied to Physics-Informed Neural Networks, the method exhibits faster convergence and robustness to variations of the network size compared to Xavier initialization in problems of Partial Differential Equations.

## 1 Introduction

Deep learning has enabled substantial advancements in state-of-the-art performance across various domains (LeCun et al., ; He et al., ). In general, the expressivity of neural networks exponentially increases with depth (Poole et al., ; Raghu et al., ), enabling strong generalization performance. This increased depth, though, can result in vanishing or exploding gradients and poor signal propagation throughout the model (Bengio et al., ), prompting the development of various weight initialization methods.
Xavier initialization (Glorot & Bengio, ) ensures signals stay in the non-saturated region for sigmoid and hyperbolic tangent activations, while He initialization (He et al., ) maintains stable variance for ReLU networks. Especially in ReLU neural networks, several weight initialization methods have been proposed to mitigate the dying ReLU problem, which hinders signal propagation in deep networks (Lu et al., ; Lee et al., ).
However, to the best of our knowledge, research on the initialization method to tackle the stability of extremely deep tanh networks during training is still limited. Such networks commonly use Xavier initialization (Raissi et al., ; Jagtap et al., ; Rathore et al., ) and are widely applied in various domains, such as Physics-Informed Neural Networks (PINNs) (Raissi et al., ) and Recurrent Neural Networks (RNNs) (Rumelhart et al., ), with performance often dependent on model size and initialization randomness (Liu et al., ).

The main contribution of this paper is the proposal of a simple weight initialization method for FeedForward Neural Networks (FFNNs) with tanh activation function. This method facilitates effective learning across a range of network sizes, outperforming Xavier initialization by reducing the need for extensive hyperparameter tuning such as the number of hidden layers and units. The theoretical foundation for this approach is provided through the fixed point of the function $\tanh(ax)$.
We experimentally demonstrate that the proposed method achieves higher validation accuracy and lower validation loss compared to the Xavier initialization method across various FFNN network sizes on the MNIST, Fashion MNIST, and CIFAR-10 datasets. Additionally, the proposed method demonstrates its effectiveness in training across various network configurations within PINNs. Notably, while Xavier initialization shows decreasing loss as network depth increases, it fails to maintain performance beyond a certain depth, leading to increased loss and poor training outcomes. In contrast, the proposed method continues to improve performance even at greater depths, ensuring stable training and better results.

Contributions. Our contributions can be summarised as follows:

- •
We show the conditions under which activation values do not vanish as we increase the depth of the neural network, using a fixed-point analysis (Section 3.1 and 3.2).
- •
We propose a novel weight initialization method for tanh-based neural networks that has strong robustness to variations in network size (Section 3.2 and 3.3).
- •
We experimentally demonstrate that the proposed method is more robust to variations in network size than Xavier initialization on image benchmark datasets and PINNs (Section 4).

## 2 Related works

The expressivity of neural networks typically grows exponentially with depth, resulting in improved generalization performance (Poole et al., ; Raghu et al., ). Weight initialization is crucial for training deep networks effectively (Saxe et al., ; Mishkin & Matas, ). Xavier (Glorot & Bengio, ) and He He et al. () initialization are common initialization methods typically used with tanh and ReLU activation functions, respectively. Various initialization methods have been proposed to facilitate the training of deeper ReLU neural networks (Lu et al., ; Bachlechner et al., ; Zhao et al., ; Lee et al., ). However, to the best of our knowledge, research on weight initialization for neural networks with tanh activation remains limited.
Tanh neural networks have been increasingly used, particularly in physics-informed neural networks (PINNs).

PINNs have shown promising results in solving forward, inverse, and multiphysics problems arising in science and engineering. (Lu et al., ; Karniadakis et al., ; Cuomo et al., ; ; Yin et al., ; Wu et al., ; Hanna et al., ; Bararnia & Esmaeilpour, ; Shukla et al., ; Zhu et al., ; Hosseini et al., ; Mao et al., ). PINNs approximate solutions to partial differential equations (PDEs) using neural networks and are typically trained by minimizing a loss defined by the sum of least-squares that incorporates the residual of PDE, boundary conditions, and initial conditions. This loss is usually minimized using gradient-based optimizers such as Adam (Kingma, ), L-BFGS (Liu & Nocedal, ), or a combination of both. Universal approximation theories (Cybenko, ; Hornik et al., ; Hornik, ; Park et al., ; Guliyev & Ismailov, ; Shen et al., ; Guliyev & Ismailov, ; Maiorov & Pinkus, ; Yarotsky, ; Gripenberg, ) guarantee the capability and performance of neural networks as an approximation of the analytic solution to PDE. However, PINNs still face challenges in accuracy, stability, computational complexity, and tuning optimal hyperparameters of loss terms. To alleviate these issues, many authors have introduced enhanced versions of PINNs: (1) the self-adaptive loss balanced PINNs (lbPINNs) that automatically adjust the hyperparameters of loss terms during the training process (Xiang et al., ), (2) the Bayesian PINNs (B-PINNs) that are specialized to deal with forward and inverse nonlinear problems with noisy data (Yang et al., ), (3) Rectified PINNs (RPINNs) that are trained with the gradient information from the numerical solution by the multigrid method and designed for solving stationary PDEs  (Peng et al., ), (4) Auxiliary Pinns (A-PINNs) that effectively handle integro-differential equations (Yuan et al., ), (5) conservative PINNs (cPINNs) and exetended PINNs (XPINNs) that adopt the domain decomposition technique (Jagtap et al., ; Jagtap & Karniadakis, ), (6) parrel PINNs that reduces the computational cost of cPINNs and XPINNs (Shukla et al., ), (7) gradient-enhanced PINNs (gPINNs) that use the gradient of the PDE loss term with respect to the network inputs  (Yu et al., ).

PINNs primarily employ Xavier initialization for training (Jin et al., ; Son et al., ; Yao et al., ; Gnanasambandam et al., ; Song et al., ), but our experimental results indicate that this method limits the performance of larger network sizes. Although there have been recent results on initialization methods for PINNs, most of them have relied on transfer learning (Tarbiyati & Nemati Saray, ). Thus, we propose a weight initialization method that does not require transfer learning and is robust to variations in network size.

Figure: (a) Xavier Initialization
Refer to caption: extracted/5897822/figure/Max_min_xavier.png

## 3 Proposed Weight Initialization method

In this section, we discuss the proposed weight initialization method. Subsection 3.1 introduces the theoretical motivation behind the methodology. Subsection 3.2 presents how to derive the initial weight matrix that satisfies the conditions outlined in Subsection 3.1. Finally, in Subsection 3.3, we suggest the optimal hyperparameter $\sigma_{z}$ in the proposed method.

### 3.1 Theoretical motivation

Experimental results in Figure 1 reveal that when Xavier initialization is employed in FFNNs with tanh activation, the distribution of activation values tends to cluster around zero in deeper layers. This vanishing of activation values can hinder the training process due to a discrepancy between the activation values and the desired output. However, theoretically preventing this phenomena is not straightforward. In this subsection, we gives a theoretical analysis based on a fixed point of $\tanh(ax)$ to bypass the phenomena. Before giving the theoretical foundations, consider the basic results for a tanh activation function. Recall that $x^{*}$ is a fixed point of a function $f$ if $x^{*}$ belongs to both the domain and the codomain of $f$, and $f(x^{*})=x^{*}$. The proofs of Lemma 1 and Lemma 2 are provided in Appendix A.

###### Lemma 1 .

For a fixed $a>0$ define the function $\phi_{a}:\mathbb{R}\to\mathbb{R}$ given as

$$ $\phi_{a}(x):=\tanh(ax).$ $$

Then, there exists a fixed point $x^{\ast}$. Furthermore,

- $(1)$
if $0<a\leq 1$, then $\phi$ has a unique fixed point $x^{\ast}=0$.
- $(2)$
if $a>1$, then $\phi$ has three distinct fixed points: $x^{\ast}=-\xi_{a}$, 0 0, $\xi_{a}$ such that $\xi_{a}>0$.

Remark that the function $\phi_{a}$ can be considered as one-layer tanh FFNN.

###### Lemma 2 .

For a given initial value $x_{0}>0$ define

$$ $x_{n+1}=\phi_{a}(x_{n}),\quad n=0,1,2,\ldots.$ $$

Then $\{x_{n}\}_{n=1}^{\infty}$ converges regardless of the positive initial value $x_{0}>0$.
Moreover,

- $(1)$
if $0<a\leq 1$, then $x_{n}\to 0$ as $n\to\infty$.
- $(2)$
if $a>1$, then $x_{n}\to\xi_{a}$ as $n\to\infty$.

Note that the parameter $a$ in Lemma 2 does not change across all iterations. In Propositions 3 and Corollary 4, we address cases where the value of $a$ varies with each iteration.

###### Proposition 3 .

Let $\{a_{n}\}_{n=1}^{\infty}$ be a positive real sequence, i.e., $a_{n}>0$ for all $n\in\mathbb{N}$, such that only finitely many elements are greater than $1$.
Suppose that $\{\Phi_{m}\}_{m=1}^{\infty}$ is a sequence of functions defined as for each $m\in\mathbb{N}$

$$ $\Phi_{m}=\phi_{a_{m}}\circ\phi_{a_{m-1}}\circ\cdots\circ\phi_{a_{1}}.$ $$

Then for any $x\in\mathbb{R}$

$$ $\lim_{m\to\infty}\Phi_{m}(x)=0.$ $$

###### Proof.

Set $N=\max\{n|a_{n}>1\}$. Define the sequences $\{b_{n}\}_{n=1}^{\infty}$ and $\{c_{n}\}_{n=1}^{\infty}$ such that $b_{n}=c_{n}=a_{n}$ for $n\leq N$, with $b_{n}=0$ and $c_{n}=1$ for $n>N$.
Suppose that $\{\hat{\Phi}_{m}\}_{m=1}^{\infty}$ and $\{\tilde{\Phi}_{m}\}_{m=1}^{\infty}$ are sequences of functions defined as for each $m\in\mathbb{N}$

$$ $\hat{\Phi}_{m}=\phi_{b_{m}}\circ\phi_{b_{m-1}}\circ\cdots\circ\phi_{b_{1}}, \quad\tilde{\Phi}_{m}=\phi_{c_{m}}\circ\phi_{c_{m-1}}\circ\cdots\circ\phi_{c_{ 1}}.$ $$

Then, the inequality $\hat{\Phi}_{m}\leq\Phi_{m}\leq\tilde{\Phi}_{m}$ holds for all $m$.
By Lemma 1, for any $x\geq 0$, we have $\lim_{m\to\infty}\hat{\Phi}_{m}=0$ and $\lim_{m\to\infty}\tilde{\Phi}_{m}=0$. Therefore, the Squeeze Theorem guarantees that $\lim_{m\to\infty}\Phi_{m}(x)=0$.

∎

###### Corollary 4 .

Let $\epsilon>0$ be given. Suppose that
$\{a_{n}\}_{n=1}^{\infty}$ be a positive real sequence such that only finitely many elements are lower than $1+\epsilon$. Then for any $x\in\mathbb{R}\setminus\{0\}$

$$ $\left|\lim_{m\to\infty}\Phi_{m}(x)\right|\geq\xi_{1+\epsilon}$ $$

###### Proof.

Set $N=\max\{n\mid a_{n}<1+\epsilon\}$. Define the sequence $\{b_{n}\}_{n=1}^{\infty}$ such that $b_{n}=a_{n}$ for $n\leq N$, and $b_{n}=1+\epsilon$ for $n>N$. The remainder of the proof is analogous to the proof of Proposition 3.
∎

###### Lemma 1 .

###### Lemma 2 .

###### Proposition 3 .

###### Proof.

###### Corollary 4 .

###### Proof.

### 3.2 The derivation of the proposed weight initialization method

To establish the notation, consider a feedforward neural network with $L$ layers.
The network processes $K$ training samples, denoted as pairs $\{(\bm{x}_{i},\bm{y}_{i})\}_{i=1}^{K}$, where $\bm{x}_{i}\in\mathbb{R}^{N_{x}}$ is training input and $\bm{y}_{i}\in\mathbb{R}^{N_{y}}$ is its corresponding output. The iterative computation at each layer $\ell$ is defined as follows:

$$ $\bm{x}^{\ell}=\tanh(\mathbf{W}^{\ell}\bm{x}^{{\ell}-1}+\mathbf{b}^{\ell})\in \mathbb{R}^{N_{\ell}}\quad\text{for all }\ell=1,\ldots,L,$ $$

where $\mathbf{W}^{\ell}\in\mathbb{R}^{N_{\ell}\times N_{\ell-1}}$ is the weight matrix, $\mathbf{b}^{\ell}\in\mathbb{R}^{N_{\ell}}$ is the bias, and $\tanh(\cdot)$ is an element-wise activation hyperbolic tangent function.

We present a simplified analysis of signal propagation in FFNNs with the tanh activation function.
For notational convenience, it is assumed that all hidden layers, as well as the input and output layers, have a dimension of $n$, i.e., $N_{\ell}=n$ for all $\ell$. Given an arbitrary input vector $\bm{x}=(x_{1},\ldots,x_{n})$, the first layer activation $\bm{x}^{1}=\tanh(\mathbf{W}^{1}\bm{x})$ can be expressed component-wise as:

$$ $x^{1}_{i}=\tanh\big{(}w^{1}_{i1}x_{1}+\cdots+w^{1}_{in}x_{n}\big{)}=\tanh\bigg {(}\Big{(}w^{1}_{ii}+\sum_{\begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{n}\frac{w^{1}_{ij}x_{j}}{x_{i}}\Big{)}x_{i}\bigg{)},\, \text{for }i=1,\dots,n.$ $$

For the $k+1$-th layer, $i=1,\ldots,n$, this expression can be generalized as:

$$ $x_{i}^{k+1}=\tanh\left(a_{i}^{k+1}x_{i}^{k}\right),\,\text{where}\quad a_{i}^{ k+1}=w^{k+1}_{ii}+\sum_{\begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{n}\frac{w^{k+1}_{ij}x_{j}^{k}}{x_{i}^{k}}.$ (1) $$

According to Lemma 2, when $a>1$, for an arbitrary initial value $x_{0}>0$ or $x_{0}<0$, the sequence $\{x_{k}\}$ defined by $x_{k+1}=\tanh(ax_{k})$ converges to $\xi_{a}$ or $-\xi_{a}$, respectively, as $k\to\infty$. This result indicates that the sequence converges to the fixed point $\xi_{a}$ regardless
of the initial value $x_{0}$ and ensures that the activation values do not vanish as network depth increases. Furthermore, by Lemma 2, if $a^{k}_{i}\leq 1$ for all $N\leq k\leq L$, then $x_{i}^{L}$ approaches zero. Therefore, to ensure that $($i$)$ $a_{i}^{k}$ remains close to 1 and $($ii$)$ $a_{i}^{k}\leq 1$ does not hold for all $N\leq k\leq L$, we design the initial weight matrix as $\mathbf{W}^{\ell}=\mathbf{D}^{\ell}+\mathbf{Z}^{\ell}\in\mathbb{R}^{N_{\ell}
\times N_{\ell-1}}$, where $\mathbf{D}^{\ell}_{i,j}=1$ if $i\equiv j\pmod{N_{\ell-1}}$, 0 otherwise, and $\mathbf{Z}^{\ell}$ is a noise matrix drawn from $\mathcal{N}(0,\sigma_{z}^{2})$,
where $\sigma_{z}$ is set to $\alpha/\sqrt{N^{\ell-1}}$ and $\alpha=0.085$. Then $a_{i}^{k+1}$ follows the distribution:

$$ $a_{i}^{k+1}\sim\mathcal{N}\Bigg{(}1,\,\sigma_{z}^{2}+\sigma_{z}^{2}\sum_{ \begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{n}\bigg{(}\frac{x_{j}^{k}}{x_{i}^{k}}\bigg{)}^{2}\Bigg {)}.$ (2) $$

According to Equation 2, the mean of $a_{i}^{k+1}$ is $1$, so choosing an appropriate $\sigma_{z}$ satisfies condition $($i$)$. For condition $($ii$)$, if $x_{i}^{k}$ becomes small relative to other elements in $\bm{x}^{k}$, the variance of $a_{i}^{k+1}$ increases, as indicated by Equation 2. As a result, the probability that the absolute value of $x_{i}^{k+1}$ surpasses that of $x_{i}^{k}$ is higher. However, if $\sigma_{z}$ is too small, the increase in the variance of $a_{i}^{k+1}$ becomes limited. Therefore, choosing an appropriate $\sigma_{z}$ is crucial.

Figure: (a) Same dimension
Refer to caption: extracted/5897822/figure/Max_min_proposed_std_same1.png

### 3.3 Preventing activation saturation via appropriate σ z subscript 𝜎 𝑧 \sigma_{z} italic_σ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT tuning

In this subsection, we discuss how $\sigma_{z}$ impacts the scale of the activation values.
Equation 2 indicates that $a_{i}^{k}$ follows a normal distribution, with variance depending on $\sigma_{z}$. Firstly, we experimentally investigated the impact of $\sigma_{z}$ on the scale of the activation values.
As demonstrated in Figure 2, increasing $\sigma_{z}=\alpha/\sqrt{N_{\ell-1}}$ causes the activation values in any layer to be distributed over a broader range. However, setting $\sigma_{z}$ to a large value can lead to saturation, where most activations converge towards $-1$ and $1$. If $\sigma_{z}$ is too large, the probability that $a_{i}^{(k)}$ takes values far from $1$ (e.g., -10, 5, etc.) increases.
This, in turn, increases the value of $1+\epsilon$ mentioned in Corollary 4, potentially bounding the activation values in sufficiently deep layers by $\xi_{1+\epsilon}$. Consequently, the activation values in deeper layers become less likely to approach zero and tend to saturate toward specific values. On the other hand, if $\sigma_{z}$ is too small, as mentioned in Subsection 3.2, the variance of $a_{i}^{k}$ becomes restricted. This is demonstrated experimentally in Figure 2, when $\alpha=0.00001$. For this reason, we experimentally found an optimal $\sigma_{z}=\alpha/\sqrt{N^{\ell-1}}$, with $\alpha=0.085$, that is neither too large nor too small. Results from experiments solving the Burgers’ equation using PINNs with varying $\sigma_{z}$ are presented in Appendix B.1.

**Table 1: Validation accuracy and loss are presented for FFNNs with varying numbers of nodes $(2,8,32,128)$, each with $20$ hidden layers using the tanh activation function. All models were trained for $20$ epochs, and the highest average accuracy and lowest average loss, computed from $10$ runs, are presented. When comparing different initialization methods under the same experimental settings, the better-performing method is highlighted in bold. Underlined values indicate the highest accuracy when only the number of nodes is varied.**
|  | 2 | 8 | 32 | 128 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MNIST | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 49.78 | 1.632 | 68 | 0.958 | 91.67 | 0.277 | 95.45 | 0.154 |
| Proposed | 62.82 | 1.185 | 77.95 | 0.706 | 92.51 | 0.255 | 96.12 | 0.134 |
| FMNIST | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 42.89 | 1.559 | 68.55 | 0.890 | 81.03 | 0.533 | 86.2 | 0.389 |
| Proposed | 51.65 | 1.324 | 71.31 | 0.777 | 83.06 | 0.475 | 87.12 | 0.359 |
| CIFAR10 | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 32.82 | 1.921 | 43.51 | 1.608 | 48.62 | 1.473 | 47.58 | 1.510 |
| Proposed | 38.16 | 1.780 | 47.04 | 1.505 | 48.80 | 1.463 | 48.51 | 1.471 |

**Table 2: Validation accuracy and loss are presented for FFNNs with varying numbers of layers $(10,50,100)$, each with $64$ number of nodes using the tanh activation function. All models were trained for $40$ epochs, and the highest average accuracy and lowest average loss, computed from $10$ runs, are presented.**
|  | 10 | 50 | 100 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| MNIST | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 96.55 | 0.112 | 96.57 | 0.123 | 94.08 | 0.194 |
| Proposed | 97.04 | 0.102 | 96.72 | 0.109 | 96.06 | 0.132 |
| FMNIST | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 88.73 | 0.319 | 87.72 | 0.344 | 83.41 | 0.463 |
| Proposed | 89.42 | 0.305 | 88.51 | 0.324 | 86.01 | 0.382 |
| CIFAR10 | Accuracy | Loss | Accuracy | Loss | Accuracy | Loss |
| Xavier | 48.39 | 1.468 | 47.87 | 1.474 | 46.71 | 1.503 |
| Proposed | 48.41 | 1.458 | 48.71 | 1.461 | 48.96 | 1.437 |

## 4 Experiments

In this section, we conduct a series of experiments to validate the proposed weight initialization method. In Subsection 4.1, we evaluate the performance of an FFNN with the tanh activation function on benchmark datasets. In Subsection 4.2, we solve the Burgers’ equation and Allen-Cahn equation using Physics-Informed Neural Networks. Both experiments are conducted across various network sizes to verify whether the proposed method consistently performs well, independent of network depth and width.
The experiments were conducted in TensorFlow without skip connections, normalization layers, and learning rate decay in any of the experiments.

### 4.1 Width Independence in Classification Task

To evaluate the effectiveness of the proposed weight initialization method, we conduct experiments on the MNIST, Fashion MNIST, and CIFAR-10 (Krizhevsky & Hinton, ) datasets, utilizing the Adam optimizer.
All experiments are conducted with a batch size of 64 and a learning rate of 0.0001.
Fifteen percent of the total dataset is allocated for validation.

We apply the proposed weight initialization method to evaluate its effectiveness in training tanh FFNNs, emphasizing its robustness to variations in network width. Four tanh FFNNs are created, each with 20 hidden layers, and with 2, 8, 32, and 128 nodes per hidden layer, respectively. In Table 1, for both the MNIST and Fashion MNIST datasets, the network with 128 nodes achieves the highest accuracy and lowest loss when our proposed method is employed. However, for the CIFAR-10 dataset, the network with 32 nodes yields the highest accuracy and lowest loss when employing the proposed method. In summary, our proposed method demonstrates robustness regardless of the number of nodes in tanh FFNNs. We provide more detailed experimental results in Appendix B.2.

### 4.2 Depth Independence in Classification Task

It is well known that the expressivity of neural networks generally increases exponentially with depth, enabling strong generalization performance (Poole et al., ; Raghu et al., ). Therefore, we employ the proposed weight initialization method to investigate its effectiveness in training deep FFNNs with the tanh activation function, emphasizing its robustness to variations in network depth. We create three tanh FFNNs, each with 64 nodes in all hidden layers, but with 10, 50, and 100 hidden layers, respectively. In Table 2, for both the MNIST and Fashion MNIST datasets, the network with 10 hidden layers achieves the highest accuracy and lowest loss when our proposed method is employed. Both initialization methods perform best in networks with the fewest layers, with performance degrading as the depth increases. However, for the CIFAR-10 dataset, we observe that the performance of the proposed method improves as the number of layers increases.

Furthermore, we conduct experiments with varying hidden layer dimensions, as shown in Figure 3. The network consists of 60 hidden layers, where the number of nodes alternates between 32 and 16 in each layer. We demonstrate superior performance in terms of both loss and accuracy across all epochs on the MNIST and CIFAR-10 datasets.

Figure: (a) MNIST Accuracy
Refer to caption: extracted/5897822/figure/MNIST_acc_notsame.png

### 4.3 Network size independence in pinn

Xavier initialization is the primary method used for training PINNs  (Jin et al., ; Son et al., ; Yao et al., ; Gnanasambandam et al., ).
In this section, we experimentally demonstrate that the method’s training performance is highly dependent on randomness and network size. Additionally, empirical results are provided demonstrating that the proposed method is more robust to variations in network size.

All experiments on Physics-Informed Neural Networks (PINNs) use full-batch training with a learning rate of 0.001. In this section, we solve the Allen-Cahn and Burgers’ equations using a tanh FFNN-based PINN with 20,000 collocation points. For the Allen-Cahn equation, the diffusion coefficient is set to $d=0.01$. The initial condition is defined as $u(x,0)=x^{2}\cos(\pi x)$ for $x\in[-1,1]$, with boundary conditions $u(-1,t)=-1$ and $u(1,t)=-1$, applied over the time interval $t\in[0,1]$. Similarly, for the Burgers’ equation, a viscosity coefficient of $\nu=0.01$ is employed. The initial condition is given by $u(x,0)=-\sin(\pi x)$ for $x\in[-1,1]$, with boundary conditions $u(-1,t)=0$ and $u(1,t)=0$ imposed for $t\in[0,1]$.

The Allen-Cahn equation is expressed as:

$$ $\frac{\partial u}{\partial t}-d\frac{\partial^{2}u}{\partial x^{2}}=-\frac{u^{ 3}+u}{d}$ $$

where $u(x,t)$ represents the solution, $d$ is the diffusion coefficient, and the nonlinear term $u^{3}-u$ models the phase separation dynamics.

The Burgers’ equation is given by:

$$ $\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}=\nu\frac{\partial ^{2}u}{\partial x^{2}}$ $$

where $u(x,t)$ is the velocity field, and $\nu$ is the viscosity coefficient.

Eight tanh FFNNs are created, each with 16 nodes in all hidden layers, but with 5, 10, 20, 30, 40, 50, 60, and 80 hidden layers, respectively. As shown in Table 3, for the Allen-Cahn equation, Xavier initialization achieves the lowest loss at a network depth of 20. However, as the depth increases, the loss gradually rises. In contrast, the proposed method achieves the lowest loss at a depth of 50 and maintains a loss of 0.00057 even at a depth of 80 layers.
For the Burgers’ equation, the proposed method achieves the lowest loss at a depth of 60, while at the same depth, Xavier initialization results in a loss that is an order of magnitude higher (approximately $10^{2}$ difference).

Next, we double the number of nodes to observe the impact of node size on the loss. Eight new tanh FFNNs are created, each with 32 nodes in all hidden layers, and with 5, 10, 20, 30, 40, 50, 60, and 80 hidden layers, respectively. As shown in Table 3, for the Allen-Cahn equation, Xavier initialization achieves the lowest loss at a network depth of 30. However, as the depth increases, the model becomes untrainable, with a loss of 0.694 at a depth of 80. In contrast, the proposed method achieves the lowest loss at a depth of 40 and maintains a loss of 0.00059 even at a depth of 80 layers. For the Burgers’ equation, both methods show similar loss values up to a depth of 30. Beyond a depth of 40, however, the loss steadily increases with the Xavier method, while the proposed method records the lowest loss at a depth of 50.

**Table 3: A PINN loss is presented for FFNNs with varying numbers of layers $(5,10,20,30,40,50,60,80)$ using the tanh activation function. The top table shows results with $16$ nodes per layer, and the bottom table shows results with $32$ nodes per layer. All models were trained for $300$ iterations using Adam and $300$ iterations using L-BFGS. The median PINN loss from the final iteration for the Burgers and Allen–Cahn equations, computed over 5 runs, is presented.**
| Allen-Cahn (16 Nodes) | 5 | 10 | 20 | 30 | 40 | 50 | 60 | 80 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Xavier | 9.58e-04 | 8.16e-04 | 7.61e-04 | 1.06e-03 | 1.1e-03 | 1.24e-03 | 3.55e-03 | 1.81e-03 |
| Proposed | 9.21e-04 | 7.29e-04 | 5.76e-04 | 5.29e-04 | 5.37e-04 | 4.03e-04 | 4.73e-04 | 5.77e-04 |
| Burgers (16 Nodes) | 5 | 10 | 20 | 30 | 40 | 50 | 60 | 80 |
| Xavier | 6.97e-03 | 1.11e-02 | 7.9e-03 | 9.71e-03 | 2.45e-02 | 2.65e-02 | 6.5e-02 | 5.71e-02 |
| Proposed | 6.19e-03 | 5.08e-03 | 5.28e-03 | 9.31e-04 | 3.56e-03 | 8.27e-04 | 3.43e-04 | 2.05e-03 |

## 5 Conclusion

In this paper, we have introduced a novel weight initialization method for tanh FFNNs, grounded in the theoretical analysis of fixed points of the $\tanh(ax)$ function. Through our fixed-point analysis, we established conditions under which the vanishing or exploding of activation values can be prevented, even as the depth of the network increases.

Our proposed method exhibits strong robustness to variations in network size, as demonstrated across a variety of FFNN configurations and benchmark datasets, including MNIST, Fashion MNIST, and CIFAR-10. In contrast to Xavier initialization, which struggles to maintain stable performance as network depth increases, the proposed method consistently achieves superior results by preserving activation values. Furthermore, we explored the impact of the initialization hyperparameter $\sigma_{z}$ on the distribution of activation values. We demonstrated both theoretically and experimentally that the choice of $\sigma_{z}$ plays a significant role in maintaining the proper range of activations, balancing between vanishing and saturation. In the context of PINNs, the proposed initialization method shows improved performance in solving PDEs such as the Burgers’ equation and the Allen-Cahn equation. By maintaining a stable loss function and achieving faster convergence compared to Xavier initialization, our method demonstrates its practical utility in training networks for physical systems.

A key advantage of the proposed method lies in its robustness to network depth and width, significantly reducing the need for extensive hyperparameter tuning. By maintaining stable performance across varying network configurations, our approach helps to minimize the time and effort spent on searching for optimal network architectures, allowing researchers to focus on model design and other aspects of the training process. This makes the proposed method particularly valuable in large-scale and resource-constrained applications where efficient training is critical.

## Appendix A Proofs of the theoretical results

### A.1 Proof of Lemma 1

###### Proof.

We define $g(x)=\tanh(ax)-x$. Since $g(x)$ is continuous, and $g(-M)>0$, $g(M)<0$ for a large real number $M\in\mathbb{R}^{+}$, the Intermediate Value Theorem guarantees the existence of a point $x$ such that $g(x)=0$.

First, consider the case $0<a\leq 1$. Since $0<a\leq 1$, the derivative $g^{\prime}(x)=a\cdot\text{sech}^{2}(ax)-1$ satisfies $-1\leq g^{\prime}(x)\leq a-1<0$ for all $x$. Hence, $g(x)$ is strictly decreasing and therefore $g(x)$ has the unique root. At $x=0$, $\phi(0)=\tanh(a\cdot 0)=0$. Hence, $x=0$ is the unique fixed point.

Let us consider the case $a>1$. For $0<x\ll 1$, $\tanh(ax)-x\approx(a-1)x$. Since $a>1,\tanh(ax)-x>0$. On the other hand, since $|\tanh(ax)|<1$ for all $x$,

$$ $\lim_{x\to\infty}[-1-x]\leq\lim_{x\to\infty}[\tanh(ax)-x]\leq\lim_{x\to\infty} [1-x].$ $$

By the squeeze theorem, $\lim_{x\to\infty}[\tanh(ax)-x]=-\infty$.
By the intermediate value theorem, therefore, there exists at least one $x>0$ such that $\tanh(ax)=x$. To establish the uniqueness of the positive fixed point, we investigate the derivative $g^{\prime}(x)=a\operatorname{sech}^{2}(ax)-1$. We find the critical points to be $x=\pm\frac{1}{a}\sec^{-1}(\frac{1}{\sqrt{a}})$. It is straightforward to see that $g^{\prime}(x)>0$ in $\left(-\frac{1}{a}\sec^{-1}(\frac{1}{\sqrt{a}}),\frac{1}{a}\sec^{-1}(\frac{1}{
\sqrt{a}})\right)$ and $g^{\prime}(x)<0$ in $\mathbb{R}\backslash\left(-\frac{1}{a}\sec^{-1}(\frac{1}{\sqrt{a}}),\frac{1}{a
}\sec^{-1}(\frac{1}{\sqrt{a}})\right)$. i.e. $g(x)=0$ has exactly two fixed points. Because $g(x)$ is an odd function, if $x^{\ast}$ is a solution, then $-x^{\ast}$ is also a solution. Thus, for $a>1$,
there exists a unique positive fixed point if $x>0$ and a unique negative fixed point if $x<0$.
∎

###### Proof.

### A.2 Proof of Lemma 2

###### Proof.

(1) Since $(\tanh(ax))^{\prime}=a\operatorname{sech}^{2}(ax)<1$ for all $x>0$,
it holds that $x_{n+1}=\phi_{a}(x_{n})<x_{n}$ for all $n\in\mathbb{N}$. Thus the sequence $\{x_{n}\}_{n=1}^{\infty}$ is decreasing. Since $x_{n}>0$ for all $n\in\mathbb{N}$, by the monotone convergence theorem, it converges to the fixed point $x^{*}=0$.
(2) Let $x_{0}<\xi_{a}$. Since $\phi^{\prime}(x)$ decreasing for $x\geq 0$, with $\phi^{\prime}(0)>1$ and $\xi_{a}$ is the unique fixed point for $x>0$, it holds that $x_{n}<x_{n+1}<\xi_{a}$ for all $n\in\mathbb{N}$. Thus, by the monotone convergence theorem, the sequence converges to the fixed point $\xi_{a}$.
The proof is similar when $x_{0}>\xi_{a}$.
∎

###### Proof.

## Appendix B Additional experimental results

### B.1 Preventing activation saturation via appropriate σ z subscript 𝜎 𝑧 \sigma_{z} italic_σ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT tuning

Figure: (a) Same dimension
Refer to caption: extracted/5897822/figure/Burgers_noise_same.png

### B.2 Width independence in Classification tasks

Figure: Figure 5: Validation accuracy and loss are presented for tanh FFNNs with varying numbers of nodes $(2,8,32,128)$, each with 20 hidden layers. All models were trained for 20 epochs on the MNIST dataset, with 10 different random seeds.
Refer to caption: extracted/5897822/figure/MNIST_acc.png

Figure: Figure 6: Validation accuracy and loss are presented for tanh FFNNs with varying numbers of nodes $(2,8,32,128)$, each with 20 hidden layers. All models were trained for 20 epochs on the Fashion MNIST dataset, with 10 different random seeds.
Refer to caption: extracted/5897822/figure/FMNIST_acc.png

Figure: Figure 7: Validation accuracy and loss are presented for tanh FFNNs with varying numbers of nodes $(2,8,32,128)$, each with 20 hidden layers. All models were trained for 20 epochs on the CIFAR-10 dataset, with 10 different random seeds.
Refer to caption: extracted/5897822/figure/CIFAR_acc.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See above in weakness section.
soundness: 3
strengths: The strength of this paper lies in the development of a novel weight initialization technique that facilitates faster convergence and enhances performance in physics-informed neural networks (PINNs) utilizing the tanh activation function.
summary: The paper introduces a novel weight initialization technique specifically designed for neural networks using the tanh activation function. This technique is evaluated against the well-known Xavier initialization method using benchmark datasets. The experimental results demonstrate that the proposed initialization method enhances the convergence speed of Physics-Informed Neural Networks (PINNs) utilizing the tanh function, showing greater robustness to variations in network size. The findings indicate that the new initialization technique outperforms Xavier initialization in solving various problems related to Partial Differential Equations (PDEs).
weaknesses: 1. The comparison of the proposed weight initialization technique solely with Xavier is insufficient; it should also be experimentally evaluated against other state-of-the-art weight initialization methods.
2. Using tanh activation function in entire neural network is not good practice that it has the drawback of the vanishing gradients for the very high and very low values of x.
3. The formulation is more complex than standard methods, which could complicate implementation as shown in Equation (1). 
4. The optimal value of 𝛼 can be highly context-dependent, varying across different architectures, datasets, and tasks, which makes it less universally applicable. Additionally, the choice of 𝛼 can interact with other hyperparameters, such as learning rate and batch size, complicating the overall tuning process during backpropagation, as described in Equation (2).
5. In Section 4.1, the evaluation process utilizes three datasets—MNIST, FMNIST, and CIFAR-10—employing the tanh activation function in every layer. As shown in Table 2, as the number of hidden layers increases, loss gradually increases, which is indicative of overfitting. It would be more effective to use the proposed weight initialization in conjunction with state-of-the-art architectures for training deep neural networks.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: In sections 4.1 and 4.2, the experiment trains for 20 or 40 epochs. Do networks converge to their best accuracy? What is the difference in accuracy when training for more epochs?

The experiments in section 4 are not too computationally intensive, is it possible to include more
data sets or PDE can show the improvements are general?

Can the code used in the experiment can be provided to improve reproducibility?
soundness: 3
strengths: Unlike optimization problems with theoretical guarantees on fixed points, weight initialization
is an important task in deep learning. An initialization scheme with theoretical backing can
have a long-lasting impact, even just for a sub-field of deep learning.

Experiments do show significant improvement when using PINNs to solve PDE.
summary: This work first provides a theoretical analysis of weight initialization when exclusively using tanh
as an activation function. Providing reasons for the clustering behavior when initializing networks.
Based on the developed theory, an initialization scheme is proposed, and the hyperparameter σz
is empirically determined. There are two types of experiments mainly comparing the proposed
initialization with Xavier. The first type of experiment concerns the classification accuracy in the
early training phase. Showing there is an improvement in accuracy across different data sets and
configurations. The second type of experiment concerns solving PDE with PINNS, showing that the
proposed method has a good performance.
weaknesses: The impact depended on exclusively using tanh as an activation function is fundamentally
beneficial in PINNs. As the current state of the paper, there is not enough support for this.

Given that the experiments are not too computationally intensive and the experiment section
only considers a few data sets or PDEs, the demonstrated improvement may not be general.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The novelty of this work is strong, and the topic sounds interesting. However, the writing and structure should be revised again. There are some questions that the authors should be concerned about. 

1. In Eq. 2, $\sigma_{z}$ is set to $\alpha/\sqrt{N^{l}-1}$ and $\alpha = 0.085$. From Figure 2, we can find that the optimal value of $\alpha$ is $0.085$. Is there any theoretical reason why $\alpha$ should be $0.085$. Or should we manually try the value accordingly?

2. Additionally, please scribe what is $\alpha$. There is no definition of $\alpha$.

3. In Figure 3 (c) and (d), the proposed method seems to decrease after 6 epochs. Although the accuracy curve can rapidly reach the peak (faster than Xavier), the robustness of this method also should be discussed. For example, the initialization method can first provide prior knowledge to neural networks, but if it can keep the stability of training or not. Or is it the reason for the high $\alpha$?

4. In Appendix A.1, the authors discussed different conditions. When $x = 0$, whether the vanishment problem will abscond. Please highlight the strategy on how this method can process it.
soundness: 2
strengths: 1. this article proposes one novel weight initialization method for the FFNN and PINN.
2. the authors prove that the activation values cannot vanish when increasing the depth of the neural network by using a fixed-point analysis.
summary: Weight initialisation is an old topic, and most studies have verified that weight initialization methods can improve the performance of neural networks. However, because that neural network’s depth increasing rapidly,  most neural networks, especially Feedforward Neural Networks (FFNNs) should face the gradient vanishment problem.  In this article, authors proposed one weight initialisation method for FFNNs and Physics-Informed Neural Networks (PINNs). Based on an analysis of the fixed points of the function tanh(ax), this method determines values of $a$ that prevent the saturation of activations during the training progress. In terms of robustness, this method presents a stronger and more efficient performance. In the experiment, verified on MNIST, Fashion MNIST, and CIFAR10 datasets, this method also shows acceptable results compared to the Xavier.
weaknesses: 1. The universality of this method should be improved. For example, more experiments on the feasibility of other neural networks except FNN and PINN.
2. The details of hyperparameters should be mentioned, such as what is the $Threshold$ of FFNN, which training strategy (supervised or unsupervised) of FFNN is used?   
3. FFNN is specifically designed to visualize the trained features, which also should be discussed.
4. The consistency of results should be guaranteed. For example, in Figure 3 (c) and (d), there is a different trend (Xavier tends to equal 
the proposed method), which also should be discussed. In case that after 20 epochs, the performance would be totally different to the presented result.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Layer norm is added to handle the diminishing activation problem. Any reason why you did not compare the performance of the proposed approach with Xavier weight initialization with Layer norm?

2. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean can be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that can lead to tanh saturation.
soundness: 3
strengths: Originality
The paper presents a novel weight initialization method specifically designed for tanh-based neural networks, addressing an understudied area in neural network initialization. This approach is distinct in its use of fixed-point analysis to prevent activation saturation and improve training robustness across network sizes, particularly in the context of Physics-Informed Neural Networks (PINNs). By emphasizing robustness and performance consistency in both traditional classification tasks and PINNs, the paper makes a valuable contribution to the field of weight initialization. The originality is strong, given the lack of prior research focusing on tanh-based initialization methods.

Quality
The paper demonstrates high quality in both theoretical and experimental aspects. The method is grounded in rigorous mathematical analysis, leveraging fixed-point theory to derive conditions that ensure stable activation propagation. The provided lemmas, proofs, and propositions add credibility and depth to the approach.
Experiments are well-designed and span various network configurations, datasets (MNIST, Fashion MNIST, CIFAR-10), and applications (PINNs for solving differential equations). The results consistently show that the proposed method outperforms Xavier initialization, particularly in deeper networks and varying network sizes.

Clarity:
Overall the paper is very well written, with some exceptions mentioned in the weakness section. All the sections in the paper are laid out clearly. The notations are consistent across the paper.
summary: The paper proposes a new method to initialize weights for FCNNs and PINNs with tanh activation. The paper claims that the proposed weight initialization method will not lead to diminishing activations for very deep networks unlike Xavier weight initialization. The paper claims that the proposed weight initialization is robust to network depth and number of units in hidden leayers.
weaknesses: Significance:
The paper compares their proposed weight initialization with Xavier weight initialization for FCNN with tanh activation. Xavier is known to show diminishing gradients and activations problem for deeper networks, but this is solved by using layer normalization. I am therefore considering this work not significant because the problem that the authors are trying to solve does not exist for Xavier + Layer norm and the authors did not do any comparative analysis with and without layer norm.

Other major issue:
1. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean will be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that leads to tanh always saturated.

Meta-review: The paper proposes a novel initialization method for tanh neural networks, based on a fixed-point analysis of the layer.

The reviews are mixed, ranging from marginal rejection to strong acceptance. The reviewers had several technical questions (see below for a detail), but these were addressed in the rebuttal. They were also concerned about some missing baselines (e.g., other initialization techniques, layer normalization), which the authors added in the rebuttal. The only remaining concern is the limited scope of the paper, which focuses on a narrow type of neural network.

Overall, the paper is technically correct; the authors provided a very significant rebuttal which was appreciated by all reviewers who interacted during the rebuttal phase. In addition, they argue correctly that tanh models are still important in several sub-fields, such as PINNs. As a result, I lean towards acceptance.

Decision: Accept (Poster)