Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis
Authors: Hyun woo Lee
Hayoung Choi
Hyunju Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Weight initialization
Signal propagation
Physics informed neural networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: As a neural network's depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal propagation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents a novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, the proposed method aims to determine values of $a$ that mitigate activation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outperforms Xavier initialization methods (with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at https://github.com/1HyunwooLee/Tanh-Init.

Published as a conference paper at ICLR 2025
ROBUST WEIGHT INITIALIZATION FOR TANH NEURAL NETWORKS WITH FIXED POINT ANALYSIS
Hyunwoo Lee', Hayoung Choi Hyunju Kim
KKyunpoook National University, "Korea Institute of Energy Technology [1hw9088, hayoung. choi knu.ac.... kr. Pnjkimekentech. ac. kr
ABSTRACT
As a neural network's ' depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal prop- agation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function tanh(ar), the proposed method aims determine values of a that mitigate acti- vation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outper- forms Xavier initialization methods (with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at hittps///githue com http///yynwwwwweeeeeeee /ann Init
INTRODUCTION
Deep learning has significantly advanced state-of-the-art performance across various domains (Le- Cun et al. 2015 He al., 2016). The expressivity of neural networks increases exponentially with depth, enhancing generalizatt performance (Poole et al., 2016 Raghu et al. 2017). However, deeper networks often face gradient and signal propagation issues (Bengio al. 1993). These chal- lenges have driven the development of effective weight initialization methods designed for various activation functions. Xavier initialization (Glorot & Bengio 2010) prevents saturation in sigmoid and tanh activations, while He initialization (He et al. (0115) stabilizes variance for ReLU networks. Especially in ReLU neural networks, several weight initialization methods have been proposed to mitigate the dying ReLU problem, which hinders signal propagation in deep networks (Lu et al., 2019 Lee al., 2024). However, the best of our knowledge, research on initialization methods that are robust across different sizes of tanh networks is underexplored. Tanh networks commonly employ Xavier initialization (Raissi et al. 2019: Jagtap et al. 2022 Rathore et al. 2024) and are ap- plied in various domains, such as Physics-Informed Neural Networks (PINNs) (Raissi et al. 2019) and Recurrent Neural Networks (RNNs) (Rumelhart et al. 1986), with performance often dependent on model size and initialization randomness (Liu et al. 2022).
The main contribution of this paper is the proposal a simple weight initialization method for FeedForward Neural Networks (FFNNs) with tanh activation function. The proposed method is data-efficient and demonstrates robustness across different network sizes. Moreover, it reduces de- pendency on normalization techniques such Batch Normalization (BN) (loffe] 2015) and Layer Normalization (LN) (Ba 2016). As a result, alleviates the need for extensive hyperparameter tuning, such as selecting the number of hidden layers and units, while also eliminating the com- putational overhead associated with normalization. The theoretical foundation of this approach is based on the fixed point of the function tanh(az). We evaluate the proposed method on two tasks: classification and PINNs. For classification tasks, we assess its performance across various FFNN sizes using standard benchmark datasets. The results demonstrate improved validation accuracy and lower loss compared to Xavier initialization with BN or LN. For PINNs, the method exhibits robust- ness across diverse network sizes and demonstrates effectiveness solving wide range of PDE
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
problems. Notably, for both tasks, the proposed method outperforms Xavier initialization in data efficiency; that is, it achieves improved performance even with limited data. Our main contributions can be summarized follows:
We identify the conditions under which activation values do not vanish as the neural net- work depth increases, using a fixed-point analysis (Section 3.1 and 3.2)
We propose novel weight initialization method for tanh neural networks that is robust across different network sizes and demonstrates high data efficiency (Section 3.2 and 3.3)
We experimentally show that the proposed method is more robust across different network sizes on image benchmarks and PINNs (Section 4
We experimentally show that the proposed method is more data-efficient than Xavier ini- tialization, with without normalization, image benchmarks and PINNs (Section 4
2 RELATED WORKS
The expressivity of neural networks grows exponentially with depth, resulting in improved general- ization performance (Poole et al., 2016 Raghu al. 2017). Proper weight initialization crucial for effectively training deep networks (aax et al. 2014;] Mishkin & Matas 2016). Xavier (Glorot Bengio 2010) and He He et al. (2015) initialization are common initialization methods typically used with tanh and ReLU activation functions, respectively. Several initialization methods have been proposed to improve the training of deep ReLU networks (Lu et al. 2019 Bachlechner et al., 2021 Zhao et al. 2022 Lee et al., 2024b. However, research on weight initialization for tanh-based neural networks remains relatively limited. Nevertheless, tanh networks have gained increasing attention in recent years, particularly in applications such PINNs, where their performance is highly sensitive to initialization randomness.
PINNs have shown promising results in solving forward, inverse, and multiphysics problems in sci-
ence and engineering. (Mao et al., 2020 Shukla et al. 2020; Lu et al.
2021] Karniadakis et al.
2021 Yin et al.] [202]] Bararnia & Esmaeilpour] 2022
Cuomo et al. 2022 Hanna al.) 2022
Hosseini et al.J 2023 Wu et al. [202]] Zhu et al. 2024). PINNs approximate solutions to partial
differential equations (PDEs) using neural networks and are trained by minimizing loss function, typically formulated as a sum of least-squares terms incorporating PDE residuals, boundary condi- tions, and initial conditions. This loss is commonly optimized using gradient- based methods such as Adam (Kingma & Ba 2014). L-BFGS (Liu & Nocedal 1989), combination of both. Univer- sal approximation theorems (Cybenko 1989; Hornik et al.) 1989 Hornik 1991 Park et al., 2021
Guliyev & Ismailov 2018b; [Shen et al.) [ Guliyev & Ismailov 2018a 8a Maiorov & Pinkus,
1999 Yarotsky| 2017; Gripenberg 2003) establish the theoretical capability of neural networks to
approximate analytic PDE solutions. However, PINNs still face challenges in accuracy, stability, computational complexity, and tuning appropriate hyperparameters of loss terms.
To address these challenges, various improved variants of PINNs have been proposed: (1) Self- adaptive lass-balanced PINNs (IbPINNs), which automatically adjust the hyperparameters of loss terms during training (Xiang et al. 2022), (2) Bayesian PINNs (B-PINNs), designed to han- dle forward and inverse nonlinear problems with noisy data (Yang et al.] (021). (3) Rectified PINNs (RPINNs), which incorporate gradient information from numerical solutions obtained via the multigrid method and are specifically designed for solving stationary PDEs (Peng et al.) 2022). (4) Auxiliary PINNs (A-PINNs), developed effectively handle integro-differential equa- tions Yuan et al.) 2022), (5) Conservative PINNs (cPINNs) and Extended PINNs (XPINNs), which employ domain decomposition techniques (Jagtap et al. 2020 Jagtap & Karniadakis 2020). (6) Par- allel PiNNs, designed to reduce the computational cost of cPINNs and XPINNs (Shukia et al., 202T), and (7) Gradient-enhanced PINNS (gPINNs), which incorporate the gradient of the PDE loss term with respect to network inputs YYu et al. 2022).
While these advancements address various challenges in PINNs, activation functions and their ini- tialization strategies remain crucial for achieving optimal performance. The tanh activation function has been shown to perform well in PINNs (Raissi et al. 2019). with detailed experimental results provided in Appendix C.2 Xavier initialization widely employed the standard approach for
tanh networks in existing studies (Jin et al., 202T] [on et al., 2023 Yao et al., 2023] Gnanasam-
bandam al. 2023 Song al. 2024). However, our experimental results indicate that employing
2

---

Published as a conference paper at ICLR 2025
Xavier Weight Initialization
Proposed Weight Initialization
10
Mox-Minn Difference
10-22
Max Min Difference
100
10-30
10-%
10"23
10-A)
4x3001
10-337
10-1
Layer
Layer
(a) Xavier Initialization
(b) Proposed Initialization
Figure The difference between the maximum and minimum activation values at each layer when propagating 3, 000 input samples through 10, 000-layer tanh FFNN, using Xavier initializa- tion (left) and the proposed initialization (right). Experiments were conducted on distinct networks with 10, 000 hidden layers, each having the same number of nodes: 16, 32, 64, or 128.
Xavier initialization leads decreased model performance network size increases. Moreover, performance gains from Batch Normalization or Layer Normalization remain limited, and Xavier initialization exhibits sensitivity the amount of training data, particularly in smaller datasets. Although recent study has proposed an initialization method for PINNs, it depends on transfer leamnng (Tarbiyati & Nemati Saray, (023). To address these limitations, we propose weight ini- tialization method that is robust across different network sizes, achieves high data efficiency, and reduces reliance on both transfer learning and normalization techniques.
PROPOSED WEIGHT INITIALIZATION METHOD
In this section, we discuss the proposed weight initialization method. Section 3.1 introduces the theoretical motivation behind the method. Section 3.2 presents how to derive the initial weight
matrix that satisfies the conditions outlined in Section 3.1 Finally, in Section 3.3 optimal hyperparameter in the proposed method.
we suggest the
3.1 THFORETICAL MOTIVATION
Experimental results in Figure iindicate that when Xavier initialization is employed in FFNNs with tanh activation, the activation values tend to vanish toward zero in deeper layers. This vanishing of activation values can hinder the training process due to a discrepancy between the activation values and the desired output. However, it is not straightforward for preventing this phenomenon. point of tanh(ar) to mitigate this issue. Before giving the theoretical foundations, consider the basic results for a hyperbolic tangent activation function. Recall that if ʃ R + R is a function, then an element x² E R is called a fixed point of if fx=) =x".
theoretically establish the conditions this section, we present a theoretical analysis based on a fixed
Lemma 1. For fixed > 0, define the function &a R + R given as
(a(x) := Lanh(ax). Lanh())))
Then, there exists a fixed point x" of du. Furthermore,
(1) 0 S 1, then dn has unique fixed point x² 0.
(2) if 1, then &a has three distinct fixed points: x" -E. 0, &a such that &a
Proof. The proof is detailed in Appendix A.1
Note that tanh(x) < æ for all r > 0. However, according to Lemma I for a > 1 the behavior of tanh(az) changes. If &a (resp. x < Ea), then tanh(oz) < x (resp. tanh(az) x). At Ea. the equality tanh(ar) = x holds. The following lemma addresses the convergence properties of iteratively applying tanh(ax) for any 0.
3

---

Published as a conference paper at ICLR 2025
Lemma 2. For given initial value Zo 0, define
In+1 (Zn), n = 0,1,
Then ( converges regardless of the positive initial value zu 0. Moreover,
(1) if0 then Vn -
oo.
(2) a 1, then Fa - Ea as n + co.
Proof. The proof is detailed in Appendix A.2
Note that for > 1 and zy > &a > 0, the sequence {ra} satisfies tanh (ax,) > tanh(arw+1) > Ea for all N. Similarly, when < 2g < Ea. the sequence satisfies tanh (Kr,) fanh(ar,+) & for all n E N. Given > and < 0, the sequence converges to -Ea as - oo due to the odd symmetry lanh(ar). According to Lemma when a > 1, the sequence {xn) converges to &o or -E., respectively,
oc, for an arbitrary initial value zo > 0 or zo < 0.
Note that the parameter a in Lemma 2 does not change across all iterations. Propositions and Corollary consider the case where the value of a varies for each iteration.
Proposition 3. Let {an]m= be positive real sequence, i.e., 0 foror all n E N, such that only finitely many elements are greater than 1. Suppose that {Pm 100 m= 1 sequence of functions defined
as for each m E N
4 m Dam Dan
Pay
Then for any
lim Pm (x) = O.
m=+00
Proof. The proof is detailed in Appendix
Corollary 4. Let E > 0 be given. Suppose that {an { be a positive real sequence such that only finitely many elements are lower than + Then for any R {00
lim p (x)) 2 Eite.
Proof. The proof is detailed in Appendix A.4
there exists a sufficiently large N E such that On (resp. an 1+ e) for all n 2 N, then for any E RR (0), (m) + 0 (resp. (4 (2o) 2 E1+e as m - 00. This result implies that if the sequence {Φ 1M is finite, a, for I n S M, where N is arbitrarily chosen index close to M, significantly influence the values of 4 (2o)-
Based on Proposition and Corollary
3.2 THE DERIVATION OF THE PROPOSED WEIGHT INITIALIZATION METHOD
Based on the theoretical motivations discussed in the previous section, we propose weight initial- ization method that satisfies the following conditions the initial forward pass:
It prevents activation values from vanishing towards zero in deep layers.
(ii) It ensures that the distribution of activation values in deep layers is approximately normal.
Notations.
Consider feedforward neural network with L layers. The network processes K training samples, denoted pairs {(x;, K where æ; R is training input and y, R, is its corresponding output. The iterative computation at each layer is defined as follows:
ae = Lanh(W's f-1 b¹) E RR for all f= 1,
where 8 E NexNe is the weight matrix, be E Te is the bias, and tanh(-) an element-wise activation hyperbolic tangent function. Denote W [w]f]-
Signal Propagation Analysis. We present a simplified analysis of signal propagation in FFNNs with the tanh activation function. For notational convenience, it is assumed that all hidden layers, well
4

---

Published as a conference paper at ICLR 2025
Max Difference
107
Max Min Difference
100
05
Layer
Layer
(a) Same dimension
(b) Varying dimensions
Figure 2: The difference between the maximum and minimum activation values each layer when propagating 3, 000 input samples through a 10, 000-layer tanh FFNN, using the proposed initial- ization with a set 0.04, 0.085, 0.15, and 0.5. Network with 10, 000 hidden layers, each with 32 nodes (left), and network with alternating hidden layers of 64 and 32 nodes (right).
the input and output layers, have dimension of i.e., N2 for all Given arbitrary input vector æ = (Z1,.... In), the first layer activation x: = tanh ('rr) can be expressed component- wise
Way I2
2t tanh (w + ... =.. = tanh
2i
for = 1, 1,...,7.
For the (k + 1)-th layer this expression can be generalized as
a tanh (a.-11a) =f). where att
k++
for i = 1,...,
Equation I follows the form of tanh(ar). as discussed in Section 3.2 According Lemma
when 1. for an arbitrary initial value > 0 or < 0, the sequence {ør defined by Jk+1 tanh(ark)) converges to & or -aa respectively, as ht cc. This result indicates that the sequence converges to the fixed point &a regardless of the initial value zp. From the perspective of signal propagation in tanh-based FFNNs, this ensures that the activation values do not vanish as the network depth increases. Furthermore, by Proposition 3 if af < for all N < k: S L, where N is an arbitrarily chosen index sufficiently close to L. the value of x! approaches zero. Therefore, to satisfy condition (i), of remains close to 1, and the inequality af < 1 does not hold for all N L.
The proposed initial weight matrix is defined as W D E N where D if = j (mod Nt-1), and 0 otherwise (Examples of D' are provided in Appendix D). The noise matrix Z is drawn from (0, where o₂ is set to a a/VNe-1 with 0.085. Then ++1 follows the distribution:
Proposed Weight Initialization.
22
a ~
0,
+ 0
(2)
follows a Gaussian distribution with a mean of Additionally, r becomes small relative other elements in x the variance of the distribution in (2) increases. Consequently, the probability that the absolute value of *A+11 exceeds that of z becomes higher. Figure [
According to Equation
(b) shows that activation values maintain consistent scales in deeper layers.
3.3 PREVENTING ACTIVATION SATURATION VIA APPROPRIATE o2 TUNING
In this section, we determine the appropriate value of a in o2 = a/ N, that satisfies condi-
tion (ii). Condition (ii) is motivated by normalization methods (Ioffe 2015 Ba 2016). Firstly,

---

Published as a conference paper at ICLR 2025
0,0003
0.015
0.3
12000
-2006
10000
0003
140
3000
3500
0003
3005
0230
750
500
2003
Figure 3: The activation values in the 1000th layer, with 32 nodes per hidden layer, were analyzed using the proposed weight initialization method with o, values 0.0003, 0.015, 0.3, and The analysis was conducted on 3, 000 input samples uniformly distributed within the range [-1, 1].
we experimentally investigated the impact of d= on the scale of the activation values. As shown in Figure 2 increasing a/ v N broadens the activation range in each layer, while decreasing 0g narrows it.
When d₂ is Large. Setting as to a large value can lead to saturation. If og is too large, Equation (2) implies that the likelihood of a deviating significantly from increases. This increases the likeli- hood of activation values being bounded by {i+t in sufficiently deep layers, as stated in Corollary Consequently, deeper layers, activation values are less likely to approach zero and tend to saturate toward specific values. Please refer to the Figure 3 for the cases where oN 0.3 and 3.
When o₂ Small. o. is too small, Equation implies that the distribution of at has a standard deviation close to zero. Consequently, x; can be approximated as the result applying tanh(x) to o; > repeatedly a finite number of iterations, k. Since lanh' (xx) decreases values resulting from finite iterations eventually saturate. Plese refer to the Figure 3 when 0z 0.0003.
x 0, the
For these reasons, experimentally determined optimal 0₂ that avoids being excessively large small. As shown in Figure Ø= = 0.015 maintains an approximately normal activation distribution without collapse. Additional experimental results are provided in Appendix A.5 Considering the number of hidden layer nodes, we set ag a V with 0.085. Experimental results for solving the Burgers' equation using PINNs with varying are provided in Appendix C.3
EXPERIMENTS
In this section, present series of experiments to evaluate the proposed weight initialization method. In Section 4.1 we evaluate the performance of tanh FFNNs on benchmark classification
datasets. In Section 422 apply Physics-Informed Neural Networks solve PDEs. Both experi- ments assess the proposed method's robustness to network size and data efficiency.
Table 1: Validation accuracy and loss are presented for FFNNs with varying numbers of
nodes (2, 32, 128, 512), each with 20 hidden layers using tanh activation function. All models were trained for 20 epochs, and the highest average accuracy and lowest average loss, computed over 10 runs, are presented. The better-performing method is highlighted in bold when comparing different initialization methods under the same experimental settings.
Nodes
8 Nodes
32 Nodes
128 Nodes
512 Nodes
Method
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Xavier
49,78
1.632
68
0.958
91.67
0.277
95.45
0.154
97.35
0.087
MNIST
Proposed
62.82
1.185
77.95
0.706
92.51
0.255
96.12
0.134
97.96
FMNIST
Xavier 42.89
1.559
68.55
0.890
81.03
0.533
86.20
0.389
88.28
0.331
Proposed
$1.65
1.324
71.31
0.777
83,06
0.475
87.12
0.359
88.59
0.323
CIFAR-10
Xavier 32.82
1.921
43.51
1.608
48.6
1.473
47.58
1,510
$1.71
1.369
Proposed
38.16
1.780
47.04
1,505
48,80
1.463
48.51
1.471
52.21
1.359
Xavier
10.87
4.065
18.53
3.619
23.71
3.301
23,83
3.324
17.72
3,672
CIFAR-100
Proposed
15.22
3,818
23.07
3,350
24.93
3,237
24.91
3.240
22.80
3,435

---

Published as a conference paper at ICLR 2025
Table 2: Validation accuracy and loss are presented for FFNNS with varying numbers of lay- ers (3, 10, 50, 100), each with 64 number of nodes using the tanh activation function. All models
were trained for 40 epochs, and the highest average accuracy and lowest average loss, computed over 10 runs, are presented.
Dataset
Method
3 Layers
10 Layers
Layers
100 Layers
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
MNIST
Xavier
95.98
0.130
96.55
0.112
96.57
0.123
94.08
0.194
Proposed
96.32
0.123
97.04
0.102
96.72
0,109
96.06
0.132
Xavier
85.91
0.401
88,73
0.319
87.72
0,344
83.41
0,463
FMNIST
Proposed
86.51
0.379
89.42
88.51
0.324
0,382
Xavier
42.91
1.643
48.39
47.87
1.474
46.71
1,503
CIFAR-10
Proposed
45.05
1,588
48,41
1.458
48.71
1.461
48.96
1.437
Xavier
19.10
3.628
22.73
3,400
24.27
3,283
20.32
3.515
CIFAR-100
Proposed
19.30
3.609
23.83
3.309
25.07
3,190
24.41
3.234
4.1 CLASSIFICATION TASK
Experimental Setting. To evaluate the effectiveness of the proposed weight initialization method, we conduct experiments on the MNIST, Fashion MNIST (FMNIST). CIFAR-10, and CIFAR- 100 (Krizhevsky & Hinton 2009) datasets with the Adam optimizer. All experiments with a batch size of 64 and a learning rate of 0.0001. Fifteen percent of the dataset is allocated for validation. The experiments are implemented in TensorFlow without skip connections and learning rate decay in any of the experiments.
performed
Width Independence in Classification Task. We evaluate the proposed weight initialization method in training tanh FFNNs, focusing on its robustness across different network widths. Five tanh FFNNs are designed, each with 20 hidden layers, and with 2, 8, 32, 128, and 512 nodes per hidden layer, respectively. In Table for the MNIST, Fashion MNIST and CIFAR-10 datasets, the network with 512 nodes achieves the highest accuracy and lowest loss when the proposed method is employed. However, for the CIFAR-100 dataset, the network with 32 nodes yields the highest accuracy and lowest loss when employing the proposed method. In summary, the proposed method demonstrates robustness across different network widths in tanh FFNNs. Detailed experimental results are provided in Appendix B.T
Depth Independence in Classification Task. The expressivity of neural networks is known in- crease exponentially with depth, leading to improved generalization performance (Poole et al., 2016 Raghu et al., 2017). To evaluate the robustness of the proposed weight initialization method across different network depths, we conduct experiments on deep FFNNs with tanh activation functions. Specifically, we construct four tanh FFNNS, each with 64 nodes per hidden layer and 3, 10, 50, and 100 hidden layers. respectively. In Table 2 for both the MNIST and Fashion MNIST datasets, the network with 10 hidden layers achieves the highest accuracy and lowest loss when our proposed method is employed. Both initialization methods exhibit lower performance in networks with 3 lay- ers compared to those with more layers. Moreover, for more complex datasets such as CIFAR- 10 and CIFAR-100 the proposed method demonstrated improved performance when training deeper networks.
044
033
Propaued
630
Savor
(a) MNIST
Fashion MNIST
(c) CIFARIO
(d) CIFARI00
Figure 4: Validation accuracy for tanh FFNN with 50 hidden layers (32 nodes each). Xavier BN and Xavier LN represent Xavier initialization with Batch Normalization or Layer Normalization applied every 5 layers, respectively.

---

Published as a conference paper at ICLR 2025
Table 3: Validation accuracy and loss for a 10-layer FFNN (64 nodes per layer) trained on datasets containing 10, 20, 30, 50, and 100 samples. Results show the highest average accuracy and lowest average loss over 5 after 100 epochs.
Dataset
Method
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Xavier
31.13
45.05
1.771
58.45
MNIST
Xavier BN
22.46
2.267
31.73
2.053
37.13
2.042
39.78
1.944
57.51
1.464
Proposod
37.32
2.204
46.79
1,656
48.60
1.645
61.54
1.131
68.44
1,433
Xavier
53.86
FMNIST
Xavier BN
Xavier LN
34.94
35.44
2.362
2.136
37.90
38.58
1.925
53.27
40.16
1.819
53.93
1.728
59.78
1.073
1.237
Proposed
37.31
2,217
49.25
1,681
55.19
1.372
66.14
1.057
67.58
0.914
Normalization Methods. Xavier initialization is known to cause vanishing gradients and activation problems in deeper networks. These issues can be mitigated by applying Batch Normalization (BN) or Layer Normalization (LN) in the network. Therefore, we compare the proposed method with Xavier, Xavier with BN, and Xavier with LN. To evaluate the effectiveness of normalization, we conducted experiments using sufficiently deep neural network with 50 hidden layers. As shown in Figure for datasets with relatively fewer features, such as MNIST and FMNIST, Xavier with nor- malization converges faster than Xavier. However, for feature-rich datasets such as CIFAR-10 and CIFAR-100, the accuracy of Xavier with normalization is lower than that of Xavier. Normalization typically incurs 30% computational overhead, and additional hyperparameter tuning required to determine which layers should apply normalization. In contrast, the proposed method achieves the best performance across all datasets without requiring normalization.
Data Efficiency in Classification Task. Based on the results in Table 2 we evaluated data effi- ciency on network with 50 hidden layers, each containing 64 nodes, where Xavier exhibited strong performance. As shown in Table 3, the highest average accuracy and lowest average loss over runs after 100 epochs are presented for datasets containing 10, 20, 30, 50, and 100 samples. The proposed method achieved the best performance across all sample sizes.
Non-uniform Hidden Layer Dimensions. We evaluate the performance of the proposed initializa- tion in networks where hidden layer dimensions are not uniform. As shown in Figure [ the network consists of hidden layers, where the number of nodes alternates between and 16 each layer. The proposed method demonstrates improved performance in terms of both loss and accuracy across all epochs on the MNIST and CIFAR- 10 datasets. Additionally, Appendix presents experiments on networks with larger variations in the number of nodes. Motivated by these results, Appendix B.2 further explores autoencoders with significant differences hidden layer dimensions.
10
Velcgaton Accurace
Vallsatoo Logs
3.00
2.55
Proposed
3.5
10
3 10
(a) MNIST Accuracy
(b) MNIST Loss
(c) CIFARIO Accuracy
(d) CIFARIO Loss
Validation accuracy and loss for a tanh FENN with hidden layers, where the number of nodes alternates between 32 and 16 across layers, repeated 30 times. The model trained for 20 epochs on the MNIST and CIFAR- 10 datasets.
Figure 5:
4.2 PIYYSICS-INFORMED NEURAL NETWORKS
Xavier initialization is the most commonly employed method for training PINNs (Jin et al. 2021 Son et al. 2023 Yao et al. 2023 Gnanasambandam al.) 2023). In this section, experimentally demonstrate that the proposed method is more robust across different network sizes and achieves higher data efficiency compared to Xavier initialization with without normalization methods.
8

---

Published as a conference paper at ICLR 2025
Experimental Setting. All experiments on Physics-Informed Neural Networks (PINNs) use full- batch training with a learning rate of 0.001. In this section, we solve the Allen-Cahn, Burgers, Diffusion, and Poisson equations using tanh FFNN-based PINN with 20,000 collocation points. Details on the PDEs are provided in Appendix C.1
Network Size Independence in PINNs. We construct eight tanh FFNNs, each with 16 nodes per hidden layer and 5, 10, 20, 30, 40, 50, 60, or 80 hidden layers. As shown in Table 4 for the Allen- Cahn and Burgers' equations, Xavier+BN and Xavier+LN achieve the lowest loss at a network depth of 30. However, their loss gradually increases as depth grows. In contrast, the proposed method achieves the lowest loss at depths of 50 and 60, respectively, maintaining strong learning performance even in deeper networks. For the Diffusion and Poisson equations, Xavier+LN achieves the lowest loss at depths of 5 and 10, respectively. While all methods exhibit increasing loss as network depth increases, the proposed method consistently maintains lower loss in deeper networks. Similar trends are observed with 32 nodes. Across all tested network sizes and PDEs, the proposed method consistently achieves the lowest loss. The proposed method eliminates the computational overhead and hyperparameter tuning required for normalization methods.
Table 4: A PINN loss presented for FFNNS with varying numbers of lay- ers (5, 10, 20, 30, 40, 50, 60, 80) using the tanh activation function. The top table shows results with 16 nodes per layer, and the bottom table shows results with 32 nodes per layer. All models were trained for 300 iterations using Adam and 300 iterations using L-BFGS. The median PINN loss at the final iteration for the Burgers, Allen-Cahn, Diffusion, and Poisson equations, computed over runs, is presented.
Allen-Cahn (16 Nodes)
5
10
20
30
40
60
80
Xavier Xavier BN
9.58c-04
8.166-04
7.6le-44
1.06x-03
I.le-03
1.24e-03
3.55c-03
1.81c-03
144:-00
8.17c-04
8.56c-04
7.07s-04
7,77c404
8.876-04
9.11c-04
2.15c-03
Xavier + 1.N
6.29c-01
1.77c-03
6 6.
1.27e-03
1.82c-03
6.65e-01
3.29e-01
5.86e-01
Proposed
9,21e-04
7,29c-04
5.76e-04
5.29c-04
5,37e404
4.03e-04
4.73e-04
5.77e-04
Burgers (16 Nodes)
5
10
20
30
40
50
60
80
Xavier
6.97-003
1.11e-02
7.9c-03
9.71e-03
2.45c-02
2.65c-02
6.5c-02
5.71e-02
Xavier + BN
8.07e-03
7.72c-03
6.24e-03
1.70a-02
1.50c-02
1.85e-02
2.1c-022
6.84e-02
Xavier LN
3.89c-02
1.88c-02
9.48c-03
9.28c-03
2.46c-02
3.30c-02
6.1c-02
4.42a-02
Proposed
6.19c-03
5.08c-03
5.28e-03
9.31e-04
3.56c-03
8.27e-04
3.43e-04
2.05e-03
Diffusion (16 Nodes)
5
10
20
30
40
60
80
Xavier Xavier BN
2.52c-03
4,82c-03
9.69c-03
1.33-002
2.08c-02
1 1.50c-02
2.920-02
1.24<-02
2.89c-03
5.77e-03
1.05c-02
9.65e-03
2.76c-02
1.0e-02
9.07c-03
1.4e-002
Xavier + LN
1.72e-03
6.10e
8.04e 03
9.48c 03
0.14e-02
7.59e-03
2.05e.02
0.21e
Proposed
9.14c-04
2.59c-03
2.40c-03
L01c-03
1.97c-03
1.21e-03
1.12-00
1.91e43
Poisson (16 Nodes)
5
10
20
30
40
50
60
80
Xavier
1.52--02
2.87c-02
1228-011
9.82e-02
1.15c401
1.37c-01
1826-01
2.55e-01
Xavier + BN
1.62s-02
2.02c-02
8.72o-02
1.120-01
2,45c-01
9.85c-02
1.00-01
1.34-01
Xavier I.N
5.39c-01
4.40e-02
1.34c-01
3.91
2.52c+02
2.58
9.79e402
N/A
Proposed
1.37e-02
1.70c-02
4.62e-02
2.4e-022
3.75c-02
4.03e-02
6.07e-02
6.01e-02
Allen-Cale (32 Nudes)
5
10
20
30
40
50
60
80
Xavier
3.13c-01
5.03e-02
3.64c-03
2.37e-03
4.03e-03
5.27c-03
1.73e-02
6.9c-01
Xavier + BN
4.05e-01
8.85c-04
0.4
7.82c 04
9.97e-04
6.80e 04
9.34e
6.94e
Xavier LN Proposed
3.31e-01
2.10e-03
0.99e-04
6.71e-04
1.49e-03
1.29k-03
3.31e-02
6.93e-01
1.4e-03
6.92e-04
5,34e-04
4.26e-04
3.31e-04
3.52e-04
3.85e-04
5.96e-04
Burgers (32 Nodes)
5
10
20
30
40
50
60
80
Xavier
1.12e-02
3.53e
2.72e 03
181e-03
7.60e
8.56e 03
9.86e
1.66e 01
Xavier BN
5.88c-03
1.04c-03
1.79e-03
2.80e-03
5.95e-03
3.666-022
6.60e-02
666e-01
Xavier + LN
4.31e-02
1.21e-02
1888 03
7.22c 03
5.54e 03
8.46e 03
9.04e 03
4.86e 02
Proposed
4.14c-03
4.11e-03
1,58e-03
1.29e-03
7.96c-04
5.85c-004
9.800-04
1.47e-13
Diffusion (32 Nodes)
5
10
20
30
40
50
60
80
Xavier
1.69:003
6.85c-03
7.63e-03
4.50e-03
8.98e-03
5.67e-03
6.33e-01
1.59
Xavier + BN
1.68-003
2.66c-03
1.08c-02
6.0s-033
8. 8.58c-03
6.60e-03
5.66e-02
1.69x402
Xavier I.N
8.166-04
2.85c-03
8.46e--3
4.57e-03
9.40e-03
1.04e-02
2.42e-01
1.67++02
Proposed
2.89c-04
8.03c-04
5.25e-04
5.07e-04
5,33e-04
6.17e-04
9.80e-04
1.53e-03
Poisson (32 Nodes)
5
10
20
30
50
60
80
Xavier
109c-02
1.33e-02
3.132-02
7.69-022
6.72e-02
8.90e-02
9 9.68e+02
440x402
Xavier BN Xavier + LN
1.14z:02
1.47e-02
2.68c-02
3.55e-02
8.25e.02
.997e-02
4.50e-02
7.75e-01
2.36c-02
18c-220
3.07c-02
3.85c-01
1.0
4.69
2.60
6.14
Proposed
9.63e-103
8.29c-03
1.41e-02
188e-422
1.65e-022
188e-002
1.73e-02
3.59e-022
9

---

Published as a conference paper at ICLR 2025
Xavier Proposed
Prapssed
Mean Absorute Error
Mean Absotute Enror
1011
19.
40000
Points
(a) Burgers Equation
(b) Diffusion Equation
Figure 6: Mean absolute error between the exact solution and PINN- PINN-predicted solution with varying numbers of collocation points. The FFNN has 30 hidden layers (32 nodes each) and is trained for 300 iterations using Adam followed by 300 iterations using L-BFGS. The results averaged over 5 experiments.
Data Efficiency in PINNs. Based on the results in Table we evaluate data efficiency network with 30 hidden layers, each containing 32 nodes, where Xavier initialization achieved the lowest PINN loss. As shown in Figure 6 for the Burgers equation, the Mean Absolute Error (MAE) of the proposed initialization differs significantly from that of Xavier initialization across varying numbers of collocation points. In contrast, for the Diffusion equation, the difference in MAE between the two methods becomes more pronounced when the number collocation points exceeds 20, 000, Addi- tionally, Figure 7illustrate that increasing the number of collocation points enables PINNs with the proposed initialization to predict solutions with lower absolute error. For detailed experiments on the Burgers equation, please refer to Appendix C.4
3000
10000
20000
3.5
43
400
Figure 7: Absolute error between exact solution the PINN-predicted solution for the Diffu- sion equation with varying numbers of collocation points (3000, 10000, 20000, 50000) using (upper row) Xavier and (lower row) the proposed initialization. The FFNN has 30 hidden layers (32 nodes each) and is trained for 300 iterations using Adam followed by 300 iterations using L-BFGS. The color bar ranges from 0 to 0.05, with values outside this range shown white.
5 CONCLUSION
In this study, proposed novel weight initialization method for tanh neural networks, baesed on a theoretical analysis of fixed points of tanh(az) function. The proposed method is experimentally demonstrated to achieve robustness variations in network size without normalization methods and to exhibit improved data efficiency. Therefore, the proposed weight initialization method reduces the time and effort required for training neural networks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work of Hyunwoo Lee and Hayoung Choi was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea govermment (MSIT) (No. 1P22R1A51 and RS- 2024-00342939). The work of Hyunju Kim was supported by the funds of the Open R&D program of Korea Electric Power Corporation.NNo-R2555..
REFERENCES
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv16607.06450 2016.
Thomas Bachlechner. Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence. pp. 1352-1361. PMIR, 2021.
Hassan Bararnia and Mehdi Esuaeilpour. On the application of physics informed neural networks to solve boundary layer thermal-fluid problems. International Communications in Heat and Mass Transfer, 132: 10580,, 2022.
Yoshua Bengio, Paolo Frasconi, and Patrice Simard. The problem of learning long-term dependen- cies in recurrent networks. In IEEE International Conference Neural Networks, pp. 1183- 1188. IEEE, 1993.
Amm Clevert. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.0720 2015.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo. Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informat neural networks: Where we and What's next. Journal of Scientific Computing, 92(3):88, 2022.
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 204):303-314, 1989.
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
Raghav Gnanasambandam, Bo Shen, Jihoon Chung, Xubo Yue, and Zhenyu Kong. Self-scalable tanh (stan): Multi-scale solutions for physies-informed neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12):15588-156)) 2023.
G. Gripenberg. Approximation by neural networks with a bounded number of nodes each level. Journal of Approximation Theory, 122(2):260-22) 266, 2003.
Namig J. Guliyev and Vugar E. Ismailov. Approximation capability of two hidden layer feedforward neural networks with fixed weights. Neurocomputing, 316:262- 269, 201 8a.
Namig J. Guliyev and Vugar E. Ismailov. On the approximation by single hidden layer feedforward neural networks with fixed weights. Neural Networks, 98:296-3000 2018b.
John M. Hanna, José V. Aguado, Sebastien -annaa,,, Ramzi Askri, and Domenico Borzac- chiello. Residual-based adaptivity for two-phase flow simulation in porous media using physics- informed neural networks. Computer Methods in Applied Mechanics and Engineering, 396: 115100, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1036-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, (2):251-257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359-366, 1989.
Vahid Reza Hosseini, Abbasali Abouei Mehrizi, Afsin Gungor, and Hamid Hassanzadeh Afrouzi. Application of a physics-informed neural network to solve the steady-state bratu equation arising from solid biofuel combustion theory. Fuel, 332: 125908, 2023.
Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covari- shift. arXiv preprint arxiv:/502.033667 2015.
Ameya Jagtap and George Em Karniadakis. Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5), 2020.
Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative thysics-informet neural networks discrete domains for conservation laws: Applications forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365: 13028, 2020.
Ameya D Jagtap, Zhiping Mao, Nikolaus Adams, and George Em Karniadakis. Physics-informed neural networks for inverse problems in supersonic flows. Journal of Computational Physies, 466:1 2022.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 2021.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.694116 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report, Citeseer, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep leaming. Nature, 521(7553):4364444 2015.
Hyunwoo Lee, Yunho Kim, Seung Yeop Yang, and Hayoung Choi. Improved weight initialization for deep and narrow feedforward neural network. Neurul Networks, 176: 106362, 2024.
Dong C Liu and Jorge Nocedal, On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1):503-528, 1989.
Xu Liu, Xiaoya Zhang, Wei Peng, Weien Zhou, and Wen Yao. A novel meta-learning initializa- tion method for physics-infone innsice-informed neural networks. Neural Computing and Applications, 34(17): 14511-14534, 2022.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis, Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:1903.06333 2019.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 2021.
Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neu- rocomputing, 25(1) 25(1):81-91, 1999.
Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis. informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 1 12789, 2020.
12

---

Published as a conference paper at ICLR 2025
Dmytro Mishkin and Jiri Matas. All you need is good init. In International Conference Learning Representations, 2016.
Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. Minimum width for universal approximation. In International Conference on Learning Representations, 2021.
Pai Peng. Jiangong Pan, Hui Xu, and Xinlong Feng. RPINNS: Rectified-physies informed neural networks for solving stationary partial differential equations. Computers & Fluids, 245:105583, 2022.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the ex- pressive power of deep neural networks. In International Conference on Machine Learning, pp- 2847-2854, 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Prajit Ramachandran. Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training PINNs: loss landscape perspective. International Conference on Machine Learning. 2024.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 32366088)5533-536 1986.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representa- tions, 2014.
Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of relu networks in terms of width and depth. Journal de Mathématiques Pures Appliquées, 157:101-135, 2022.
Khemraj Shukla, Patricio Clark Di Leoni, James Blackshire, Daniel Sparkman, and George Em Karniadakis. Physics-informed neural network for ultrasound nondestructive quantification of surface breaking cracks. Journal of Nondestructive Evaluation, 39:1-20, 2020.
Khemraj Shukla, Ameya D Jagtap, and George Em Karniadakis. Parallel physics-informed neural networks via domain decomposition. Journal of Computational Physics, 447:110683, 2021.
Hwijae Son, Sung Woong Cho, and Hyung Ju Hwang. Enhanced physics-informed neural networks with augmented lagrangian relaxation method (AL-PINNs). Neurocomputing, 548: 16424, 2023.
Yanjie Song, He Wang, He Yang, Maria Luisa Taccari, and Xiaohui Chen. Loss-attentional physics- informed neural networks. Journal of Computational Physies, 501:112781, 2024.
Homayoon Tarbiyati and Behzad Nemati Saray. Weight initialization algorithm for physics- informed neural networks using finite differences. Engineering with Computers, pp. 1-17, 2023.
Zhiyong Wu, Huan Wang, Chang He, Bing Zhang, Tao Xu, and Qinglin Chen. The application of physics-informed machine learning in multiphysics modeling in chemical engineering. Industrial & Engineering Chemistry Research, 62, 2023.
Zixue Xiang, Wei Peng, Xu Liu, and Wen Yao. Self-adaptive loss balanced physics-informett neural networks. Neurocomputing, 496: 1 1-34, 2022.
Liu Yang, Xuhui Meng, and George Em Kamiadakis. B-PINNs: Bayesian physics-informe neu- ral networks for forward and inverse pde problems with noisy data. Journal of Computational Physics, 10991, 2021.
13

---

Published as a conference paper at ICLR 2025
Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, and Jun Zhu. Multiadam: Parameter wise scale-invariant optimizer for multiscale training of physics-informen neural net- works. In International Conference on Machine Learning, pp. 39702-39722 PMLR, 2023.
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94: 103-114, 2017.
Minglang Yin, Xiaoning Zheng, Jay D Humphrey, and George Em Karniadakis. Non-invasive infer- ence of thrombus material properties with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 375: 13603, 2021.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Kamiadakis. Gradient-enhanced physics-informet neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering. 393:1 14823, 2022.
Lei Yuan. Yi-Qing Ni, Xiang- Yun Deng, and Shuo Hao. A-PINN: Auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differennial equations. Journal of Computational Physics, 462:111266 2022.
Jiawei Zhao, Florian Tobias Schaefer, and Anima Anandkumar. 7ero initialization: Initializing
neural networks with only zeros and ones. Transactions on Machine Learning Research, 2022.
ISSN 2835-8856.
Jing ang Zhu, Yiheng Xue, and Zishun Liu. A transfer learning enhanced physics-informed neural network for parameter identification in soft materials. Applied Mathematics and Mechanics, 45 (10):1685-1704 2024.
14

---

Published as a conference paper at ICLR 2025
A ANALYSIS OF SIGNAL PROPAGATION
In this section, we provide proofs for the statements presented in Section 3.1 Each proof follows from the fundamental properties of tanh(ox) and analytical derivations. An empirical analysis is performed on the activation distribution for normally and beta-distributed input data.
A.1 PROOF OF LEMMA I
Proof. Define g(x) = tanh(ax) - x. Since g(z) is continuous, and g > O, ((M) < 0 for large real number M, the Intermediate Value Theorem guarantees the existence of point such that ((x) =
First, consider the case < a S Since < a S 1, the derivative g'(x) a sech (ax) satisfies -1 S gxxx) I 1 < for all z. Hence, g(x) is strictly decreasing and therefore g has a unique root. At æ = 0, (0) = tanh(a 0) = O. Hence, x 0 is the unique fixed point.
We consider the case 1. For < << 1, tanhhaa)) -r 2 (a- 1)x. Since >> 1, tanh(ax))
On the other hand, since (tanh(ax) < 1 for all x,
lim [-1 x| S lim [tanh(ar) x| S lim - x²). 2
2-000
00000
By the squeeze theorem, sim (tanh(ar) x] = -c. By the intermediate value theorem, therefore, there exists at least one
such that tanh(ar) x. To establish the unique- investigate the derivative g g'(x) aech (ax) 1. We straightforward to see that g' > sec " d sec (+a)) and ((xx) < 0 in R (-d sec sec--e- ( a sec (2)) i.e. g(x) has exactly two fixed points. Because g(z) is an odd function, if xx is solution, then -x" is also solution. Thus, for > 1, there exists unique positive fixed point if z > 0 and unique negative fixed point if < O.
ness of the positive fixed point,
find the critical points to be x = + see
se (2)))
A.2 PROOF OF LEMMA 2
Proof. (1) Since (tanh(ar)) sech < for all x > 0, it holds that En+1 (xx,) Fn for all n E N. Thus the sequence {xa | is decreasing. Since In > O for all n E N, by the monotone convergence theorem, it converges the fixed point x²' 0.
(2) Let xp < Ea- Since d'(x) is decreasing for x 2 0, with ø' (0) > and &a is the unique fixed point for 0, it holds that Fi #n+1 <a for all n E N. Thus, by the monotone convergence theorem, the sequence converges to the fixed point &a. The proof is similar when To > Ea. By the monotone convergence thcorem, the sequence also converges the fixed point &a.
A.3 PROOF OF PROPOSITION 3
Proof. Set N max(nia, a, > 1). Define the sequences {b, j and {c, Ja such that b, On an for n N, with b, and e, for N. Suppose that {2m) 100 and 1sc are sequences of functions defined as for each m E N
Φ Dbm Dbm--
ØbL
Th dem Dem-1 de
Then the inequality Im < 2m have limm-+o &m O and lim Im = O. Therefore, the Squeeze Theorem guarantees that
< im holds for all m. By Lemma for any x 2 0, we
limm pm 0.
A.4 PROOF OF COROLLARY
Proof. Set N max On 11 c). Define the sequence {b, 1o such that b, an for n < N, and b, = + E for n > N. The remainder of the proof is analogous to the proof of Proposition 3
15

---

Published as a conference paper at ICLR 2025
A.5
ACTIVATION DISTRIBUTION BASED ON INPUT DATA DISTRIBUTION
0.0003
230
40000
2000
3200
3000
280
2000
C35037 935
0.015
14330
12300
I003
$300
1000
3000
X60
252
120
1300
100 815 000 C31 XO 335 050 070
Figure 8: The activation values in 1000 layer, with 32 nodes per hidden layer, were analyzed using the proposed weight initialization method with ØN values of 0.0003, 0.015, 0.3, and 3. The upper row shows results for 3000 input samples drawn from standard normal distribution, while the lower row presents results for samples drawn from Beta distribution with parameters 2.0 and 5.0.
B CLASSIFICATION TASKS
In this section, we present experimental results for the benchmark classification datasets MNIST, Fashion- MNIST, CIFAR-10, and CIFAR-100 when employing the proposed weight initialization method.
B. WIDTH INDEPENDENCE IN CLASSIFICATION TASKS
Here, we present the detailed experimental results of Table 1 Validation accuracy and loss for MNIST, Fashion-MNIST CIFAR-I0, and CIFAR-100 are presented for tanh FFNNs with varying numbers of nodes (2, 8, 32, and 128), each with 20 hidden layers.
Number of nodes:
Number of nodes: 8
Number of nodes: 32
Number of nodes: 128
Valibaton escurace
Pre
0.5
0.50
0.25
0.3
0.6
0.00
0.2
0.3
0.70
0.08
5.3
0.05
0.85
Epochs
Number of nodes; 2
Number of nodes: 8
Number of nodes: 32
Number of nodes: 128
1.4
222
38ldation 1003
3.75
0.40
2.8
3.50
2.8
3.25
1,20
0.6
12
0.75
0.4
6.0
0
15
10
15
13
20
Epochs
Epochs
Figure 9: Validation accuracy and loss are presented for tanh FFNNs with varying numbers of nodes (2, 8, 32, and 128), each with hidden layers. All models were trained for 20 epochs on the MNIST dataset, with 10 different random seeds.
16

---

Published as a conference paper at ICLR 2025
Number nodes:
Number nodes:
Number nodes: 32
Number nodes:
Sodker
valbaton accUracy
033
0.6
0.75
0.3
0.05
0.00
63
0.50
6.1
0.3
0.55
0.09
Ebochs
Epochs
Number nodes:
Number of nodes:
Number of nodes: 32
Number of nodes: 128
2.2
2.0
Velibato 005
2.0
1.
3.8
1.6
1.0
1.4
0.45
1.0
0.40
12
0:35
10
Epachs
Figure 10:
Validation accuracy and loss are presented for tanh FFNNS with varying numbers of nodes (2, 8, 32, and 128), each with 20 hidden layers. All models were trained for 20 epochs on the Fashion MNIST dataset, with 10 different random seeds.
Number of nodes: 2
Number of nodes:
Number of nodes: 32
Number of nodes: 128
0.455
0.42
Valcaton acciaa
0,100
0.43
8/415
8,400
0.45
0.25
0.375
0.45
0.20
Kivier
0.42
0.40
10 Epochs
15
10
Epochs
Epachs
Number of nodes:
Number of nodes: 8
Number of nodes: 32
Number of nodes: 128
2.0
1.70
Velidston 002
2.2
1.65
2.2
1.8
3.0
2.0
1.
1.55
13-
3.9
1.50
1.5
1.8
0
15
10 Epochs
15
15
20
Epachs
Epochs
Figure 11: Validation accuracy and loss are presented for tanh FFNNs with varying numbers of nodes (2, 8, 32, and 128), each with 20 hidden layers. All models were trained for 20 epochs on the CIFAR- 10 dataset. with 10 different random seeds.
B.2 NON-UNIFORM HIDDEN LAYER DIMENSIONS
Tanh neural networks have been less widely employed compared to ReLU networks due to their higher computational complexity, susceptibility to the vanishing gradient problem, and the supe- rior empirical performance of ReLU in various deep learning tasks. However, the recent success of PINNs using tanh neural networks has renewed interest in their application. This section com- pares the performance of four initialization methods on architectures that are generally challenging to train: (1) tanh activation with Xavier initialization, (2) tanh activation with the proposed initial- ization, (3) ReLU activation with He initialization + BN, and (4) ReLU activation with orthogonal initialization. The experiments performed FFNN and autoencoder models.
17

---

Published as a conference paper at ICLR 2025
Number nodes:
Number nodes:
Number nodes: 32
Number nodes:
0.24
Valtatoon ac0/racy
6.14
0.22
0.24
6.10
0.20
0.14
0.12
hopooe
0.30
Epachs
Epochs
Number of nodes: 2
Number nodes:
Number of nodes: 32
Number of nodes: 128
1.5
Velidatton 000
4.7
5.2
3.
4.0
4.
3.8
3.5
1.8
Epochs
Epochs
Epochs
Figure 12:
Validation accuracy and loss are presented for tanh FFNNS with varying numbers of nodes (2, 8, 32, and 128), each with 20 hidden layers. All models trained for 20 epochs on the CIFAR 100 dataset, with 10 different random seeds.
The experiment is performed using an FFNN with structure in which hidden layers al- ternated between 16 and 4 nodes, repeated 50 times, and trained over 100 epochs. The results shown in Figure 13 (a). Both Xavier and the proposed method successfully train the network, with the proposed method showing overall better performance. Due to its strong performance despite sub- stantial variations hidden layer sizes, the proposed method is further evaluated on autoencoders with large variations in layer sizes, shown in Figure [3 (b).
FFNN.
Autoencoder: The autoencoder architecture consists of an encoder and a decoder, both employing batch normalization and dropout (0.2) for regularization. The encoder compresses the input through layers of 512, 256, and 128 units before mapping to 64-dimensional latent space. The decoder reconstructs the input by symmetrically expanding the latent space through layers of 128, 256, and 512 units, followed by an output layer with sigmoid activation. In Figure (3).. (b), the model is trained on the MNIST dataset with a batch size of 256, while in (c), is trained on the FMNIST dataset with batch size of 512.
kyreer
Xower
(atten Accusay
EN
6.
175
Epochs
Szects
Spoctc
(a) FENN (MNIST)
(b) Autoencoder (MNIST)
(c) Autoencoder (FMNIST)
Figure 13: (a) Validation loss for an FFNN with alternating hidden layers of 16 and 4 nodes, repeated 50 times, comparing four methods: Tanh with Xavier initialization, Tanh with the proposed initial- ization, ReLU with He initialization + BN, and ReLU with orthogonal initialization. (b) Validation loss for autoencoder with encoder-decoder layers of 512, 256, 128, and 64 units, comparing the same four methods. (c) Same as (b), but on the FMNIST dataset.
18

---

Published as a conference paper at ICLR 2025
C PHYSICS-INFORMED NEURAL NETWORKS
In this section, we present additional experiments employing the proposed weight initialization method Physics-Informed Neural Networks (PINNs). Specifically, empirically analyze its relationship with different activation functions (Swish, ELU, Sigmoid, and ReLU) and its impact on various PDE problems (Burgers' equation, Allen- Cahn equation, Poisson equation, and Diffusion equation).
C.I
PDE DETAILS
Here we present the differential equations that we study in experiments.
Allen-Cahn Equation. The diffusion coefficient is set to d = 0.01. The initial condition is defined as u(r, 0) x² (xx) for x [-1, 1], with boundary conditions u(-1,t) = and u(1, 1) I. applied over the time interval E [o, 1]. The Allen-Cahn equation is expressed as
Ou 8 w +u
d
Øt Ox
where u(x, t) represents the solution, d is the diffusion coefficient, and the nonlinear term u² models the phase separation dynamics.
Burgers' Equation. The Burgers' equation, a viscosity coefficient of v = 0.01 employed. The initial condition is given by u(x, 0) sin(xx) for [-1, with boundary conditions u(-1,t) t) = and u(1, t) imposed for t E [0, 1]. The Burgers' equation is expressed as
du Du pu
0t + u 8z
dr2
where u(x, t) is the velocity field, and v is the viscosity coefficient.
Diffusion Equation. The diffusion equation includes time-dependent source term and is defined over the spatial domain x E [-1, 1] and temporal interval f E [0. 1]. The initial condition is specified
as u(x, 0) sin(T.x), with Dirichlet boundary conditions 1(-1,14 t) = and u(1, () = 0. The diffusion equation is expressed as
du
8uu e- (sin(TTx) x (in((()))
Øt
dz
where u(x, is the solution.
Poisson Equation. The Poisson equation is defined over the spatial domain x: E [0, 1] and y E [0, 1].
The Poisson equation is expressed
duu u dr²
8u
+
f(x,)), v).
dy²
where u(x,z) y) is the solution, and fxx v) is the source term given by
f(z v) 2n sin( sin(mry).
C.2 EFFECT OF ACTIVATION FUNCTION ON PINNS
We experimentally demonstrate that when employing the proposed weight initialization method, the absolute error between the exact solution and the PINN-predicted solution is smaller when using the tanh activation compared to ReLU, sigmoid, Swish (Ramachandran et al.) 2017), and ELU (Clev- ert. 2015), as shown in Figure 14 The FFNN consists of 30 hidden layers (32 nodes each) and trained for 300 iterations using Adam, followed by 300 iterations using L-BFGS. For the Burg- ers' and diffusion equations, the PINN with tanh activation achieves the closest approximation to the exact solution. It is well established that PINNs with tanh activation effectively approximate
19

---

Published as a conference paper at ICLR 2025
tanh
relu
sigmoid
swish
elu
Figure 14: Absolute error for (upper row) the Burgers' equation and (lower row) the diffusion equation with varying activation functions. Values outside the color bar range are depicted in white.
solutions to the Burgers' and diffusion equations. Given that the choice of activation function and weight initialization are dependent, this result provides valuable insights into the interaction between initialization methods and activation functions in PINNs..
Figure 15 presents experiments that employ the proposed initialization and Xavier initialization for PDE problems using the Swish activation function, which is widely used in PINNs.
Sower
Testers
Tersttor
(a) Diffusion equation
(b) Poisson equation
(c) Burgers' equation
(d) Allen-Cahn equation
Figure 15: PINN loss for Swish FFNN with (a) 20 hidden layers, each containing 32 nodes, and
(b), (c) hidden layers, each containing nodes, and (d) hidden layers, each containing 32 nodes.
C.3 Tg FOR BURGERS' EQUATION
Ravier
Xever
38"
10"4
Los5
38
Loss
2319
2274
Iterations
Iterations
(a) Same dimension
Varying dimensions
Figure 16: Here, STD refers az. (a) shows the PINN loss for the Burgers' equation, using FFNN with layers and 32 nodes in each hidden layer. (b) shows the PINN loss for an FFNN with 30 layers, where the hidden layers alternate between 64 and 32 nodes, repeated 15 times.
20

---

Published as a conference paper at ICLR 2025
Here, present the PINN loss for the Burgers' equation function of the initialization hyper- parameter dz. As shown in Figure [6 the experimental results demonstrate that o, significantly impacts training. Therefore, selecting an appropriate o₂ is crucial for optimizing performance.
C.4
ABSOLUTE ERROR BURGERS' EQUATION
Figure 17 demonstrates that increasing the number of collocation points allows PINNs with the proposed Initialization to predict solutions with lower absolute enror compared those using Xavier initialization. The proposed method is experimentally demonstrated to achieve faster convergence speed and higher data efficiency.
20000
4.5
420
403
Figure 17: Absolute error between the exact solution and the PINN-predicted solution for the Burg- ers' equation with varying numbers of collocation points (3000, 10000, 20000, 50000) using (upper row) Xavier and (lower row) the proposed initialization. The FFNN has 30 hidden layers (32 nodes each) and is trained for 300 iterations using Adam followed by 300 iterations using L-BFGS.
D EXAMPLES OF THE MATRIX D
An example of D' from the initialization methodology proposed in Section 3.2 is presented, In the heatmap Figure 18 white represents the value 0, while black represents the value
16
36
14
52
Figure 18: Examples of the matrix D R NeNeNN in Section 3.2 with No 4,- (left),
N; = 4, Ne-1 = (middle), and N₂ = 8, N₂-1 = 4 (right), respectively.
21


Reviews:
[{"review_id": "cHn4sT0oel", "reviewer": "Reviewer_79Su", "summary": "The paper introduces a novel weight initialization technique specifically designed for neural networks using the tanh activation function. This technique is evaluated against the well-known Xavier initialization method using benchmark datasets. The experimental results demonstrate that the proposed initialization method enhances the convergence speed of Physics-Informed Neural Networks (PINNs) utilizing the tanh function, showing greater robustness to variations in network size. The findings indicate that the new initialization technique outperforms Xavier initialization in solving various problems related to Partial Differential Equations (PDEs).", "questions": "See above in weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strength of this paper lies in the development of a novel weight initialization technique that facilitates faster convergence and enhances performance in physics-informed neural networks (PINNs) utilizing the tanh activation function.", "weaknesses": "1. The comparison of the proposed weight initialization technique solely with Xavier is insufficient; it should also be experimentally evaluated against other state-of-the-art weight initialization methods.\n2. Using tanh activation function in entire neural network is not good practice that it has the drawback of the vanishing gradients for the very high and very low values of x.\n3. The formulation is more complex than standard methods, which could complicate implementation as shown in Equation (1). \n4. The optimal value of 𝛼 can be highly context-dependent, varying across different architectures, datasets, and tasks, which makes it less universally applicable. Additionally, the choice of 𝛼 can interact with other hyperparameters, such as learning rate and batch size, complicating the overall tuning process during backpropagation, as described in Equation (2).\n5. In Section 4.1, the evaluation process utilizes three datasets—MNIST, FMNIST, and CIFAR-10—employing the tanh activation function in every layer. As shown in Table 2, as the number of hidden layers increases, loss gradually increases, which is indicative of overfitting. It would be more effective to use the proposed weight initialization in conjunction with state-of-the-art architectures for training deep neural networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "06qHtRHL7v", "reviewer": "Reviewer_6yoV", "summary": "This work first provides a theoretical analysis of weight initialization when exclusively using tanh\nas an activation function. Providing reasons for the clustering behavior when initializing networks.\nBased on the developed theory, an initialization scheme is proposed, and the hyperparameter σz\nis empirically determined. There are two types of experiments mainly comparing the proposed\ninitialization with Xavier. The first type of experiment concerns the classification accuracy in the\nearly training phase. Showing there is an improvement in accuracy across different data sets and\nconfigurations. The second type of experiment concerns solving PDE with PINNS, showing that the\nproposed method has a good performance.", "questions": "In sections 4.1 and 4.2, the experiment trains for 20 or 40 epochs. Do networks converge to their best accuracy? What is the difference in accuracy when training for more epochs?\n\nThe experiments in section 4 are not too computationally intensive, is it possible to include more\ndata sets or PDE can show the improvements are general?\n\nCan the code used in the experiment can be provided to improve reproducibility?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Unlike optimization problems with theoretical guarantees on fixed points, weight initialization\nis an important task in deep learning. An initialization scheme with theoretical backing can\nhave a long-lasting impact, even just for a sub-field of deep learning.\n\nExperiments do show significant improvement when using PINNs to solve PDE.", "weaknesses": "The impact depended on exclusively using tanh as an activation function is fundamentally\nbeneficial in PINNs. As the current state of the paper, there is not enough support for this.\n\nGiven that the experiments are not too computationally intensive and the experiment section\nonly considers a few data sets or PDEs, the demonstrated improvement may not be general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qc50qazs92", "reviewer": "Reviewer_kh6z", "summary": "Weight initialisation is an old topic, and most studies have verified that weight initialization methods can improve the performance of neural networks. However, because that neural network’s depth increasing rapidly,  most neural networks, especially Feedforward Neural Networks (FFNNs) should face the gradient vanishment problem.  In this article, authors proposed one weight initialisation method for FFNNs and Physics-Informed Neural Networks (PINNs). Based on an analysis of the fixed points of the function tanh(ax), this method determines values of $a$ that prevent the saturation of activations during the training progress. In terms of robustness, this method presents a stronger and more efficient performance. In the experiment, verified on MNIST, Fashion MNIST, and CIFAR10 datasets, this method also shows acceptable results compared to the Xavier.", "questions": "The novelty of this work is strong, and the topic sounds interesting. However, the writing and structure should be revised again. There are some questions that the authors should be concerned about. \n\n1. In Eq. 2, $\\sigma_{z}$ is set to $\\alpha/\\sqrt{N^{l}-1}$ and $\\alpha = 0.085$. From Figure 2, we can find that the optimal value of $\\alpha$ is $0.085$. Is there any theoretical reason why $\\alpha$ should be $0.085$. Or should we manually try the value accordingly?\n\n2. Additionally, please scribe what is $\\alpha$. There is no definition of $\\alpha$.\n\n3. In Figure 3 (c) and (d), the proposed method seems to decrease after 6 epochs. Although the accuracy curve can rapidly reach the peak (faster than Xavier), the robustness of this method also should be discussed. For example, the initialization method can first provide prior knowledge to neural networks, but if it can keep the stability of training or not. Or is it the reason for the high $\\alpha$?\n\n4. In Appendix A.1, the authors discussed different conditions. When $x = 0$, whether the vanishment problem will abscond. Please highlight the strategy on how this method can process it.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. this article proposes one novel weight initialization method for the FFNN and PINN.\n2. the authors prove that the activation values cannot vanish when increasing the depth of the neural network by using a fixed-point analysis.", "weaknesses": "1. The universality of this method should be improved. For example, more experiments on the feasibility of other neural networks except FNN and PINN.\n2. The details of hyperparameters should be mentioned, such as what is the $Threshold$ of FFNN, which training strategy (supervised or unsupervised) of FFNN is used?   \n3. FFNN is specifically designed to visualize the trained features, which also should be discussed.\n4. The consistency of results should be guaranteed. For example, in Figure 3 (c) and (d), there is a different trend (Xavier tends to equal \nthe proposed method), which also should be discussed. In case that after 20 epochs, the performance would be totally different to the presented result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zRDEiYIjMf", "reviewer": "Reviewer_YCAx", "summary": "The paper proposes a new method to initialize weights for FCNNs and PINNs with tanh activation. The paper claims that the proposed weight initialization method will not lead to diminishing activations for very deep networks unlike Xavier weight initialization. The paper claims that the proposed weight initialization is robust to network depth and number of units in hidden leayers.", "questions": "1. Layer norm is added to handle the diminishing activation problem. Any reason why you did not compare the performance of the proposed approach with Xavier weight initialization with Layer norm?\n\n2. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean can be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that can lead to tanh saturation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\nThe paper presents a novel weight initialization method specifically designed for tanh-based neural networks, addressing an understudied area in neural network initialization. This approach is distinct in its use of fixed-point analysis to prevent activation saturation and improve training robustness across network sizes, particularly in the context of Physics-Informed Neural Networks (PINNs). By emphasizing robustness and performance consistency in both traditional classification tasks and PINNs, the paper makes a valuable contribution to the field of weight initialization. The originality is strong, given the lack of prior research focusing on tanh-based initialization methods.\n\nQuality\nThe paper demonstrates high quality in both theoretical and experimental aspects. The method is grounded in rigorous mathematical analysis, leveraging fixed-point theory to derive conditions that ensure stable activation propagation. The provided lemmas, proofs, and propositions add credibility and depth to the approach.\nExperiments are well-designed and span various network configurations, datasets (MNIST, Fashion MNIST, CIFAR-10), and applications (PINNs for solving differential equations). The results consistently show that the proposed method outperforms Xavier initialization, particularly in deeper networks and varying network sizes.\n\nClarity:\nOverall the paper is very well written, with some exceptions mentioned in the weakness section. All the sections in the paper are laid out clearly. The notations are consistent across the paper.", "weaknesses": "Significance:\nThe paper compares their proposed weight initialization with Xavier weight initialization for FCNN with tanh activation. Xavier is known to show diminishing gradients and activations problem for deeper networks, but this is solved by using layer normalization. I am therefore considering this work not significant because the problem that the authors are trying to solve does not exist for Xavier + Layer norm and the authors did not do any comparative analysis with and without layer norm.\n\nOther major issue:\n1. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean will be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that leads to tanh always saturated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)