Sinusoidal Initialization, Time for a New Start
Authors: Alberto Fernández-Hernández
Jose I. Mestre
Manuel F. Dolz
José Duato
Enrique S. Quintana-Orti
Venue: NeurIPS 2025 poster ()
Keywords: Initialization
Sinusoidal
Deep neural network
Activation
Primary area: deep_learning
TL;DR: Sinusoidal initialization replaces random weight seeding with a deterministic, structured scheme that balances weights and neuron activations from the outset, yielding faster, more stable training and higher accuracy across diverse models.

Abstract: Initialization plays a critical role in Deep Neural Network training, directly influencing convergence, stability, and generalization. Common approaches such as Glorot and He initializations rely on randomness, which can produce uneven weight distributions across layer connections. In this paper, we introduce the Sinusoidal initialization, a novel deterministic method that employs sinusoidal functions to construct structured weight matrices expressly to improve the spread and balance of weights throughout the network while simultaneously fostering a more uniform, well‑conditioned distribution of neuron activation states from the very first forward pass. Because Sinusoidal initialization begins with weights and activations that are already evenly and efficiently utilized, it delivers consistently faster convergence, greater training stability, and higher final accuracy across a wide range of models, including convolutional neural networks, vision transformers, and large language models. On average, our experiments show an increase of 4.8 % in final validation accuracy and 20.9 % in convergence speed. By replacing randomness with structure, this initialization provides a stronger and more reliable foundation for Deep Learning systems.

Sinusoidal Initialization, Time for a New Start
Alberto Fernández-Hernánde"" fernandez@upy.es
Jose Mestre jmiravetduji.es
Manuel F. Dolz²
dolzm@uji.e
Jose Duato deseeeeuttooopohhpp..... com
Enrique S. Quintana-Orti
quintanaddisca, upv.
Universitat Politècnica de València
Universitat Jaume I
Openchip & Software Technologies S.L..
Abstract
Initialization plays a critical role in Deep Neural Network training, directly influ- encing convergence, stability, and generalization. Common approaches such as Glorot and He initializations rely randomness, which can produce uneven weight distributions across layer connections. In this paper, we introduce the Sinusoidal initialization, a novel deterministic method that employs sinusoidal functions construct structured weight matrices expressly to improve spread and balance of weights throughout the network while simultaneously fostering more uniform, well-conditioned distribution of neuron activation states from the very first forward pass. Because Sinusoidal initialization begins with weights and activations that are already evenly and efficiently utilized, it delivers consistently faster convergence, greater training stability, and higher final accuracy across wide range of models, including convolutional neural networks, vision transformers, and large language models. On average, experiments show increase of 4.9% in final validation accuracy and 20.9% in convergence speed. By replacing randomness with struc- ture, this initialization provides a stronger and more reliable foundation for Deep Learning systems.
Introduction
Weight initialization remains a critical yet often underappreciated component in the successful training of Deep Neural Networks (DNNs) (including Convolutional Neural Networks (CNNs) and Transformer based architectures). Although numerous training strategies and architectural innovations receive substantial attention, initialization itself frequently remains in the background, implicitly accepted solved trivial problem. Nonetheless, the initial setting of weights profoundly affects training dynamics, influencing convergence speed, stability, and even the final model's ability to generalize [LeCun al. 1998 Glorot and Bengio 2010 (e al. 2015b]
Historically, weight initialization has evolved significantly. Early methods such as those proposed by LeCun et al. 1998) were primarily heuristic, offering basic variance scaling principles. The seminal work Glorot and Bengio (2010) rigorously connected weight initialization to variance preservation across layers, introducing randomized scaling approach that became foundational for Deep Learning (DL). Later, He et al. [2015b) refined this idea, tailoring variance preservation specifically to networks with Rectified Linear Units (ReLUs), and their approach quickly established itself as a default choice modern DNN training pipelines.
"These authors contrihuted equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite the established efficacy of these stochastic variance question remains largely unchallenged in current literature: Is randomness fundamentally necessary for effective neural network initialization? Random initialization introduces variability that can complicate reproducibility, debugging, and systematic experimentation. Moreover, it implicitly assumes that the optimal starting conditions for training must inherently involve stochasticity, without thoroughly exploring deterministic alternatives that might offer similar or even superior statistical properties.
variance-preservine preserving methods. a fundamental
In this paper, we directly address this scarcely explored assumption by proposing a novel deterministic initialization scheme, termed Simusoidal initialization. This technique preserves the critical variance- scaling property central to the effectiveness of Glorot and He initialization schemes, while completely removing randomness from the initialization procedure. It achieves this by employing Sinusoidal functions to deterministically initialize weight matrices, ensuring that each neuron given layer receives a distinct weight configuration. This use of Sinusoidal patterns enables rich diversity in weight values while maintaining global variance characteristies, striking a balance between structure and expressive power.
Specifically, our contributions are threefold:
We introduce the Sinusoidal initialization, a deterministic initialization strategy utilizing Sinusoidal patterns, explicitly emphasizing symmetry, functional independence, and rigorous variance control for signal propagation, thus eliminating stochastic uncertainties.
We present sound theoretical framework that reveals how stochastic initialization schemes can hinder neuron activation efficiency by inducing asymmetries and imbalances in neuron behavior. By contrast, deterministic approaches that enforce structural symmetry, such as the proposed Sinusoidal initialization, naturally mitigate these issues, leading to more stable activations and improved gradient propagation across layers.
Through extensive empirical evaluations involving CNNs, including some efficient architec- tures, Vision Transformers, and language models, we rigorously demonstrate the empirical superiority the Sinusoidal initialization over standard stochastic initializations, demon- strating faster convergence and higher final accuracy.
The Sinusoidal initialization consistently yields higher final accuracy and faster convergence, as detailed in Section 4 These benefits are already evident in the case of ResNet-50 trained on CIFAR- 100, presented in Figure I additional DNN architectures are also provided in Section
representative example. The experimental setup and results for
ResNet-50 on CIFAR-100
Validation Accuracy
70%
Intrilizaaionn
60%
Onhogonal
LSUV
Sausoidal
50%
50
100
Epoch
Figure 1: Validation accuracy training epochs for ResNet-50 on CIFAR-100, comparing Sinu- soidal with other initialization schemes.
By challenging the ingrained belief in randomness as a necessary condition for initialization, we encourage new avenues for rethinking foundational aspects of DNN training, advocating for deter- ministic and comprehensive practices in the future of DL research.
The remainder of this paper is organized as follows. We begin by revisiting the foundations of weight initialization and its role in DL in Section 2 We then introduce the proposed initialization scheme, outlining its definition and theoretical motivations along Section 3 This is followed by an extensive empirical evaluation Section across diverse architectures and tasks. Finally, we conclude by summarizing our findings and outlining future research directions in Section

---

2 Background: Revisiting Initialization
Effective weight initialization is deeply rooted in the principles of variance preservation, which aim to maintain stable signal propagation across layers. This stability is crucial to avoid the notorious problem of vanishing or exploding gradients, which hinder DNN training (Hochreiter et al. 200T].
Formally, for a layer with weight matrix W E x Glorot and Bengio (2010) proposed Glorot initialization, where each weight is sampled from a distribution (either uniform normal) with zero mean and variance Var(W)) = (m + n). specifically chosen to balance the flow of information during both the forward and backward passes.
Glonee Subsequently, Glorot et al. [201T] observed that ReLUs produce naturally sparse activations, typically leaving roughly 50% of hidden units active (positive) and the other 50% inactive (zero). They highlighted the significance
this balanced activation pattern for efficient gradient flow and effective learning. Recognizing this characteristic, He et al. (2015b) refined this initialization specifically for networks using ReLU activations, proposing the variance (ar(W) = 2/n which analytically accounted for ensuring robust data propagation.
While these stochastic methods have been widely successful, their intrinsic randomness introduces significant variability. In particular, the random sampling procedure can sometimes produce weight matrices that lead to suboptimal neuron configurations in the hidden layers. This can cause slow convergence, instability, or even divergence early in training.
Various methods have attempted reduce structure this randomness. Orthogonal initializa- tion [Saxe et al. 2014], for instance, ensures orthogonality of weight matrices, stabilizing signal propagation and improving gradient norms. but still involves random orthogonal matrix sampling. Layer-Sequential Unit-Vare Varianee (LSUV) [Mishkin and Matas [016]] takes step further by adjust- ing randomly initialized weights iteratively until achieving precise activation variance. However, LSUV remains data-dependent and partially stochastic, limiting direct reproducibility and practical convenience.
Building on this line of work, some recent approaches have explored fully deterministic alternatives random initialization. [Blumenfeld et al. [2020) state that DNNs can be trained from nearly all- zero, symmetric initializations, as long as some form of symmetry-brekking such as dropout or hardware-level non-determinism, present. Zhao et al. [2022] propose scheme based on identity and Hadamard transforms, which achieves benchmark performance on ResNet architectures while preserving signal propagation. While both methods demonstrate that randomness is not strictly necessary for successful training, they primarily aim to match the performance of standard random initializations in terms of final accuracy. Crucially, they do not report improvements in convergence speed, robustness across architectures, or theoretical insights into the limitations of randomness. These works play an important role in establishing deterministic initialization but remain early steps in the broader effort to understand and fully exploit its potential.
viable alternative,
A notable gap thus remains in the literature: Despite these advances, there is currently no widely adopted deterministic initialization method with demonstrated potential to surpass the performance of random initializations. This scarcely explored direction motivates our work, wherein we propose a fully deterministic alternative, explicitly designed to deliver improved accuracy and convergence speed across a wide range of architectures.
3 Definition and Theoretical Foundation of Sinusoidal Initialization
In this section, we introduce a novel deterministic initialization scheme for DNNs, which we refer to as Sinusoidal initialization. This approach is proposed as a theoretically alternative to conventional stochastic methods, aiming to enable efficient signal propagation and promote functional independence among neurons from the very first training step.
3.1 Sinusoidal Initialization
Let W E RRmx denote the weight matrix of a neural network layer, where n represents the number of input features and m the number of output neurons. We define the weights deterministically using
3

---

it then follows that sinusoidal functions as
W[i,j] sin(k; Ij 0z);
(I)
where
k; 2Ti, 2j = j/n and Φi 2
for Prrie E {1, m and j (1,2....,1!
Under this formulation, each row W corresponds to uniformly sampled sinusoidal waveform of frequency k; and phase offset Φs. evaluated over an evenly spaced grid of points in the interval [o, 1]. The linear increase in frequency with the row index ensures that the function fi(z) a-sin(k; x o;) completes exactly full oscillations. The phase offsets Φ); are also chosen to be equally spaced to avoid identical first values across the of matrix W.
The amplitude a is chosen according to theoretical foundations established by Glorot and Bengio [010) He et al. [2015b), which dictate the variance of weights in order enable efficient signal propagation. Specifically, in our particular case, we adhere to the variance proposed by Glorot, setting the variance of W to be 2/(m n). If denotes the variance of W when 2/(m + n) = a u²v, which yields natural way to determine the amplitude a.
Furthermore, we follow standard practice by initializing all neuron biases to zero, ensuring no preactivation offset is introduced the start of training.
Weight Distribution
Unlike traditional random initializations, this Sinusoidal scheme yields a non-Gaussian distribution of weights. As illustrated Figure the weights follow an arcsine distribution, exhibiting higher density near the extrema ta and lower density near zero. This contrasts with the symmetric bell-shaped curves centered at zero typically produced by standard stochastic initialization methods. Additionally, 3 a heatmap of matrix W, showcasing the underlying harmonic structure induced by the Sinusoidal construction. The structured and oscillatory pattern of each row is reminiscent of sinusoidal positional encoding employed in Transformer architectures (Vaswani et al. 2017] thereby offering spatial diversity without relying on randomness.
0.1
Welghtt Density
Input feature
0.0
-0.1
Outpur feature
+
Figure 2: Weight distribution under Sinusoidal initialization.
Figure 3: Heatmap of the initialized weight matrix.
This initialization strategy provides well-conditioned starting point for early training stages, will be further discussed in the next subsection.
3.2 Theoretical Motivation
This section provides a theoretical motivation for the design of the proposed initialization. We emphasize ability foster efficient training dynamics from the earliest stages. particular, we analyze the impact that initialization induces on neuron activation patterns in the intermediate layers of DNNs. Furthermore, we introduce a quantitative framework to assess activation quality.
We develop the following theoretical analysis for DNNs employing ReLU-like activations, which are functions such Gaussian Error Linear Unit (GELU), Sigmoid Linear Unit (SiLU), Parametric
indeed the initial inspiration behind initialization. The reader may note some differences, such the use phase shifts the waves, the exclusive use sine (as opposed to both sine and cosine), and the adoption of increasing rather than decreasing frequencies. Nonetheless, the underlying connection lies in the expressive power of harmonic functions to assign independent vectors, a fact long understood since Fourier.
PPositionll Encodings

---

Rectified Linear Unit (PReLU), whose shapes resemble the classical rectified activation. In such networks, we consider a neuron be active if its output is positive, and inactive otherwise. Unlike ReL.U, these activations may still propagate small negative values in the inactive state, resulting in smoother transition between activation regimes.
Intuitively, a neuron is inefficient when its activation pattern is imbalanced-i.e. when it remains either active or inactive for disproportionate amount of samples. Such behavior severely limits its representational power. We formalize this notion through the concept of a skewed neuron:
Definition 1. Let Z be random variable representing the output of neuron with corresponding 1, W2 ...... Wn, and let a (0, 1/2). The neuron associated with Z is said to be skewed with degree a if
weights Wi,
P(z > 0 W1, W W,) - 1/2| > a.
This definition quantifies the deviation of a neuron's 's activation from the ideal 50% - 50% balance. For instance, with = 0.3, neuron is considered skewed if the probability difference between activation and inactivation in absolute value exceeds the 80% 20% threshold.
A high prevalence of skewed neurons impairs training efficiency by reducing the network's capacity to exploit nonlinearities and undermining its expressive power, and hence impeding optimization and slowing down convergence.
Empirical evaluation of activation imbalance
To analyze how different initialization schemes affect activation balance, we conduct an experiment with the VIT B16 on the ImageNet- Ik dataset. The goal is to compare the extent of activation imbalance across several common initialization strategies, such as Glorot, He, Orthogonal, LSUV, and our proposed Sinusoidal initialization. Figure 4 visually illustrates neuron activation states the last feed-Forwadd of the ViT. Each plot maps neuron indices (horizontal axis) against 768 randomly selected input samples (vertical axis), with white pixels indicating active (positive output) neurons and black pixels indicating inactive (negative output) ones.
Gloros Normal
He Normal
Orthogonal
LSUV
Sipusoidal
Nemons
Neurow
Nourons
Neurows
Necrons
Figure 4: Neuron activation states, in white active neurons (> 0), for different initializations.
Traditional initializations exhibit distinct vertical white or black bands, indicating neurons that remain consistently active or inactive-ise skewed neurons. As noted in Section 2 Glorot et al. (0IT) emphasized the importance of balanced ReL.U activations, with roughly 50% of neurons active to
ensure effective learning. Skewed neurons disrupt this balance, limiting the network's 's expressive power by underutilizing its nonlinearity and pushing it toward more linear behavior. In contrast, the Sinusoidal initialization yields a more irregular, noise-like pattern with no dominant columns, suggesting more balanced and expressive activation regime.
To complement this visual analysis, we quantified the proportion of skewed neurons under two thresholds for a: 0.1 and 0.3. As shown in [ conventional initialization schemes result in an unexpectedly high proportion of skewed neurons, whereas the Sinusoidal approach barely exhibits any skewed neuron. These results highlight the potential of Sinusoidal initialization to better preserve activation diversity. For further experimental details, refer to Appendix C.2
Table 1: Percentage of skewed neurons under different a thresholds and initialization schemes.
Glorot
He
Orthogonal
LSUV
Sinusoidal
0.1
82.7 %
82.8 % 51.4 %
84.0 %
84.3 %
0.2 %
0.3
51.9 %
49.6 %
50.9 %
0.2 %
5

---

These findings reveal a strikingly high proportion of skewed neurons under conventional random initializations, whereas the proposed deterministic Sinusoidal initialization yields significantly lower rate. This contrast naturally raises a critical question: What is the reason for this phenomenon? The following subsection aims to shed light on the inner workings of random initializations, why they tend to produce skewed neurons, and why the Sinusoidal scheme successfully mitigates this issue.
Impact of randomness
is traditionally assumed that randomness weight initialization is essential for effective training in DNNs. However, the results presented in this work challenge this long-standing belief, revealing that randomness may in fact seed unfavorable configurations from the very beginning.
In a standard random initialization, the weights W1, ..,..., W of a neuron are independently sampled from distribution centered at zero. We identify critical statistic that governs the initial behavior of the neuron: the cumulative weight imbalance, quantified by the sum
5 W + + W..
This quantity captures the imbalance between positive and negative weights. Despite the zero-mean assumption, the value of S can deviate substantially from zero due to statistical fluctuation. For example, under the He initialization [He et al.) 2015b), where weights generated with variance o² 2/n, the Central Limit Theorem implies that S ~ (0, 2).
To empirically investigate the relationship between this statistic and the emergence of skewed neurons, we take, for simplicity, a Multilayer Perceptron (MLP) as the reference model on which to conduct experiments. For full experimental details, see Appendix C.2 When focusing solely on the first layer of the network, we observe a crystal-clear correlation between the two concepts.
To illustrate this relationship, Figure presents presentss histogram of S values across all neurons in the first hidden layer. Blue bars correspond to neurons classified as skewed at level a = 0.3 (i.e. with skewed balance greater than 80% 20%), while orange bars denote non-skewed neurons. 'he plot reveals striking pattern: neurons with large absolute values of S are invariably skewed, confirming a strong empirical correlation between [S] and the degree of skewness. This reinforces the central role of [S] in determining activation imbalances at initialization.
Clarot Normal
Ilc Normal
Orthogonal
LSUV
Sinmscidal
Danele
-2.5
2.5
Wrighex
Weightt
Sunn Weights
Sum of Wiighos
Sum of Wiightte
Sleweed (a 0.3)
Not Skewed
Figure 5: Histogram of the statistic S values, highlighting the link between large |S| and neuron skewness at a 0.3.
This observation makes it evident that, at the first layer of the model, the relationship between skewed neurons and the corresponding value of the statistic S is remarkably precise: for each value of a, there exists a unique threshold such that a neuron is skewed of degree a if and only if [S] > X. This equivalence can be formally established, as stated in the following theorem.
Theorem 1 (Threshold Equivalence). Let W1, Wz,... W,, and X1, Ne,..., Xn, be i.i.d. sequences with
[[ = 0, B[Xi] 1> 0, Var(W,) = 0² (0, Var(X,) = oz < 00. Define Sn = Σi= W; and 7n = En W,X, Then for any a E (0, 1/2), define
\n(a)= == (fa (m/mm//n /n क (1/2 a), where --1 is the quantile function of the standard normal distribution. Then, as n - co, the following equivalence holds with probability tending to one:
IP(Z, > 0 W1, W ..... Wn) 1/2) > ind only if 18,] > (a).

---

This theoretical result establishes a tight connection between the statistic S and the skewness property of neurons. As direct consequence, stochastic initializations obtained by sampling from any zero-mean distribution inevitably lead to skewed neurons with probability tending to one. Formally:
Theorem 2 (Asymptotic skewness under random initialization). Consider a neuron with zero bias and input X1 xi....; Xnn and weights WI,..., Wn- Assume (W,) are ii.d. with E EW]
and
Var(W;) = f² E (0,00), oo), and {X;} } are iidd, independent of [W.}. with [[Xi] = μ > 0 and
Var(Xi) = o² < oc. Let Un = Li=1 W,X, Then for every deterministic sequence an + 0, the
probability of having
p( 00 Wi...., W..) 1/2) >
an
tends to one as nL + 0c. Equivalently, with probability tending to one there exists nonzero skewness level which the neuron skewed.
Therefore, in order to design initializations that do not systematically induce skewed neurons, it is necessary to move beyond the classical family of random schemes that fall under the assumptions of Theoremm 2 This motivates the exploration of deterministic initializations specifically crafted to enforce balance from the outset. In the particular case of the Simusoidal initialization, the connection with Theorem Theorem[]eccomee becomes especially meaningful. Unlike standard random initialization schemes which offer no guarantees on the aggregate behavior of individual neurons, the Sinusoidal initialization exhibits an inherent structural regularity. This symmetry manifests in the exact cancellation of weights along each row of the initialization matrix, ensuring that each neuron begins with a perfectly balanced contribution across inputs. The following result makes this property precise:
Theorem (Row Cancellation). Let W E R=x" be defined as in Equation with m < n. Then, for every row index i E (1,2...., m.) the entries along that row sum to zero:
Σ W(ij]] 0.
The proofs and further theoretical analysis can be found in Appendix B
Consequently, if the objective is to initialize the weights without introducing any biased (skewed) neurons, then, at least in the first layer, we should aim to enforce
for every neuron.
For deeper layers though, the correlation between neuron skewness and the magnitude of S progres- sively fades due to the asymmetries introduced by preceding layers in the established stochastic initializations. Indeed, the input to an intermediate layer is already biased, and the mean of each input component can deviate more and more, gradually decoupling the notions of skewness and [S]. This issue, however, is naturally mitigated by the Sinusoidal initialization. Owing to its strong symmetry, which ensures S 0, it guarantees that each neuron begins with an input that is perfectly centered, thereby eliminating any intrinsic directional bias. As result, the Sinusoidal initialization theoretically guaranteed to prevent the emergence of skewed neurons, an undesirable behavior frequently observed under stochastic schemes. This conclusion is not only supported by the theorems presented above but also empirically validated in Ta where the proportion of skewed neurons identically zero across all thresholds tested.
The relevance of this symmetry during training will be quantitatively assessed in Section 4 However, before addressing training dynamics, it is crucial to examine a second foundational property: whether individual neurons are initialized with sufficiently diverse and independent functional responses.
Functional independence across neurons
Another critical consideration is the functional independence among neurons in a layer. An optimal initialization scheme should ensure that each neuron focuses on distinct aspects of the input, thereby enhancing representational efficiency.
To detect potential correlations among neurons, we examine horizontal patterns in Figure 4 A dark or white horizontal band would indicate that all neurons respond similarly to a specific input sample, signaling redundancy. As observed, none of the schemes, including the Sinusoidal one, exhibit dominant horizontal patterns. In our case, functional independence is enforced through distinct frequencies k; and phase offsets o; for each neuron, evoking the orthogonality of Fourier bases.
7

---

In order to further look into more intrinsic horizontal dependencies, the Setritting-Unndfittinng Indicator (OUT), as introduced by the authors in Fermandez- to numerically assess functional independence across neurons. This indicator measures the similarity between activation patterns of sample pairs within given layer, assigning values in the range [0, 1], where 0 indicates highly correlated activations and 1 denotes complete independence. In this context, the activation pattern of a sample is understood as binary vector, where each entry corresponds to the on/off' activation state of hidden neuron. Hence, OUI quantifies how differently pairs of samples activate the network, providing a direct measure expressive capacity at initialization.
al. [2025], can be employed
While all methods yield reasonably high OUI scores-Glorot (0.56), He (0.59), Orthogonal (0.57), and LSUV (0.58) the Simusoidal initialization consistently achieves the highest value, reaching an outstanding 0.98. This indicates that, although traditional initializations already promote a significant degree of activation diversity, the Sinusoidal scheme provides a strictly better starting point with respect to this metric, offering a more expressive and balanced activation regime from the very beginning.
Theoretical conclusions
The findings of this section lead to two key insights regarding initialization strategies for DNNs:
1. Random initializations naturally induce high proportion of skewed neurons, which can compromise training efficiency during early stages.
2. Deterministic initializations can substantially mitigate this issue, provided they are designed with a structure that incorporates symmetry, functional independence, and variance control.
From this perspective, the proposed Sinusoidal initialization emerges as a strong candidate: combines bounded function family with inherent symmetry, suffieient expressiveness ensure neuronal independence, and an analytically tractable formulation. Together, these properties make a promising alternative for establishing new standard in the initialization of DNNs.
Experimental Benchmarking of Sinusoidal initialization
Having introduced the Sinusoidal initialization, a novel deterministic scheme theoretically motivated, turn to its empirical validation against both classical and state-of-the-an (SOTA) initialization methods across a range of DNN configurations.
Our experimental validation specifically addresses two main hypotheses: 1) Accuracy improvement: The Sinusoidal initialization achieves higher final validation accuracy compared to alternative initial- ization schemes in a significant proportion of cases. 2) Accelerated convergence: The Sinusoidal initialization converges faster than any other initialization considered, as measured by the Area Under the Curve (AUC) metric, which besides for classification evaluation, can be used to compare the training convergence speeds (Huang and Ling 2005].
To evaluate these hypotheses rigorously, we conducted experiments involving five diverse config- urations of DNNs and datasets: Re (He al., 2015a) trained on CIFAR- 100 Krizhevsky
et al. [2009], MobileNetV3 (Howard et al.) 2019] trained on Iiny-ImageNet[L and Yang [2015). EfficientNetV2 [Tan and Le| 2021] trained on Tiny-ImageNee, ViT-B16 [Dosovitskiy et al., 2021 trained on ImageNet- 1K Russakovsky et al. (2015). and BERT-mini [Devlin et 2019 trained on WikiText Merity et al. [2016]. For the first three configurations, we utilized SGD, Adam, and AdamW optimizers, while ViT-B16 and BERT-mini were trained exclusively with SGD and AdamW, respectively, due to the inability of other optimizers to achieve effective training. To ensure a fair comparison between training processes, learning rate schedulers were not used, which may affect the comparability of final accuracies with benchmarks focused on maximizing accuracy.
We trained each configuration fully from scratch using four initialization methods: of the He initialization detailed in Table Orthogonal (Orth.), LSUV, and our proposed Sinusoidal initialization (Sin.). The exact training procedures, including hyperparameter settings and imple- mentation details, are provided in Appendix C.3 We measured the maximum validation accuracy achieved at epoch epoch 10, and overall, as well as the AUC for every training scenario.
Default (variants
8

---

Table 2: Default initializations for each model.
Model
Layer type
Default initialization
Activation
Conv2D
normal He uniform
ResNei-50
Linear
ReL.U
MobilcNetV3
Conv2D
He normal Normal
HardSwshh
Linear
E0icienNNeVV2
Conv21
Linear
normal Uniform
SILU
VTT-B16
Conv2D
Truncated nomaal
Glurot uniform.
GeLU
Rel.11
BEKTTmiii
Liner
Normal
GeLU
In all our setups, to fairly compare initializations using AUC across different runs, we ensure that all models are trained for the same number of epochs. For each combination of model, dataset, and optimizer, training is performed with early stopping to guarantee convergence for every initialization. Once the slowest run has converged, the remaining models continue training for the same number of additional epochs until reaching the maximum epoch count. This procedure ensures that (1) all trainings reach their maximum validation accuracy, and (2) the number of epochs used to calculate the AUC for measuring convergence speed is consistent across initializations. The detailed experimental outcomes summarized in Table B and illustrated in Figure 6
Table 3: Validation accuracy (%) after epoch, 10 epochs, and maximum accuracy, well the AUC for each optimizer and initialization scheme (Default, Orthogonal, LSUV, Sinusoidal..
Model/ Dataset
Optim.
epoch aceuracy (%)
10 spocls accuracy (%)
Maximuns accurncy
AUC
Del.
Orth.
LSUV
Sin.
Def.
Orth.
LSUV
Sin.
Def.
Orth.
LSUV
Sin.
Def.
Orth.
LSUV
Sin.
CIEAR 100
ResNes-50
Adamn
Adam
SGD
12.9
10.1
2.6
12.5
13.9
4.0
12.6
11.8
3,6
23.1
19.9
5.0
58.5
$9.9
9.7
44.1
18.2
14.8
62.5
(3.1
615.7
24.3
46.7
37.3
53.1
67.5
67.7
58.6
46.5
02.7
56.3
44.8
61.5
51.9
71.0
25
G4
48
65
51
55
31
51
06
68
57
42
MobileNetV3 EonylmgeNee
SGD
0.7
1.2
1.5
3.0
2,0
4.5
18.4
25.8
26.0
21.6
26
38
35
Adam
5.0
32.8
34.8
71
Adamn
12.3
EffiientNerV2 HinylmageNet
Adam
2.9
23.0
27.7
281.8
32.7
36.6
22.0
AdaunW
9.6
9.3
11.8
9.4
48.1
48.2
48.7
53.0
50.0
50.2
40.5
68.5
100
100)
100)
106
Ie
VIT-16
SGD
2.8
3.8
3.5
2.6
15.2
15.2
14.1
13.9
28.6
28.2
29.6
31.5
23
25
24
25
HERT-mini
Wikilext
Adam
9.4
11.0
13.8
10.9
14.8
10.4
15.9
41.1
58
32
RosNet-50 on 110AR-100
MobileNetV3 on Ting-Inaeenet
EfficientNeVV2 on Tiny-Imagenet
Validation Accuney
70%
Validation Accuncy
44%
55%
42%
50%
60%
40%
Validation
45%
38%
50%
36%
50
100
100
200
100 Epoch
200
Epuch
Epoch
VIT B16 on ImageNet-1K
BERT-min mini on WikiText
40%
Validaaion Accamey
30%
Validaiin Accuncy
25%
30%
20%
20%
19%
10%
10%
50
100
100
200
Epoch
Epoch
Default initialization
Orthogonal
LSUV
Sinusoidal
Figure 6: Training curves showing that Sinusoidal initialization leads to faster convergence compared to default, orthogonal, and LSUV initializations across architectures, using the optimizer that achieved the highest accuracy (SGD for VIT, Adam for the others).

---

Analyzing these tables, several clear trends emerge. Firstly, our Sinusoidal initialization consistently achieves higher final validation accuracy compared to other initializations, supporting our first hypothesis. Notably, in the case of MobileNetV3 on TinylmageNet, Orthogonal initialization slightly outperforms our method. Despite this isolated exception, the general trend strongly favors our Sinusoidal initialization. This particular case, however, highlights that this research field remains rich with open avenues for exploration. We conducted an additional experiment replacing the HardSwish activation in MobileNetV3 with ReLU, and under this setting, Sinusoidal once again outperformed all other initialization strategies, consistent with the rest our results. This suggests that the observed behavior stems from the interaction between the HardSwish activation and the initialization scheme. This particular case, however, highlights that this research field remains rich with open avenues for exploration. The behavior observed in this architecture offers valuable opportunity to further advance our understanding of the DNN training process and refine Sinusoidal initialization strategy accordingly.
Secondly, regarding convergence speed, the Sinusoidal initialization outperforms every other initial- ization in AUC, except in MobileNet, which indicates a faster convergence and higher accuracy in average over the training process. also consistently surpasses all other methods terms of valida- tion accuracy at early stages of training, specifically at epochs I and 10, unequivocally confirming our second hypothesis.
Integrating these observations into high-level metrics, our experiments collectively yield an average increase final validation accuracy of 4.9% over the default initialization, 1.7% over orthogonal and 3.4% over LSUV. Considering the overall training process, the AUC indicates an average improvement of 20.9% over the default initialization, 5.7% over orthogonal and 18.0% over LSUV.
These results strongly support the theoretical predictions from the previous section and underline the practical advantages of the proposed Sinusoidal initialization. Such significant and consistent gains in accuracy and convergence speed emphasize the potential of deterministic, symmetry-driven initialization methods as effective tools for enhancing the performance and efficiency of DNN training.
5
Conclusion
In this article, we introduced and extensively studied the Sinusoidal initialization, a novel deterministic approach to initializing DNNs. Our method fundamentally departs from the traditional paradigm of stochastic initializations, employing structured Sinusoidal patterns designed to enforce symmetry, functional independence, and precise variance control from the outset.
rigoroos rigorous theoretical analysis, we demonstrate that conventional random initialization schemes inherently introduce asymmetries and imbalances, resulting in high prevalence of what we define as skewed neurons, notion we formalize and quantify in this work. This skewness substantially undermines neuron activation efficiency and may restrict the expressiveness of the network. In contrast, the deterministic structure of the Sinusoidal initialization is theoretically proven to inherently mitigate these effects.
Empirical evidence strongly supports our theoretical claims. Extensive experiments across diverse DNN architectures, clearly indicate the superiority of Sinusoidal initialization. Specifically, this approach consistently achieved higher final validation accuracy and accelerated convergence rates compared to SOTA stochastic initialization methods. Our results quantify an average accuracy improvement over the default initialization of approximately 4.9% and notable boost in convergence speed of around 20.9%, underscoring the method's 's robustness and versatility.
In conclusion, our findings challenge the longstanding assumption that randomness is essential for effective model initialization, presenting strong theoretical and empirical arguments in favor of deterministic methods. The Sinusoidal initialization represents significant breakthrough in DNN initialization practices, paving the way for future explorations into deterministic initialization schemes, potentially redefining standard practices the design and training of DNNs.
10

---

Acknowledgments and Disclosure of Funding
This research was funded by the projects PID2023- 1-8---116666----68 C21 and BID2023-1465699 C22 supported by MICIU/AEI/10./ (10AA////101339/5000 10001 1033 and ERDF/UE. Alberto Fernández- Hernández was supported by the predoctoral grant PREP2023-001822 supported by MI- CIU/AEE/10.13039/50 10000000010000000000000000000000 grant ACIF/2021/281 the Generalitat Valenciana. Manuel F. Dolz was supported by the Plan Gen-T grant CIDEXG/202/013 of the Generalitat Valenciana.
and ESF+ Jose I. Mestre was supported by the predoctoral
References
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: Is feature di- versity necessary in deep neural network initialization? In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings Machine Learning Research, pages 960-969. PMLR, July 13-18 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. 2019. URL hRLpttps::// httth:ttttattiiii org/ aba/
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale, 2021. URL /tttpt/////axiii org/ abs 2010. 11929
Alberto Fermández- Jose I. Mestre, Manuel F. Dolz, Jose Duato, and Enrique S. Quintana- Ortí. Oui need to talk about weight decay: new perspective on overfitting detection. In 2025 International Conference on Advanced Machine Learning and Data Science (AMLDS), pages 96-105, 2025. doi: 10. 100100//MMDDS6318..0000 202.5633918.2221111555
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May mlr. press/v9 v glorot10a..... html
2010. PMLR.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statisties, pages 315-323. JMLR Workshop and Conference Proceedings, 2011.
Kaiming He. Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recogmition, 2015a. URL http:: http://axxvvvoo org/ httg//bb//1112/ http//1522.0335555
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing numan-leved performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015b. doi: 10. 11111//C...201...20
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In Georg Dorffner, Horst Bischof, and Kurt Hornik, editors, Artificial Neural Networks ICANN 2001, pages 87-94, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44668-2.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc Le, and Hartwig Adam. Searching for MobileNetV3, 2019, URL httpt: //axivv org/ /aobs/19955. 02244
Jin Huang and Charles X Ling. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on knowledge and Data Engineering, 17(3):299-311 2005.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.

---

Yann LeCun, Leon Bottou, Genevieve B. Oir, and Klaus Robert Müller. Efficient BackProp, pages 9-50. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 978-3-540-44430-00. doi: 10.1007733-500-4443388 URL htt.:tts:://doi..... 10. 11001/0000000--544-9944-------
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL httph:tttt:/arrt wwwp://g///bbbbbbeeee.o 07843
Dmytro Mishkin and Jiri Matas. All you need is good init, 2016. URL httph:://tsxiiii org/
abs Abs/1511. 06422
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (UCV), 115 (3):211-252, 2015. doi: 10.1007/s1 1211077/11263--055-8866666
Andrew M. Saxe, James I.. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014. /tk.iittps://arxiv.oor abs/ 1312. 6120
Mingxing Tan and Quoe V. Le. EfficientNetV2: Smaller models and faster training, 2021. URL https://arxiv org ab5/2104. 00298
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ neurips.co.mooo #f5e2255477ee1100000ccCccaaaB Paper .dff
Jiawei Zhao, Florian Tobias Schäfer, and Anima Anandkumar. Zero initialization: Initializ- ing neural networks with only zeros and ones. Transactions on Machine Learning Research, 2022. URL http://ppaarrreieeeeeee//rrrrrrr arXiv:21 10. 12661..
Accepted by TMLR; preprint
12

---

A Limitations and Societal Impact
This work opens the door to exploring deterministic initializations that exploit symmetry to endow neural networks with desirable properties beyond the reach of stochastic schemes. Among the possible designs, we focused on the Sinusoidal initialization due to its natural structure and analytical tractability. Nevertheless, the broader space of structured deterministic initializations remains scarcely explored, offering a fertile avenue for future research.
In terms of empirical evaluation, while our method consistently outperforms alternatives across most settings, results on MobileNetV3 fall slightly behind those of Orthogonal initialization. Understanding why this architecture deviates from the general trend, despite its apparent similarity to EfficientNetV22 could reveal important insights and guide further refinements. We regard this as a localized irregularity rather than fundamental shortcoming.
Finally, although we have a clear theoretical understanding of how Sinasoidal initialization guarantees the absence of skewed neurons in MLPs regardless of depth, and unlike traditional stochastic methods, further research needed to fully characterize this behavior across a broader range of architectures. In particular, establishing a general theoretical framework that ensures the absence of skewed neurons all established architectures remains an open and promising direction for future work.
Regarding the societal impact, the proposed Sinusoidal initialization enables faster convergence during training, reducing the number of epochs required reach target performance. This directly translates into lower computational demands, which in turn reduces both the economic cost and the environmental footprint of training DNNs.
Given that a substantial portion of the energy consumed in large-scale machine learning pipelines originates from high- perfforrfee computing centers, many of which still rely on carbon-emitiing energy sources, the adoption of more efficient initialization strategies could contribute to a reduction in overall CO2 emissions. While the exact magnitude of this impact depends on deployment scale and infrastructure specifics, our results highlight a concrete opportunity to align algorithmic efficiency with broader sustainability goals.
B Theoretical details
This appendix provides the theoretical foundations underlying the results stated in Section 3 and in
particular formal proofs of Theorems I 2 and 3 We also present general version of Theorem [ extending the simplified setting used in the main text a more flexible framework that covers non-identically distributed inputs and milder moment assumptions.
Beyond these formal results, we include further technical insights that clarify the connection between random initializations and the emergent activation patterns observed in the early layers of deep neural networks.
B.1 Threshold equivalence
In Theorem I we stated, under strong i.i.d. assumptions, that the weighted sum Zn = EW, is controlled with high precision by the magnitude of the total weight S, = Σ W. Specifically, for each level a E (0, 1/2). there exists a sharp threshold X, (ax) such that the probability P(Z,, > 0 W|W,...., W,) deviates from 1/2 by more than if and only if [S,] > (a), with high probability.
In this subsection, we show that this result admits a broader formulation under significantly weaker assumptions. In particular, the data XI...., X,, need not be identically distributed, and only mild regularity is required. The result below extends Theorem and justifies its asymptotic equivalence through general concentration and normalization arguments.
Theorem 4 (Threshold Equivalence, general version). Let (W).........W.... XWn,X.) be independent pairs of real random variables, where the weights Wi,- W...... Wn. are i.i.d,, independent of the data X1,...y Xm and satisfy
B[W]] =0 Var(W1) 0² o0).
13

---

Assume that the data X1.... of E (0, 20), and that
Xn are independent with a common mean + O, variances Var(X;;)
Σ o - (0,
as + 00.
Finally, assume the conditional Lindeberg condition,
IE
W?(X: 11 (IX (IX-mbbeEWWRd)
for all > 0,
E.=1
1=1
holds almost surely. Define Sn = >i W; and Zn - W,X,, Then for any a E (0,1/2). define
>n(a)= := (OVo> pt (0/2///0001//00))))
where -- the quantile function of the standard normal distribution. Then, as n + co, the following equivalence holds with probability tending one:
P > 0 WI,.... Wn. - 1/2) > a if and only if [S,] > (n
In particular, for every E (0, (0,1/2), the threshold (a) is asymptotically unique with high probability.
Sufficient conditions. The result holds in particular when the data XI...., X, are i.i.d. with B[Xi] /a * and Var(X,) o² < In this case, the conditional Lindeberg condition is automatically satisfied by the classical Lindeberg-Feller Central Limit Theorem (CLT), since the summands W,X; form triangular array with independent rows and uniformly bounded second moments. These are precisely the assumptions stated in Theorem [ which is therefore recovered as particular case of the present general result.
Proof. Let us write 2 = Σn i=l W?o??. Given the weights Wi...., Wn, we consider the normalized sum
(Z, - HS,) W Wn.
Under the assumptions of the theorem, the classical Lindeberg- CLT for triangular arrays implies that this converges in distribution standard normal:
(Zn pSn)/Em, W Wi...., Wn 2, 1).
As consequence,
P(Zn > 0 WT, W.)) & (JS,,/En) + o(1);
where the error (1) vanishes in probability as n + oc. Now observe that since W....,, W. are i.i.d. with variance 0 we have by the law of large numbers that
Σ W2 no o(1)), and M Wiot 21
Combining these gives
# H
-
(1 o(1)) = Cu.
Σ, A vn
so that
P(Z, 0| W Wi,... ..... W.) (c. +m) o().
Now fix any (0, 1/2). Since the standard normal cdf क is strictly increasing and symmetric about zero, we find that
IP 0| Wi... Wx) 1/2) > a if and only if Ic,. - (1/2+a))
Solving for [S.]. this is equivalent to
|S + (1/2+a)/ff /c, A,) +o()).
This equivalence holds with probability tending to one, since the approximation error vanishes and S,, is independent of the remainder.
Finally, note that the monotonicity of + implies that this inequality can hold for most one threshold X, (a). Therefore, the threshold is asymptotically unique.
14

---

Theorem ([Theorem) ( (Asympiotic skewness under random initialization).. I Consider a neuron with zero bias and imput ....... Xn and weights W....., Wn. Assume {W:] are i..d. with B[Wi] and 1ar(Wi) 6² E (0, 20), and (Xx) are iid. independent of (W.), with B[X,]
0 and
Var(X,) o² oc. Let 7m Di=1 W,X; Then for every deterministic sequence On + 0, the
probability of having
|P > Wi,..., W,) - 1/2 >
On
tends to one as n oc. Equivalently, with probability tending one there exists nonzero skewness level at which the neuron is skewed.
Proof. Write Sn =E We and fix any sequence on & O. By the Threshold Equivalence Theorem for each fixed a E 10, the event
F (a) = |P 0 W
W.) 1/2 n}
is, with probability tending to one, equivalent to
(a(a) = {1su (05l>>n(aa)} )n(a)}.
where
(a = (0avn)/H x- (1/2
To apply this along a vanishing sequence, choose a decreasing sequence (k) + and integers N* 1 oc such that, for all NN. the difference between E,, (a.))) and F, (a()) has probability at
most 2 Define piecewise constant selection On (k) for N& I n < N This ensures
P (cn)
+0.
remains show that ((Fa(aa)) - 1. By the Central Limit Theorem,
Sn
0/n 1).
Since On - 0 and - is continuous at 1/2, we have that
(an) On = 0V 1
о 4
1 +an) 0.
Thus, if (0, is denoted by G, it follows that
((F)(an)) = PP() (1
Sn
3n)) (n) _ 3 > 0) =.
0 0
Combining both steps yields
((En(an)) 2 PP ((n))))) (P( - 1,
and the result follows.
Propagation of skewness across layers. At the input layer, the assumption of equal means across coordinates, central to Theorem typically satisfied due to standard data normalization practices. However, when standard stochastic initializations are used, the first hidden layer introduces an unavoidable imbalance: the randomness in the initialization causes the activations to deviate from a centered distribution, resulting in the emergence of skewed neurons. Consequently, the inputs to the second layer are no longer mean-centered across dimensions. As training progresses deeper into the DNN, these initial asymmetries are compounded, each layer propagates and potentially amplifies the imbalances introduced by the previous one.
This effect gradually breaks the correspondence between the skewness of a neuron and the statistic S). as established Theorem hinges critically on the input means μ4; being equal. Once this condition fails, the quantity Σ, H,W, governing the conditional bias of the neuron no longer proportional Sn, and the connection between sign bias and the statistic S deteriorates.
particular, the conditional expectation E Z, Wi...., W W,] =,,,
15

---

This breakdown is clearly observable in the evolution of the histogram shown in Figure 7 which displays the distribution of the skewness statistic S across layers in a multilayer perceptron with four hidden layers. If layer is initiated with a stochastic criterion (such as Glorot), the distribution of closely follows normal distribution, as predicted in Section 3 At the first hidden layer initiated with random distribution (first row and first column of Figure Z skewed neurons clearly remain concentrated in the tails. However, as depth increases (second, third and fourth histograms in the first row), these blue tails representing skewed neurons progressively dissolve and spread toward the center of the distribution. As result, skewed neurons begin to appear even at small values S, indicating that neuron skewness becomes less predictable from the weight imbalance alone.
Chedl Lays
Ghooi Laps
Glool Lees
Deatt
Sann Minghs
Saa Miglls
Sam Wengts
Saanskiad Layer
Sannoiial Lager
Ghaoi Exps
Desht
Drarit
Dealty
Wiides
Sias Wiidaas
Widdes Saraniial Laper
Sarankial Laper
Strnokial Loger
Deaty
Dealy
Dealyy
Skoued (o-003)
Not Skrawi
Figure 7: Evolution the values of the statistic S for skewed neurons across layers in a MLP.
By contrast, the Sinusoidal initialization fundamentally avoids this pathology. As shown in Theorem 3 each row of the initialization matrix sums to zero, ensuring that the output of the first layer is strictly centered around zero for all neurons (first element of second row). This structural symmetry guarantees that no directional bias is introduced at the very start of the network. More importantly, prevents the cascade of asymmetries that otherwise accumulates across layers. The rest of the figure shows how Sinusoidal initializations in the first layers of the model help in progressively achieving an equilibrium terms of skewed neurons. As result, the relationship between skewness and S' remains intact even in deeper layers, and the network preserves maximal discriminatory power throughout its depth. This stability reflected both theoretically and empirically: in Table [ the proportion of skewed neurons under Simusoidal initialization remains identically zero across all tested thresholds. This key structural ingredient, the enforced symmetry at initialization, lays the foundation for qualitatively new regime in network design. It resolves long-standing inefficiencies related to neuron utilization and enables networks start from maximally expressive configuration, thereby accelerating training dynamics systematically. As such, the Sinusoidal initialization marks
16

---

turning point in the initialization literature, offering principled and effective alternative to stochastic schemes.
B.2 Row cancellation in the Sinusoidal initialization
As introduced Section S the Sinusoidal initialization induces deterministic structure on the weight matrix that stands in stark contrast with the randomness of classical schemes. This structure is not only symmetric, but also highly regular in a way that yields exact cancellation patterns across the rows of the initialization. In particular, each neuron's ' incoming weights sum zero, leading to strictly balanced preactivation regime from the very first forward pass.
The result we now prove formalizes this cancellation property. To that end, let us recall how the weights are defined under the Sinusoidal initialization. Let W Rmxn be defined coordinate-wisg
W jl = a sin(k; I + +i),
where
k 2mi, 2j j/n and di 2 i E [I,...,m] m and j [l.. Theorem (Theorem (Row Cancellation)). Let W E Tg with m be defined as in Equation Then, for every row index m.) the entries along that row sum to zero:
-
Proof. Fix any E (l....,m..
m]. We consider the sum:
Σ W Σ
sin
2Ti J
2Ti
n
m
j=1
Applying the identity sin(e (a + 3) = sin a cos B + cos a sin 3, we rewrite the sum as
Σ sin 2xi
j 2
2Tí
Σ
2
2Ti
n
2
cos
sin
COs
n
m
m
n
m j=1
j=1
Thus, to conclude the proof, it suffices to show that both trigonometric sums vanish, that is,
2
2
Σ sin
0,
Σ
cos
0.
n
n
To show this, let w E C be the unitary complex number and angle 2wi/n, which clearly satisfies that w = 1. As < m < n, follows that w + 1, and hence
Σ w
1 1 " 1- 0.
The two real-valued sums above correspond to the imaginary and real parts of this complex sum, respectively. Therefore, both must vanish, and the result follows.
C Experimental Setup
This appendix contains additional material complementing the main text. Specifically, provides complete details for all experiments presented in the paper, including training configurations and hardware specifications.
For full reproducibility. all code used to conduct the experiments is publicly available in the accompa- nying GitHub repository: https: thub. com/
To assess the robustness of our findings, all experiments supporting the main claims were repeated three times. We report mean values across these runs. Standard deviations were computed and found be consistently small. As these deviations not materially affect the conclusions, they are omitted from the main tables and figures for clarity.
17

---

C.1 Models, datasets and initializations
This work employs range DNN architectures across vision and language tasks. The models and datasets used are listed below, along with their sources and brief descriptions.
Models
ResNet-50 (Torchhision): A deep residual network with 50 layers, widely used for image classification tasks.
MobileNet small (Torchvision:: : A lightweight convolutional neural network optimized for mobile and low-resource environments.
EfficientNetV2 small (Torchvision): A family of convolutional models optimized for both accuracy and efficiency.
VIT B16 (Torchvision): Vision Transformer model that applies transformer architectures to image patches.
BERT-mini (Hugging Face): A compact version of BERT for natural language processing tasks, trained with the Masked Language Modeling (MLM) objective.
Datasets
CIFAR-100 (lorchvision): A dataset of 60,000 32x32 color images in 100 classes, with 600 images per class.
Tiny-ImageNet (Kaggle): A subset of ImageNet with 200 classes and 64x64 resolution images, sourced from the Kaggle repository xiataokang/tiine tiiyiinnnnntttreeeee
ImageNet-lk (Official Repository): ) large-scale dataset containing over 1.2 million images across 1,000 categories, used for benchmarking image classification models.
WikiText2 (HuggingFace Datasets): A small-scale dataset for language modeling, using the Eikitext-2-raxe version to preserve raw formatting.
Initialization Procedures Four initialization schemes considered in this work:
This corresponds the initialization procedure provided in the original codebase for each model, as implemented the respective libraries (e.g., Torchvision or Hugging Face).
Default:
Orthogonal and Sinusoidal: These initializations manually applied the weights of all convolutional and linear layers in the model.
LSUV (Layer-Sequential Unit Variance): Implemented using the authors' original reposi- tory wttps//coom/ccaaaiiiiiii
This method is applied convolutional and linear layers and requires a single batch data from the associated dataset to initialize.
All resources were used in accordance with their respective licenses and usage policies. Note that all dataset splits used correspond to the default configurations
C.2 Activation Imbalance Evaluation
This subsection provides a detailed description of the experiments underlying the theoretical insights discussed in Section 3
particularly those supporting Table and Figures a
Experiment 1: Neuron Activation States (VIT)
To analyze the activation state of neurons, we used the Vision Transformer (ViT-B/16) model trained on ImageNet-1k kk. The model was initialized using each of the initialization schemes considered in this
work, and forward pass was performed using 768 randomly sampled images from the ImageNet- 1k validation set. We captured the output of the second linear layer in the feed-forward block of the last encoder layer-i.e., the final layer before the classífer-rreultingg in a tensor of shape 768 768 (samples neurons). To enable visual representation within the paper, we subsampled this tensor to
18

---

250 neurons and 250 samples, preserving the structure and interpretability of the activation patterns. This experiment provides the data used in Figure 4
Experiment 2: Statistic S and skewness Relationship and OUI
To isolate the relation between the statistic and neuron skewness, we constructed a synthetic feedforward model consisting of 3-layer MLP with the structure: ReLU Linear ReLU Linear + ReLU + Linear. The input data consisted of random vectors sampled from a Normal distribution. This controlled setting enables clear observation of activation statisties before depth- related effects dilute the signal, as discussed in Appendix B.1
The histogram in Figure 5 was generated using the outputs from the first linear layer, where the relationship with the statistic S is still prominent. In contrast, the statistics reported in Table as the percentage of skewned neurons at significance levels a 0.1 and 0.3, and the OUI was computed from the final linear layer in the MLP, where cumulative effects of initialization are more pronounced.
We confirmed the robustness of these results by also sampling inputs from a Uniform distribution. observing consistent trends in skewness behavior and S-statistic correlation.
C.3 Benchmark Evaluation
This subsection provides the full experimental details for the training process corresponding the experiments presented Section [ and summarized in Table B and Figures 6 and [1] The
tables below detail the training hyperparameters (Table 4 and hardware (Table 5) model-dataset pair.
used for each
Table 4: Training hyperparameters for all evaluated models.
Model ResNet-50
Dataset
Epochs
Optimizers
LR
WD
Batch size
CIFAR-100
100
SGD, Adam, AdamW
10 10-3
10-3
10
64
MobileNetV3 EfficientNe1V2
Tiny-ImageNet
200
SGD, Adam, AdamW
10
10 10-3
64
Tiny-ImageNet.
200
SGD, Adam, AdamW
10-3
10-3
64
V6T-B16
ImageNet-Ik
100
SGD
10-3
10-3
64
BERT-mini
WikiText
200
AdamW
5-10
10-3
16
Table 5: Hardware and runtime per single training experiment.
Model
Dataset
GPU
Training time (hours)
ResNet-50
CIFAR-1OK
NVIDIA 1100-XXM--800
MobileNetV3
Tiny-ImageNet
NVIDIA A100-$XM4-8000
4
EfficientNetV2
Tiny-ImageNet
NVIDIA A A100-SXM4-80CS
4
VYT-B16
ImageNet-1k
NVIDIA H100-PCIc-94GB
168
BERT-mini
WikiText
NVIDIA A100-SXM4-80CH
2
D Additional experimental results
D.1 Aresine random initialization
To further validate that the key factor behind the effectiveness of our Sinusoidal initialization is the structure it imposes, rather than its unusual distribution, we performed an additional experiment using randomized initialization. In this variant, weights were sampled independently from the arcsine distribution, which matches the distribution of our Sinusoidal initialization but lacks its balanced structure.
We replicated the experiments from Figure 4 and Figure 5 using this randomized initialization. The results, shown in Figure 8 demonstrate that this initialization behaves similarly to other stochastic methods: the activations exhibit comparable neuron skewness. This confirms that the stochastic nature of the weights, not the distribution itself, is responsible for the observed effects.
19

---

Sinusnidal Random
Sinuscidal Random
Summoidal Random
Weight Donstty
Samples
Denaly
Nourons
-2.5
Wiiglts
Figure 8: Weight distribution, neuron activation states and histogram comparing the statistic values with neuron skewness at 0.3 for random initialization with arcsine distribution.
These findings make it clear that the structure introduced by the Simusoidal initialization is the critical component driving its benefits. This outcome is consistent with the theory established in Subsection 3.2 and further reinforces that the deterministic layout of weights plays a central role in mitigating activation skewness. To further support this conclusion, reported in Section 4 on ResNet-50 with CIFAR- 100 and SGD. The training results, presented in Figure 9 clearly show how the randomness of the arcsine variant delays convergence with respect to the structured Sinusoidal initialization, which provides faster training process.
replicated the experiment
ReaNet-50 on CIFAR-100
40%
Validation Accuracy
35%
30%
25%
20%
15%
50
100
Epoch
Simusoddl
Sinusoidal Random
Figure 9: Training curves of the Sinusoidal initialization and its randomized counterpart, showing that the structural arrangement of weights accelerates the initial stages of training.
D.2 Comparison with other deterministic initializations
In addition to stochastic baselines, a few deterministic initialization schemes have also been proposed in the literature, namely those introduced in Zhao al. [2022) and Blumenteld et al. [2020), already referenced in Section 2 To illustrate the relative performance of these methods, we provide representative experiment on ResNet-50 with CIFAR- using SGD, under the same hyperparameter configuration described in Section 4 While this comparison is restricted to single training setup. it
provides a direct assessment of how these deterministic strategies behave under identical conditions.
The results are shown in Figure 10
As be observed, the Sinusoidal initialization clearly outperforms both alternatives, achieving faster convergence and higher final accuracy. This experiment thus provides compelling evidence that, within the family of deterministic approaches, our method offers distinct advantage.
20

---

ResNet-50 on CIFAR-100
40%
Validation Accuracy
30%
20%
10%
50 Epoch
100
Default
Sinusoidal
Zero
Blumenfeld
Figure 10: Training curves of Default, Sinusoidal, Zero and Blumenfeld initializations, showing that
our approach consistently outperforms the other deterministic methods.
D.3 Additional Training Plots
To complement the main experimental results presented in Section include in Figure [] a detailed comparison of the training dynamics between the default and Sinusoidal initialization schemes. For each model and dataset combination, and for every optimizer used during training, plot both the loss and accuracy curves across epochs.
This visualization allows for direct assessment of convergence behavior and generalization per- formance throughout training. Notably, the figure highlights differences in early-stage convergence speed, final validation accuracy levels, and training loss smoothness under the two initialization schemes.
21

---

ResNct-30 on CIFAR-100 Train Loss
ResNet-50 on CIFAR-100 Val. Acc.
Validation Accuncyy
60%
Train Loss
40%-
20%
0
25
50
75
100
25
50
75
100
Epoch
Epoch
MobileNetV3 Tiny-Imagenet Train
MobilcNetV3 Tiny- lingeenteenet Val. Acc.
Validation Accuney
Train Loss
2-
0%
150
200
Epoch
Epoch
Efficiem[NeIV2 on Tiny-Imagener Train Loss
EfffcientNeeV2 on Tiny-emagenet Val. Ace.
Validation Accumcy
Train Loss
4-
40%
20%
0%
0
50
100
150
200
0
50
100
150
200
Epoch
Epoch
ViT B16 ImageNet-1K Troin Loss
VIT B16 ImageNet-IK Val. Acc.
Validation Accumey
30%
mm
Thain Loss
mm
10%
100
Epoch
Epoch
BERT-mini on WikiText Train Loss
BERT-mini on WikiText Val. Acc.
Validaiion Accuracy
40%
Train Loss
20%
10%
50
100
150
200
0
50
100
1550
200
ppoch
Epoch
SGD with Sinusoidal
Adam with Simusoidal
Adam/ with Simusoidal
SGD with Default initialization
Adam with Default initialization
Adam with Default initialization
Figure 11: Training loss and accuracy curves for all model-dataset pairs using default and Sinusoidal initialization. Each subplot shows the result for particular model-optimizer combination.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, as they are substantiated by both the theoretical proofs and the empirical evaluation presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper discusses the limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: This paper provides the full set of assumptions and complete, correct proofs for both theoretical results (Theorem 3 and Theorems 1 detailed in Subsection 3.2 and Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the necessary information to reproduce the main experimental results in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the implementation code in publicly available GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess htaili//guuies//oeeeubbmmims
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ pttppppllllllppieeee https///oooooooossssoooooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper fully discloses the training details reproduce the experimental results Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeated all key experiments multiple times and comment on their statistical significance in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources and time of execution of every training process is discussed in Appendix C.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpppp//.eerrrr...cc////b///laa
Answer: [Yes]
Justification: This paper does not violate the code ethics, as does not involve human participants, does not release new assets, poses negative societal impact, and uses copyrighted datasets such as ImageNetIK compliance with their respective licensing and usage requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no societal impacts beyond contributing to more effective model initialization, mentioned in Appendix A
26

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The datasets used for training and testing comply with their respective licenses and are properly cited in Section
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httppe///earrrpooccc///oeeeerere for what should or should not be described.
29


Reviews:
[{"review_id": "GEXv2yV1TK", "reviewer": "Reviewer_TVvJ", "summary": "This paper introduces Sinusoidal initialization, a deterministic weight initialization method for deep neural networks. The study challenges the tradotion that randomness is essential for effective initialization. This study highlights eliminating randomness while preserving key statistical properties. Sinusoidal initialization uses structured sinusoidal patterns to deterministically assign weights while maintaining crucial properties like signal propagation stability and gradient flow. The authors propose that the activation imbalance phenomenon is the key to training dynamics; such Sinusoidal initialization introduces deterministic weight distribution, eliminates cumulative weight imbalance and in the end cultivates activation balance. Through theoretical analysis and extensive empirical evaluation across CNNs, Vision Transformers, and language models, the paper demonstrates that Sinusoidal initialization leads to faster convergence, higher final accuracy, and more balanced activation patterns compared to standard stochastic methods.", "questions": "1. It would also be important to consider the quantization of deep neuron networks. Would a model initialized with sinusoidal functions be difficult to quantize because of different weight distribution? I expect to see more evaluations.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The idea is simple and clear. The proposed approach is very easy to understand and implement.\n2. The discussion of activation balance is very inisightful and helpful to analyse performance of deep neuron networks.\n3. The method is evaluated on a wide range of models.\n\nWeaknesses:\n1. It would be better to testify the generalization ability of Sinusoidal initialization on more benchmarks and datasets. For example, the paper only reports the convergence curve of BERT, but lacks performance on downstream benchmarks such as GLUE. It would also be helpful to verify on other optimizers, such as Lion&ViT.", "was_revised": true, "final_justification": "I suggest Accept. My primary considerations about this paper are:\n\n1. The paper presents a brand new aspect to deep neuron network initialization. The experiment convers different tasks and model architectures and the improvement is inspiring. This is the most import reason of my rating.\n2. Evaluation on downstream benchmarks like GLUE and evaluations on Lion optimizer is not currently not addressed due to computational cost. This is not essential and I personally believe would not diminish the advantage of the method.\n3. Discussions on other related questions (quantization, convergence on specific optimizers, etc.) is an expected improvement of this study, but not a drawback.", "rebuttal": "We sincerely thank the reviewer for the very constructive feedback. We are genuinely grateful for the kind words and we appreciate the questions and suggestions, which will help us further improve the manuscript.\n\n----\n\n## **Weakness 1: Evaluation on downstream benchmarks like GLUE**\n\nWe fully agree that downstream evaluations such as GLUE are essential in many contexts, especially when analyzing the performance of pre-trained models or evaluating methods like quantization or pruning. However, our setup differs slightly in scope. Our work focuses on **training from scratch**, not fine-tuning, and aims to understand the behavior of different initializations in this regime. Specifically, we assess whether Sinusoidal initialization improves convergence speed and optimization dynamics without sacrificing final accuracy, or even improving it.\n\nIn this context, we believe the fairest comparison is to evaluate performance on a **validation set drawn from the same distribution** as the training data, using standard train/validation splits. Since our models are trained from scratch on datasets such as WikiText, their performance on downstream tasks like GLUE would likely be poor regardless of the initialization strategy. For this reason, we consider that GLUE is not the most appropriate benchmark for our goals, as it may combine the effects of limited training data with those of initialization. We hope the reviewer agrees that, given our specific focus, the current experimental setup remains reasonable and controlled.\n\n----\n\n### **Still under Weakness 1: Evaluating with Lion optimizer**\n\nWe sincerely appreciate the reference to Lion and thank the reviewer for highlighting it. While we chose to focus our experiments on more standard and widely used optimizers to ensure comparability and clarity, we agree that **Lion is an interesting choice** for evaluating initialization schemes. Unfortunately, given our **limited computational resources**, we do not believe it is feasible to conduct a thorough evaluation with Lion across all initializations within the rebuttal period. That said, we **genuinely intend to explore this direction in future work**, particularly in the context of ViT models, where this optimizer could provide valuable additional insights. We are committed to launching these experiments as soon as possible and to reporting the results in a future version of the manuscript.\n\n----\n\n## **Question 1: Quantization**\n\nWe had not explicitly considered the implications of our initialization for **post-training quantization**, since the method mainly affects the initial stages of training. That said, we conducted preliminary experiments and observed that models initialized with sinusoidal functions behave similarly to those using standard initializations in terms of post-training dynamic **quantization robustness**. In particular, the final weight distributions after training do not differ significantly in ways that would hurt quantization performance. We did not observe any systematic degradation, though we acknowledge that this topic deserves further investigation, especially across a wider range of model architectures. We thank the reviewer again for this perspective and will explore it further in **future work**.  \n\n----\n\n**We thank the reviewer once again for the constructive feedback.** The comments highlighted **valuable directions** to take into account in future investigation. While some suggestions could not be fully addressed due to resource limitations, we are committed to **exploring them** in follow-up work."}, {"review_id": "J2m34TRxBy", "reviewer": "Reviewer_Lp9r", "summary": "This paper proposes Sinusoidal initialization, a novel deterministic initialization scheme for deep neural networks. Unlike traditional stochastic methods (e.g., Glorot or He), this approach initializes weight matrices using sinusoidal functions with carefully selected frequencies and phases to ensure symmetry, variance preservation, and functional diversity. The authors provide both theoretical and empirical evidence that Sinusoidal initialization reduces neuron skewness, improves activation balance, and promotes more expressive and efficient signal propagation from the first layer onward. Across a broad range of architectures—including CNNs, Vision Transformers, and BERT—the proposed method demonstrates consistent improvements in convergence speed and final accuracy. The authors report a 4.8% average increase in final validation accuracy and a 20.9% increase in convergence speed over standard initializations.", "questions": "1. The paper omits discussion of prior deterministic methods like [1] and [2], which also aim to eliminate randomness during initialization. Can the authors clarify key differences and provide comparisons to these works?\n\n2. The method is only tested on small to medium-scale models. Have the authors tried it on larger models (e.g., LLaMA 130M+) or foresee challenges in doing so?\n\n3. Deterministic init should improve reproducibility. Does this hold across hardware or frameworks? Also, does sinusoidal weight construction add any runtime or memory overhead?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. A fully deterministic init is interesting, and would be very useful for removing randomness and improve reproducibility. \n\n2. Paper is clearly written, with strong organization and thorough motivation. \n\n3. The paper includes promising empirical results compared to standard random init methods, over some simple to medium tasks such as ResNet on ImageNet and training BERT-mini models. \n\nWeakness\n\n1. The paper lacks discussion to previous deterministic initialization papers. This is not the first work proposing deterministic initialization. Works such as [1] and [2] already proposed deterministic and tried to address randomness issue during initialization. Would be great to discuss the similarity and difference, and also compare with them in practice.\n\n[1]  https://arxiv.org/abs/2110.12661\n[2] https://arxiv.org/abs/2007.01038\n\n2. Limited scale of empirical results: the paper mainly focuses on simple to medium tasks such as ResNet on ImageNet and training BERT-mini models. However, I believe nowadays some large-scale experiments are needed to demonstrate its effectiveness, such as pre-training on 100M-level LLM model (such as Llama 130M, 350M).", "was_revised": true, "final_justification": "The authors addressed my concerns and questions, and thus I would like to raise my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and valuable comments. We genuinely appreciate the time and insight offered, which have helped us identify concrete ways to improve the **quality** and **scope** of our work.\n\n----\n\n## **Question 1 (Weakness 1\\): Related work on deterministic initialization**\n\nWe sincerely thank the reviewer for pointing out these references, which helped us improve the **contextualization** of our work. While we were already aware that deterministic alternatives to random initialization have been explored, we were not familiar with the specific works cited. That said, it is important to stress that our objective is **not to position this work as the first deterministic initialization**, so that **we are implementing focused revisions** in the manuscript to ensure this point is clearly reflected. Rather, we offer a **theoretical explanation** of why stochastic initialization can hinder learning, through mechanisms such as activation imbalance and neuron skewness, and to propose a **constructive alternative** grounded in that theory. Sinusoidal initialization is not only new in form but also in its guiding motivation: **improving** **accuracy** and **convergence speed** through eliminating skewed neurons from the initial stage.\n\nAlthough all deterministic schemes aim to remove randomness, the cited works **differ in** **intent and scope**. For instance, *ZerO* focuses on stability in extremely deep networks, while *Blumenfeld et al.* center on conceptual replacement of randomness, without assessing learning dynamics. To the best of our knowledge, **neither approach** reports improvements in training speed or accuracy. **This sets our work apart**. We believe that the inclusion of these references, rather than weakening our contribution, helps to **clarify its distinct value**. In particular:\n\n* We evaluate **convergence time**, a notoriously challenging metric to improve, and overlooked in prior work on deterministic initializations. Its relevance, however, continues to grow due to the **substantial environmental and economic impact** of training large models. Even modest improvements in convergence time can translate into **meaningful savings in energy consumption and carbon emissions**, which underscores the significance of our results.  \n* We provide a formal theoretical framework that not only motivates the structure of our solution but also offers a **quantitative explanation** of a key phenomenon: *the emergence of skewed neurons*. Unlike most theoretical contributions in the field, ours leads to **a precise, measurable criterion** and is supported by a series of theorems that show how Sinusoidal initialization preserves this balance from the start.\n\nHowever, we fully acknowledge the relevance of the cited works. After carefully reviewing them, we are  **including a** **dedicated paragraph in the *Related Work* section** discussing their contributions and positioning them in relation to ours. We are also conducting experiments to directly compare these methods with our Sinusoidal initialization, and are **including the results in the revised *Experiments* section**. At the same time, we are confident that **the originality of our approach, its theoretical depth, and its remarkable empirical results stand on solid ground and offer a meaningful contribution**.\n\n----\n\n## **Question 2 (Weakness 2\\): Scale and larger models**\n\nWe fully agree that **evaluating scalability on very large models** is crucial to assess the robustness and generality of any initialization method. We would genuinely like to perform such an evaluation, but at this stage we simply **cannot afford it**. Our **computational resources are limited**. As a result, we are **unfortunately unable to commit to running these experiments in time for the current submission**. That said, we are keen to explore this direction and **will prioritize it as future work**, since deeply understanding how Sinusoidal initialization scales and how it can be improved in transformer-based architectures is one of our main research goals moving forward.\n\n----\n\n## **Question 3: Reproducibility and overhead**\n\nWe agree that deterministic initializations can help improve **reproducibility**. While fixing random seeds is a common practice, it often fails across different hardware platforms or framework versions. For example, operations may not be **bit-exact** across different GPUs or kernel implementations, leading to variation **even with identical seeds**.\n\nOur method removes randomness from initialization entirely, ensuring **consistent starting points** across setups. While later training stages remain stochastic (e.g., sampling, dropout), the **initialization phase becomes fully stable,** provided that the same arithmetic precision is used. As such, any reproducibility variation observed after initialization stems from the optimizer and training pipeline, not the initial weights.\n\nRegarding **overhead**, Sinusoidal initialization is computed entirely **a priori**, as a one-time offline procedure. It only involves evaluating sinusoidal functions, which is computationally inexpensive and introduces zero memory overhead and **negligible** runtime cost compared to training, even when compared to a single forward-backward pass over one batch. This contrasts with **data-dependent initializations** (e.g., LSUV), which require a forward pass through a batch of data, adding latency and memory footprint (though still minor relative to full training).\n\n----\n\nWe hope that the **revised manuscript** including the suggested works, both in the theoretical discussion and in the experimental comparison, **reflects a more complete and compelling contribution**. We believe that the originality of our approach as well as our theoretical and empirical results **make it a valuable contribution to the field**. While we acknowledge the limitations in scaling experiments due to our constrained computational resources, **we trust that the contribution will be judged primarily on its scientific merit**, and not on our group’s hardware access. We remain committed to further exploring the scalability of our method in future work and hope **the community will recognize the potential and rigor of this proposal**."}, {"review_id": "yAOQPcUPuU", "reviewer": "Reviewer_zqjc", "summary": "This paper introduces Sinusoidal Initialization, a deterministic alternative to traditional random weight initialization methods (e.g., HeInit) for deep neural networks. The core challenge addressed is the variability and activation imbalance caused by randomness, which hinders convergence stability and generalization.", "questions": "refer to weakness", "limitations": "refer to weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \n- Sinusoidal Initialization employs structured sine functions to construct weight matrices, ensuring each neuron receives a distinct, oscillatory weight configuration. This deterministic approach eliminates randomness while preserving the critical variance-scaling principle for signal propagation.  \n- It emerges as a rational choice for practitioners: 1. Each row of the initialized matrix sums to zero, eliminating inherent bias in neuron activations. 2. Theoretical analyses link weight imbalance to skewed neuron activation, proving that deterministic initialization mitigates this issue.  \n- Empirical experiments across CNNs, Vision Transformers, and language models demonstrate higher final validation accuracy and faster convergence than baselines. Additionally, drastic reductions in \"skewed neurons\" are confirmed via visual activation patterns and OUI scores.  \n- The work challenges the long-held assumption that randomness is essential for effective initialization. By replacing stochastic sampling with structured sine patterns, Sinusoidal Initialization do achieves some critical advancements\n  1. Reproducibility: Determinism eliminates trial-to-trial variability, aiding debugging and systematic research.  \n  2.  Structured weights prevent neurons from being persistently active/inactive, which is particularly valuable for sparse-activated large language models in industrial applications.  \n  3.  It provides theorems formalizing the link between weight imbalance and skewed activations, the statements in Theorem1, 3 are fundamentally sound to me, leveraging the Lindeberg-Feller CLT to establish threshold equivalence.\n\n**Weaknesses**  \n- The proposed method shows reduced effectiveness in Table 3 results for ViT on imagenet and BERT on wikitext. While effective for standard architectures, deeper networks or other task-specific models may require adapted sine patterns. It remains unclear whether this initialization technique can benefit larger models, which are increasingly trending in current research and applications.  \n- Integration with data-dependent initialization techniques could further optimize early training dynamics ?\n- Theoretical extensions to non-rectified activations remain unexplored. The analysis of skewness propagation across layers appears to align well with the claims in the main text and is intuitively plausible. However, the models considered seem rather simplistic. I am curious whether common training techniques—such as weight normalization and self-rectified activation functions (e.g., Swish, SwiGLU)—could help address the issues of stochastic initialization.\n- I partially agree with the analogy between the sinusoidal structure in weight initialization and the sinusoidal PEs in Transformers (line 130). Viewing the weight matrix as a LUT from a memory perspective might be helpful, as it hints at potential generalization benefits for extrapolation in deep learning tasks—similar to how sinusoidal PEs enable Transformers to generalize to sequence lengths beyond training. However, this connection lacks elaboration in the paper, and expanding on it could strengthen the persuasiveness of the proposed technique.", "was_revised": true, "final_justification": "I have fully followed up on and participated in the revision of the paper, as well as the discussions with the authors and reviewers, and I have made the decision to maintain the original score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful review. We greatly appreciate the level of insight and care in the feedback, which has helped us improve the article.\n\n----\n\n## **Weakness 1: On performance in ViT and BERT models**\n\nWe acknowledge that the gains in ViT and BERT models, particularly in Table 3, are not as pronounced as in other architectures such as ResNet or MLP-based models. However, we emphasize that even in these cases, **Sinusoidal initialization does not underperform** relative to common baselines. Indeed, it still **outperforms the widely used default initialization**, which is typically employed in standard training setups and was originally proposed in the papers that introduced these architectures. It also performs comparably to orthogonal initialization in both convergence speed and final validation accuracy. We believe this is already a positive result, especially given the prevalence of these baselines in large-scale systems. That said, we emphasize that **this is not a scalability issue** or a consequence of model size. Rather, we found that the specific structure of MHA layers in Transformers **may interfere** with the balance properties that Sinusoidal achieves in MLPs. In these cases, the non-linearities and reweighting introduced by the MHA layers can **distort input statistics**, weakening the assumptions under which our method is most effective. This suggests that some of the architectural peculiarities of Transformers still **deserve further investigation** in the context of deterministic initialization.\n\nTo address this, we are actively investigating a specialized initialization for MHA components that would **preserve neuron balance even across attention heads**, allowing the signal propagation properties of Sinusoidal initialization to extend deeper into Transformer pipelines. The inherent complexity of this task makes it a promising direction for **future work**, but we believe the current results and theory **already provide a solid contribution**. The strong empirical evidence makes this method an attractive initialization choice ready for publication.\n\n----\n\n## **Weakness 2: On possible integration with data-dependent techniques**\n\nIt is important to emphasize that **data-dependent initialization techniques tend to face limited adoption** due to the difficulty of integrating them broadly across different models. Therefore, an initialization method that **focuses solely on layer-wise structural information**, requiring only the weight dimensions to initialize, has much greater potential to become a meaningful and widely adopted change within the research community.\n\nIn this context, the idea we discussed earlier about **specialized initialization for MHA layers** is closely related. Furthermore, this approach can be extended to **adapt the initialization of the subsequent MLP layers** to this reconfiguration, effectively compensating for the asymmetric input distributions generated by attention layers, **without needing a full forward pass** as conventional data-dependent methods do.\n\nOur theoretical results, specifically **Theorem 3**, demonstrate that **skewness in activations arises when a layer’s input has coordinate-wise unequal means**, breaking the assumptions required for neuron balance. This is precisely what often happens after passing through an attention layer.\n\nFor these reasons, we consider that developing a **locally data-dependent initialization at the layer level**, which adjusts weights to counteract this asymmetry, is a **feasible and promising direction**. However, due to its complexity and our current computational constraints, we position this as an important avenue for **future work**.\n\n----\n\n## **Weakness 3: On extensions to non-rectified activations**\n\nWe thank the reviewer for this observation. The theoretical framework we develop is indeed **not limited to ReLU**, but it applies more broadly to what we refer to as **“ReLU-like” activation functions**, such as **GELU, SiLU, PReLU**, etc.\n\nThese functions exhibit two markedly different regimes:\n\n* One in which the output is **significantly positive** (active),  \n* And another in which the output is **zero or a small negative value** (inactive).\n\nWhile ReLU yields an exact zero in the inactive regime, the other functions return **residual negative values**. These contribute to mitigating issues like **dying neurons** and facilitate training, but their magnitude is typically small enough that they can **reasonably be considered inactive** for the purposes of our theory. This is why our analysis **generalizes naturally to this family of activations**.\n\nWe acknowledge, however, that this point is **not made sufficiently explicit** in the current version of the paper. We **are adding a clarifying note** **in the revised manuscript** to make this generality more evident to the reader.\n\n----\n\n## **Weakness 4: On the analogy with positional encodings**\n\nWe appreciate the reviewer’s observation regarding the analogy between Sinusoidal initialization and **PEs** in Transformers. In fact, the inspiration for our initialization came from the idea behind PEs: assigning **maximally independent vectors** to different inputs using harmonic functions. The main difference lies in the **ordering of frequencies**, as PEs employ **decreasing frequencies**, while our Sinusoidal initialization uses **increasing ones**.\n\nBoth, however, leverage the **expressive power of harmonic components** (a fact long understood since Fourier). We initially mentioned this resemblance only briefly, in order to keep the exposition lightweight. However, based on this helpful feedback, we **are expanding this part of the discussion** in the revised version of the paper to make the connection clearer and help the reader build a more **intuitive understanding** of the expressive potential of Sinusoidal initialization.\n\n----\n\n**In closing**, we once again thank the reviewer for the **precise and constructive feedback**. These comments are helping us improve both the **scientific rigor and clarity** of our contribution, and we are **grateful** for the opportunity to incorporate these insights into our work."}, {"review_id": "G8lhEATVUn", "reviewer": "Reviewer_kR57", "summary": "In this paper the authors propose a deterministic initialization scheme for neural networks that removes randomness and improves the convergence and final accuracy characteristics by lowering the number of \"skewed\" neurons. Authors provide both theoretical justifications as well as some experimental results to support their claims.", "questions": "-  Authors claim that randomness is a negative aspect in initialization. Could the authors provide stronger evidence or justification for this claim? How do you account for the possibility that stochastic initializations achieving the same target distribution (e.g., arcsine) might perform similarly?\n\n- How do the authors explain the substantial drop in performance for MobileNetV3 with SGD (28% to 21%)? Are there specific conditions where your method is expected to degrade performance?\n\n- Why was ImageNet-1k only evaluated with a ViT transformer? \n\n- Can the authors clarify under what circumstances or configurations you expect your proposed initialization method to improve performance? Are there cases where it should not be applied?\n\n- How does the impact of  the proposed initialization change with smaller architectures? Have the authors conducted any experiments on such architectures?\n\n- In Table 3, authors report \"Maximum accuracy\" but label it as \"accuracy at convergence.\" Could the authors clarify this discrepancy?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "First, I would like to say that I really enjoyed reading this paper. In the saturated landscape of minor changes over existing approaches, the paper provides a new perspective into a less studied and often overlooked aspect of deep neural networks that could have a significant impact on the performance of the models. The paper is clearly written and easy to follow. Authors present all concepts in a clear and very understandable way and provide appropriate motivation for their work.\n\nHowever, despite really liking the paper and the idea, I am afraid that the paper is significantly lacking in the validation part. Authors only evaluated five different combinations of architectures/datasets. Even for these selected combinations the demonstrated performance improvements do not always match the authors’ claims, apart from the initial convergence acceleration. More specifically, I have the following comments:\n\n- Authors claim at various points that randomness is actually a negative aspect in the initialization of networks. I am quite skeptical of this claim and I would like to invite authors to revisit this claim. This work demonstrates that a more careful initialization scheme can reduce the number of skewed neurons, leading to better performance. The method happens to be deterministic, yet this does not prove that stochastic initialization cannot achieve the same results. For example, it might be reasonable to assume that any initialization that leads to the same arcsine distribution (as shown in Fig.2) would probably perform the same, unless authors provide evidence against this.\n\n- ImageNet-1k is only evaluated in a ViT transformer. I think evaluating a ResNet architecture on this dataset is essential and shouldn't be more computationally complex. This is a major concern.\n\n- In a few cases, the performance is substantially lower. For example, for a MobileNetV3 using SGD, the max accuracy is reduced from 28% to 21%. This result, along with the lack of extensive experiments, creates some questions on whether the authors' claims are universally true or are valid only in some cases. Authors should better indicate when they expect improvements by applying the proposed initialization approach. It might be useful to connect the performance improvements to the designed metrics (e.g., skewed neurons).\n\n- Furthermore, not all networks seem to have converged, with Adam/AdamW-trained networks showing substantially better performance. I understand that authors, in their desire to have a fair setup, might have reduced the amount of hyper-parameter tuning performed. However, the vast differences between the employed configurations for the same architecture/dataset show that some setups might be far from optimal and some networks might not have yet converged. Therefore, some of the results might actually still show the performance at more initial stages of the convergence instead of the final one.\n\n- The paper lacks evaluation experiments on smaller architectures. Does the impact of initialization diminish as the architecture gets smaller? This is a minor issue.\n\n- In Table 3 authors claim that they report accuracy \"at convergence,\" yet \"Maximum accuracy\" is reported, which is not the same as the accuracy at convergence, if I understand the concepts correctly. This is a minor issue.\n\nEven though it seems that the results provided in the paper might be a bit premature, I would tend towards acceptance, unless there are also important concerns raised by the rest of the reviewers.", "was_revised": true, "final_justification": "Overall, I find this paper inspiring and would like to see it accepted. Authors clarrified one important concern that I had and I am increasing my rating in response. The validation could be stronger, as the other reviewers noted.", "rebuttal": "We sincerely thank the reviewer for the extremely thoughtful comments, both the positive ones and the critical ones. We truly appreciate the encouraging words about the **clarity** and **motivation** of the paper, and we are equally grateful for the **constructive feedback**, which will undoubtedly help us improve the work further. \n\n----\n\n## **Question 1: On the role of randomness in initialization**\n\nWe completely agree with this comment. In fact, we included an appendix discussing a stochastic initialization that also follows the arcsine distribution, similarly to our deterministic proposal, and we showed theoretically that this variant still results in a non-negligible proportion of skewed neurons. While we have not yet implemented or tested this variant empirically, we plan to conduct **small- and medium-scale experiments** to validate this behavior in practice and to **report the results in the corrected version** in the same appendix where this topic is addressed. We hope this will help clarify, both theoretically and experimentally, that the key factor is not the distribution itself, but rather the **disorder induced by the stochastic process**, which leads to the emergence of skewed neurons.\n\n----\n\n## **Question 2: On evaluating ResNet on ImageNet-1K**\n\nAgreed, and we appreciate the suggestion. While we fully recognize the value of including a ResNet experiment on ImageNet-1K, we also note that space constraints and the need to balance coverage across different architectures led us to focus on a **selected set of pairs of models and datasets**. Our goal was to include representative scenarios rather than exhaustively testing all combinations. That said, we consider this an excellent **addition for future work** or extended versions of the manuscript. \n\n----\n\n## **Question 3: On the drop in performance for MobileNetV3 with SGD**\n\nThis is a fair point. We have identified that the main issue here is the **Swish activation** used in MobileNetV3. We repeated the same experiments using *ReLU* instead, and observed **clear superiority of Sinusoidal** against the other initialization methods. We decided to keep the original experiment in the results section to remain fully transparent; while we could have omitted it, we believe doing so would have missed the point. This case highlights an interesting interaction that deserves **further investigation**. In general, our theoretical insights suggest that controlling the proportion of skewed neurons leads to **improved convergence speed and final accuracy** by initializing the network in a more favorable regime. However, this interaction is still under investigation, especially in architectures where the non-linearities or structural elements deviate from the *linear \\+ activation* pattern. We are **including in the manuscript an annotation** discussing the results obtained with MobileNet using *ReLU*, along with a note addressing the considerations mentioned above. Despite this, we believe the strong results on the other architectures and tasks are sufficient to consider this a **mature and promising contribution to the literature**.\n\n----\n\n## **Question 4: On possible lack of convergence in some experiments**\n\nIn all our setups, to fairly compare initializations using AUC across different runs, we ensure that all models are trained **for the same number of epochs**. For each combination of model, dataset, and optimizer, training is performed with early stopping to **guarantee convergence** for every initialization. Once the slowest run has converged, the remaining models continue training for the same number of additional epochs until reaching the maximum epoch count. This procedure ensures that (1) all trainings **reach their maximum validation accuracy**, and (2) the number of epochs used to calculate the AUC for measuring convergence speed **is consistent** across initializations. The differences in final performance across optimizers are expected: some optimizers simply reach better minima than others. However, our intention is to compare initializations for fixed optimizers, rather than comparing optimizers per se. We are adding a **clarification to the paper** describing this methodology to prevent any possible misunderstanding about whether the experiments have properly converged.\n\n----\n\n## **Question 5: On the absence of small-scale architectures**\n\nWe appreciate this comment and are going to add an experiment using *LeNet-5* on CIFAR-10 to confirm that our initialization remains beneficial at a small scale. Although the absolute gains in time or accuracy are smaller in such settings, this experiment supports the **down-scaling robustness** of our method.\n\n----\n\n## **Question 6: On the use of *“accuracy at convergence”* in Table 3**\n\nYou’re right, the table should say *“maximum accuracy”*, as we are reporting the best validation accuracy observed (tracked via callbacks). We are **correcting this label** in the final version to avoid confusion.\n\n----\n\n**In summary, we thank the reviewer once again for the detailed and constructive feedback.** We have addressed each point carefully, clarifying our choices and incorporating additional experiments or corrections where feasible. While some limitations remain due to space and computational constraints, we believe the current version presents a **well-motivated and coherent contribution**. We also see several of the reviewer’s suggestions as valuable directions for future work, and we are committed to pursuing them. We hope our responses help reinforce the clarity, scope, and significance of the work."}]

Author rebuttal:

Meta-review: This paper presents Sinusoidal initialization, a deterministic weight initialization scheme for deep neural networks. The method leverages sinusoidal functions to construct structured weight matrices and address the variability and activation imbalance caused by randomness. Faster convergence, better training stability and accuracy are observed in the experiments with various model architectures. Reviewers recognize the interesting idea, clear organization and presentation, and promising experimental results. While some reviewers show concerns regarding the limited scale of empirical verification, they deem this as a minor issue and hold a positive overall evaluation for this paper. After rebuttal, all reviewers acknowledge their questions and concerns resolved. The AC also recognizes the contributions and quality of this paper, and thus recommends Accept.
Decision: Accept (poster)