SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The idea behind Assumption 3 is that different data points admit a small similarity. To be specific, for two data points  $X_{n}$  and  $X_{n'}$  with  $n \neq n'$ , their inner product reflects the similarity of their respective (empirical) covariance matrix. We expect that this similarity is small with a high probability. The spirit of this assumption also exists in previous literature, e.g., Nguyen et al. [2021]. The constant  $\hat{c}$  determines the decay of data dissimilarity. A larger  $\hat{c}$  results in less separable data. Our assumption has no requirement on  $\hat{c}$  such that  $\hat{c}$  can be small enough, which allows for a general data distribution.
Verification of assumption. Here we experimentally validate this assumption under a standard language IMDB dataset [Maas
Figure 1: Validation of Asm. 3.
et al., 2011]. We randomly sample 100 (normalized) sentences with embedding and plot the probability (frequency) of  $\mathbb{P}\left(|\langle \pmb{X}_n^\top \pmb{X}_n, \pmb{X}_{n'}^\top \pmb{X}_{n'}\rangle| \geq t\right)$  as  $t$  increases. We repeat it over 10 runs and plot the result in Figure 1. We can observe an exponential decay as  $t$  increases, which implies that our assumption is fair. Besides, the validation of Assumption 3 on image data, e.g., MNIST by ViT, is deferred to Appendix E.2.
Now we are ready to present our main theorem under the  $d_m^{-1/2}$  setting. The proof is deferred to Appendix C.3.
Theorem 1. Under the setting of LeCun/He initialization in Table 1 and Assumptions 1 to 3, suppose  $d_{m} \geq d$ , and given  $\tau_0 = d_m^{-1/2}$ ,  $d_{m} = \tilde{\Omega}(N^{3})$ , then with probability at least  $1 - 8e^{-d_m / 2} - \delta - \exp(-\Omega((N-1)^{-\hat{c}}d_s^{-1}))$  for proper  $\delta$ , the GD training of Transformer converges to a global minimum with sufficiently small step size  $\gamma$  as in Eq. (4).
Remark: The probability relates to several randomness sources, e.g., data sampling, dissimilarity of data, and parameter initialization. The quantity  $\exp\left(-\Omega((N-1)^{-\hat{c}}d_s^{-1})\right)$  can be small for a small enough  $\hat{c}$  as discussed in Assumption 3. Further discussion on our result refers to Section 4.4 for details. Besides, our proof framework is also valid for the  $\tau_0 = d_m^{-1}$  setting. We demonstrate that  $d_m = \tilde{\Omega}(N^2)$  suffices to achieve global convergence, see Appendix C.3 for details.
Proof sketch: To check whether the conditions in Eqs. (2) and (3) hold, the key idea is to provide the lower bound of  $\alpha$ . Then we upper bound  $\bar{\lambda}_Q, \bar{\lambda}_K, \bar{\lambda}_V, \bar{\lambda}_O$  based on concentration inequalities to upper bound the initial loss, one key step is to utilize Gershgorin circle theorem [Gershgorin, 1931] to provide a lower bound for  $\alpha$ . Lastly, we plug these bound into the condition Eqs. (2) and (3) in order to obtain the requirement for the width  $d_m$ .
4.3 NTK initialization under the  $d_m^{-1}$  setting
The NTK theory, as a representative application of the  $\tau_0 = d_m^{-1}$  setting, can be also used for analysis of training dynamics. We also include the NTK results in this section for self-completeness. In this section, we first derive the limiting NTK formulation of Transformer under the  $d_m^{-1}$  scaling scheme, and then show the global convergence of Transformers.
Lemma 1. Denote  $\Phi^{\star} := \left[\frac{1}{d_s}\pmb{X}_1^\top\pmb{1}_{d_s},\dots,\frac{1}{d_s}\pmb{X}_N^\top\pmb{1}_{d_s}\right]^\top \in \mathbb{R}^{N\times d}$ , then the limiting NTK matrix  $\pmb{K} \in \mathbb{R}^{N\times N}$  of Transformer under the NTK initialization with  $\tau_0 = d_m^{-1}$  has the following form:
$$
\boldsymbol {K} = d _ {s} ^ {2} \mathbb {E} _ {\boldsymbol {w} \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I})} \left(\sigma_ {r} \left(\boldsymbol {\Phi} ^ {\star} \boldsymbol {w}\right) \sigma_ {r} \left(\boldsymbol {\Phi} ^ {\star} \boldsymbol {w}\right) ^ {\top}\right) + d _ {s} ^ {2} \mathbb {E} _ {\boldsymbol {w} \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I})} \left(\dot {\sigma} _ {r} \left(\boldsymbol {\Phi} ^ {\star} \boldsymbol {w}\right) \dot {\sigma} _ {r} \left(\boldsymbol {\Phi} ^ {\star} \boldsymbol {w}\right) ^ {\top}\right) \left(\boldsymbol {\Phi} ^ {\star} \boldsymbol {\Phi} ^ {\star^ {\top}}\right).
$$
Remark: The formulation of  $\Phi^{\star}$  implies that the self-attention layer degenerates as  $\frac{1}{d_s} \mathbf{1}_{d_s \times d_s} \mathbf{X} \mathbf{W}_V^\top$ , i.e., the dimension missing effect as mentioned before.
Now we are ready to present our convergence result under the  $d_m^{-1}$  scaling with the proof deferred to Appendix C.4.
Theorem 2. Under the setting of NTK initialization in Table 1 and Assumptions 1 to 3, suppose  $d_m = \tilde{\Omega}(N)$ , then with probability at least  $1 - 8e^{-d_m / 2} - \delta - \exp(-\Omega((N - 1)^{-\hat{c}}d_s^{-1}))$ , the GD training of Transformer converges to a global minimum with sufficiently small  $\gamma$  as in Eq. (4).
Proof sketch: The overall idea is the same as the proof of the previous theorem, i.e., we need to provide the lower bound of  $\alpha$ . However, in this case, the limit for the output of softmax exists so that we can apply concentration inequality to lower bound the  $\alpha$ . Lastly, we plug these bound into the condition Eqs. (2) and (3) in order to obtain the requirement for the width  $d_{m}$ .
Besides, the stability of NTK during training allows us to build a connection on training dynamics between the Transformer (assuming a squared loss) and the kernel regression predictor. Next, in order to show that the NTK is stable during GD training, below we prove that the spectral norm of Hessian is controlled by the width.
Theorem 3 (Hessian norm is controlled by the width). Under Assumption 1 and scaling  $\tau_0 = d_m^{-1}$ , given any fixed  $R > 0$ , and any  $\pmb{\theta}^t \in B(\pmb{\theta}, R) \coloneqq \{\pmb{\theta} : \| \pmb{\theta} - \pmb{\theta}^0 \|_2 \leq R\}$ ,  $\pmb{\theta}^0$  as the weight at initialization, then with probability at least  $1 - 8e^{-d_m / 2}$ , the Hessian spectral norm of Transformer satisfies:  $\| \pmb{H}(\pmb{\theta}^t) \|_2 \leq \mathcal{O}(d_m^{-1 / 2})$ .
Remark: By [Liu et al., 2020, Proposition 2.3], the small Hessian norm is a sufficient condition for small change of NTK. Thus, Theorem 3 can be an indicator for the stability of NTK. Besides, Theorem 3 supplements the result in [Park and Kim, 2022] which exhibits empirically a relationship between the Hessian norm and the width but lacks theoretical proof.
4.4 Discussion on convergence results
We compare the derived results under different scaling schemes, initializations, and architectures.
Comparison of scaling schemes: The global convergence can be achieved under both  $\tau_0 = d_m^{-1/2}$  and  $\tau_0 = d_m^{-1}$ . Nevertheless, as suggested by our theory, for a small  $d_m$ , there is no significant difference between these two scaling schemes on the convergence; but for a large enough  $d_m$ , the  $\tau_0 = d_m^{-1/2}$  scaling admits a faster convergence rate of Transformers than that of  $\tau_0 = d_m^{-1}$  due to a more tight estimation of the lower bound of  $\alpha$ , see Appendix C.6 for details. The intuition behind the lower convergence rate under the setting of large width and  $\tau_0 = d_m^{-1}$  is that the input of softmax is close to zero such that softmax roughly degenerates as a pooling layer, losing the ability to fit data. This can be also explained by Lemma 1 from the perspective of dimension missing: self-attention  $(X)$  degenerates as  $\frac{1}{d_s} \mathbf{1}_{d_s \times d_s} X W_V^\top$ .
The result under the  $\tau_0 = d_m^{-1}$  setting requires weaker over-parameterization than the  $\tau_0 = d_m^{-1/2}$  setting. Nevertheless, we do not claim that  $\tau_0 = d_m^{-1}$  is better than  $\tau_0 = d_m^{-1/2}$ . This is because, under the over-parameterization regime, the scaling  $\tau_0 = d_m^{-1}$  makes the self-attention layer close to a pooling layer. This analysis loses the ability to capture the key characteristics of the self-attention mechanism in Transformers.
Note that the lower bound of the minimum eigenvalue is in the constant order, which is tight (since it matches the upper bound). Based on this, by studying the relationship between  $d_{m}$  and  $\lambda_0$ , we can prove that quadratic (cubic) over-parameterization is required for  $d_m^{-1}(d_m^{-1 / 2})$  scaling. This quadratic over-parameterization requirement could be relaxed if a better relationship is given while it is still unclear and beyond our proof technique.
Comparison on initializations: Though our results achieve the same convergence rate under these initialization schemes, we can still show the separation on  $\alpha$  that affects the convergence in Eq. (4) among these initialization schemes. To be specific, we verify that under LeCun and He initialization, the lower bound of  $\alpha^2$  is tighter than that of NTK initialization, and hence admits faster convergence, see Appendix C.5 for further details. This can be an explanation of the seldom usage of NTK initialization in practice. Besides, the NTK initialization scheme allows for a larger step size than LeCun/He initialization for training. That means, if  $\alpha$  is the same in these three initialization schemes, we usually choose a large step size under the NTK initialization scheme, see Appendix E.1.
Comparison on architectures: Note that the Transformer defined in Eq. (1.2) includes a self-attention layer, a feedforward ReLU layer, and an output layer. Our result proves that a cubic (quadratic) over-parameterization condition is required for  $d_m^{-1/2}$  ( $d_m^{-1}$ ) under LeCun initialization. As a comparison, a three-layer fully-connected ReLU network under LeCun initialization requires  $d_m = \tilde{\Omega}(N^3)$  [Nguyen, 2021].
The aforementioned result holds for matrix input. Although not as frequent, some data inputs are naturally in vector form. Two ways to feed the input into Transformer are either formulating along sequence dimension  $(d = 1)$  or along embedding dimension  $(d_{s} = 1)$ . The following result shows the separation under these two settings to understand when the Transformer works well or not regarding the input.
Corollary 1 (Convergence of vector input). Consider LeCun initialization with  $\tau_0 = d_m^{-1}$  scaling, given vector input  $\pmb{x} \in \mathbb{R}^{\tilde{d}}$ , if one feeds the input to Transformer by setting  $d_s = 1$ ,  $d = \tilde{d}$ , then training with GD can converge to a global minimum. On the contrary, if one sets  $d_s = \tilde{d}$ ,  $d = 1$ , the conditions in Eqs. (2) and (3) do not hold, the convergence can not be guaranteed by our theory.
Remark: Such a result is motivated by considering least squares. Specifically, given input  $\mathbf{X} \in \mathbb{R}^{N \times 1}$ , then the NTK  $\mathbf{X}\mathbf{X}^{\top}$  is a rank-one matrix. As a result, when the augmented matrix  $[\mathbf{X},\mathbf{y}]$  is not rank-one, then there is no solution to the linear system so that GD training can not converge to zero loss. The empirical validation can be found in Figure 3.
Technical difficulty. The technical difficulty of our analysis includes handling the softmax function and scaling schemes beyond NTK initialization. Previous convergence analysis, e.g., [Du et al., 2019b, Nguyen, 2021] cannot be applied to our setting because of the following issues. First, different from classical activation functions, e.g., ReLU, in softmax each element of the output depends on all input. To tackle the interplay between dimensions, we build the connection between the output of softmax and  $\boldsymbol{X}\boldsymbol{X}^{\top}$  to disentangle the nonlinear softmax function. By doing so, a lower bound on the minimum singular value of  $\boldsymbol{F}_{\mathrm{pre}}$  in Proposition. 1 can be well controlled by  $\boldsymbol{X}\boldsymbol{X}^{\top}$  and the output of softmax.
Regarding different initializations and scaling, previous NTK-based analysis is only valid under the  $d_m^{-1}$  setting (the softmax degenerates to an all-one vector) but is inapplicable to the realistic  $d_m^{-1/2}$  setting, as discussed in the introduction. To tackle this issue, we analyze the input/output of softmax under LeCun/He initialization and identify the optimization properties of the loss function for global convergence under the finite-width setting.
5 Experimental validations
Our experiments are organized as follows: In Section 5.1, we conduct experiments with the model Eq. (1.2) on synthetic data and study the training dynamics. Next, we show convergence results on ViT [Dosovitskiy et al., 2021] on the standard MNIST dataset in Section 5.2. Additional results and detail on the experimental setup are deferred to Appendix E.
5.1 Fitting synthetic data
In this section, we corroborate our theoretical findings on the synthetic data. We generate 100 data points where the input  $\mathbf{X} \in \mathbb{R}^{10 \times 100}$  is sampled from standard Gaussian distribution. The corresponding label  $y \in \mathbb{R}$  is generated from standard Gaussian distribution. Squared loss is selected as the criterion. We apply gradient descent on the shallow Transformer defined in Eq. (1.2) with LeCun initialization and  $\tau_0 = d_m^{-1/2}$  for 400 epochs with a fixed step size  $\gamma = 1$ . We test different widths of the network including  $d_m = \{10, 100, 1000, 4000\}$  and plot the training progress in
(a) Convergence curve.
(b) Weight movement.
(c) Kernel distance.
Figure 2: Visualization of the training process of Transformers with LeCun initialization and  $\tau_0 = d_m^{-1/2}$  scaling on synthetic data. (a) Linear convergence. (b) Rate of change of the weights during training. Observe that the weights change very slowly after the  $50^{\text{th}}$  epoch. (c) Evolution of the NTK during the training. The result mirrors the plot (b) and demonstrates how the kernel varies significantly at the beginning of the training and remains approximately constant later. As the width increases, the empirical NTK becomes more stable.
$$
\operatorname {D i s t a n c e} \left(\boldsymbol {K}, \widetilde {\boldsymbol {K}}\right) = 1 - \frac {\operatorname {T r} \left(\boldsymbol {K} \widetilde {\boldsymbol {K}} ^ {\top}\right)}{\sqrt {\operatorname {T r} \left(\boldsymbol {K} \boldsymbol {K} ^ {\top}\right)} \sqrt {\operatorname {T r} \left(\widetilde {\boldsymbol {K}} \widetilde {\boldsymbol {K}} ^ {\top}\right)}},
$$
which quantitatively compares two kernels by the relative rotation, as used in Fort et al. [2020]. Figure 2c shows that the kernel changes rapidly at the beginning of training while being approximately constant at later stages. The experiment with  $\tau_0 = d_m^{-1}$ , which is deferred to Appendix E.1, obtains similar results.
Additionally, the experiment with other initialization schemes is deferred to Appendix E.1, where we observe that NTK initialization indeed yields slower convergence than LeCun/He initialization, which is consistent with our theoretical finding.
Next, in order to verify corollary 1, we feed vector input  $\pmb{x}_n \in \mathbb{R}^{100}$  into Transformer by setting either  $d_s = 1, d = 100$  or  $d_s = 100, d = 1$  under LeCun initialization with  $\tau_0 = d_m^{-1}$ . Figure 3 shows that the training of Transformer with  $d_s = 1, d = 100$  is similar to that of two-layer FCNN with the same width  $d_m = 100$ . However, the case of  $d_s = 100, d = 1$  fails to converge, which is consistent with our theoretical finding.
Figure 2. The result shows that the training can converge except for the case with sub-linear width, see the linear convergence rate in Figure 2a and the small movement of the weight in Figure 2b. For these cases, the weights do not change much after 50 epochs while the losses are still decreasing. In Figure 2c, we keep track of the evolution of NTK along each epoch. Specifically, the kernel distance is given by:
Figure 3: Convergence result on synthetic data with vector input.
5.2 Fitting real-world dataset
Beyond synthetic data, in this section, we examine the convergence performance of Vision Transformer (ViT) on classification task on MNIST dataset [LeCun et al., 1998], which includes 10 classes of images with size  $28 \times 28$ . We use a single layer and single head ViT. The dimension of  $d$  is 64. We change the dimension of the query, key, and value from 16 to 1024 and 16384. The network is optimized with SGD with step size 0.1, and momentum 0.9 for 50 epochs. We repeat the experiment for three runs. We present the convergence results over 3 runs in Figure 4a, which shows that when the width is smaller, e.g., 16, both  $\tau_0 = d_m^{-1}$  and  $\tau_0 = d_m^{-1/2}$  scaling have similar
(b) Attention map,  $d_m = 16384$ .
Figure 4: Convergence curve on MNIST dataset with different scaling schemes and different widths in (a). Visualization of attention map in (b).
convergence results. However, as the width increases to 1024 and 16384, the  $\tau_0 = d_m^{-1}$  setting admits a slower rate than that of  $\tau_0 = d_m^{-1/2}$ , especially a extremely slow rate under  $d_m = 16384$ . This is consistent with our theoretical result on the dimension missing effect such that the  $\tau_0 = d_m^{-1}$  setting makes Transformer difficult to fit data. Additionally, we visualize the attention map in Figure 4b, i.e., the output of softmax in the self-attention layer before training and after training under  $\tau_0 = d_m^{-1/2}$  setting. We could see that the self-attention layer changes a lot during training.
We present a comprehensive analysis of the global convergence properties of shallow Transformer under various scaling and initialization schemes. Regarding scaling schemes, for a large width setting, the difference on convergence between  $\tau_0 = d_m^{-1/2}$  and  $\tau_0 = d_m^{-1}$  can be demonstrated both theoretically and empirically. Our theory is able to explain this in a dimension missing view. Regarding initialization schemes, our theory prefers to using LeCun and He initialization in Transformer training, which allows for a faster convergence rate than NTK initialization. Though our analysis is limited to shallow Transformers, we believe our framework can be extended to deep Transformers. We provide further discussion into this extension in Appendix C.7. Exploring the convergence properties of deep Transformers is indeed an intriguing avenue for future research.
We thank the reviewers for their constructive feedback. We thank Zhenyu Zhu for the discussion and help in this work. This work has received funding from the Swiss National Science Foundation (SNSF) under grant number 200021_205011. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). Corresponding author: Fanghui Liu.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548, 2020.
Yongyi Yang, Zengfeng Huang, and David Wipf. Transformers from an optimization perspective. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in neural information processing systems (NeurIPS), 2022.
Atif Emre Yüksel, Yasar Alim Türkmen, Arzucan Özgür, and Berna Altinel. Turkish tweet classification with transformer encoder. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1380-1387, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017.
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, and Volkan Cevher. Generalization properties of nas under activation and skip connection search. In Advances in neural information processing systems (NeurIPS), 2022.
Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. In Advances in neural information processing systems (NeurIPS), volume 32, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine learning, 2020.
- In Appendix A, we summarize the symbols and notation used in this work.
- In Appendix B.1, we provide a theoretical overview for neural tangent kernel (NTK). The background in Sub-Exponential random variables is elaborated in Appendix B.2. More detailed related work on the convergence analysis of neural networks can be found in Appendix B.3.
- The proofs for the convergence of Transformer are included in Appendix C.
- The derivations and the proofs for the NTK are further elaborated in Appendix D, including the formulation of NTK, minimum eigenvalue of NTK, and the relationship between the Hessian and the width.
Further details on the experiments are developed in Appendix E.
- Limitations and societal impact of this work are discussed in Appendix F and Appendix G, respectively.
A Symbols and Notation
We include the core symbols and notation in Table 2 for facilitating the understanding of our work.
Table 2: Core symbols and notations used in this paper.
Symbol	Dimension(s)	Definition
N(μ,σ)	-	Gaussian distribution of mean μ and variance σ
||w||2	-	Euclidean norms of vectors w
||W||2	-	Spectral norms of matrix W
||W|F	-	Frobenius norms of matrix W
||W||*	-	Nuclear norms of matrix W
λmin(W), λmax(W)	-	Minimum and maximum eigenvalues of matrix W
σmin(W), σmax(W)	-	Minimum and Maximum singular values of matrix W
w(i)	-	i-th element of vectors w
W(i,j)	-	(i,j)-th element of matrix W
Wt	-	W at time step t
o	-	Hadamard product
σs(W)	-	Row-wise Softmax activation for matrix W
σr(W)	-	Element-wise ReLU activation for matrix W
1{event}	-	Indicator function for event
N	-	Size of the dataset
dm	-	Width of intermediate layer
ds	-	Sequence length of the input
d	-	Dimension of each token
ηQ, ηK, ηV, ηO	-	Variance of Gaussian initialization of WQ, WK, WV, wO
τ0, τ1	-	Scaling factor
γ	-	Step size
Xn	Rds×d	The n-th data point
yn	R	The n-th target
DX	-	Input data distribution
DY	-	Target data distribution
βi,n := σs(τ0Xn(i):WQWKXnT)T	Rds	The i-th row of the output of softmax of the n-th data point
y := [y1,..., yN]T	RN	Ground truth label of the data samples {Xn}N
f(θ) := [f(X1;θ), ..., f(XN;θ)]T	RN	Network output given data samples {Xn}N
Fpre(θ) := [a3(X1;θ), ..., a3(XN;θ)]T	RN×dm	Output of the last hidden layer given {Xn}N
f(θ) := f(X;θ)	R	Network output given data samples X
fpre := a3(X;θ)	Rdm	Output of the last hidden layer given X
WQ, WK	Rdm×d, Rdm×d	Learnable parameters
WV, wO	Rdm×d, Rdm	Learnable parameters
λQ ≈ ||WQ||2 + CQ, λK ≈ ||WK||2 + CK	R	Parameters norm
λV ≈ ||WV||2 + CV, λO ≈ ||wO||2 + CO	R	Parameters norm
ρ ≈ N1/2ds3/2τ1Cx, z ≈ λO2(1 + 4τ02Cx4ds2λV2(λQ+ λK2))	R	Auxiliary terms
fi	R	Output of network for input Xi
O, o, Ω and Θ	-	Standard Bachmann-Landau order notation
* The superscript with bracket represents the element of a vector/matrix, e.g.,  $\mathbf{w}^{(i)}$  is the  $i^{\text{th}}$  element of  $\mathbf{w}$ .
* The superscript without bracket symbolizes the parameters at different training steps, e.g.,  $\pmb{\theta}^t$ .
* The subscript without bracket symbolizes the variable associated to the  $n$ -th data sample, e.g.,  $X_{n}$ .
B Theoretical background
B.1 Preliminary on NTK
In this section, we summarize how training a neural network by minimizing squared loss, i.e.,  $\ell (\pmb{\theta}^t) = \frac{1}{2}\sum_{n = 1}^{N}(f(\pmb {X}_n;\pmb {\theta}^t) - y_n)^2$ , via gradient descent can be characterized by the kernel regression predictor with NTK.
By choosing an infinitesimally small learning rate, we can obtain the following gradient flow:
$$
\frac {d \boldsymbol {\theta} ^ {t}}{d t} = - \nabla \ell (\boldsymbol {\theta} ^ {t}).
$$
By substituting the loss into the above equation and using the chain rule, we can find that the network outputs  $f(\pmb{\theta}^t) \in \mathbb{R}^N$  admit the following dynamics: