DNN-based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
Year: 2021

Abstract:
We study the Solid Isotropic Material Penalisation (SIMP) method with a density field generated by a fully-connected neural network, taking the coordinates as inputs. In the large width limit, we show that the use of DNNs leads to a filtering effect similar to traditional filtering techniques for SIMP, with a filter described by the Neural Tangent Kernel (NTK). This filter is however not invariant under translation, leading to visual artifacts and non-optimal shapes. We propose two embeddings of the input coordinates, which lead to (approximate) spatial invariance of the NTK and of the filter. We empirically confirm our theoretical observations and study how the filter size is affected by the architecture of the network. Our solution can easily be applied to any other coordinates-based generation method.

heading: 1 Introduction
text: Topology optimisation [4], also known as structural optimisation, is a method to find optimal shapes subject to some constraints. It has been widely studied in the field of computational mechanics. Here we are interested in the particular case of the Solid Isotropic Material Penalisation (SIMP) method [18, 1], which is a very common method in this field.
Recently some authors have used Deep Neural Networks (DNNs) to perform topology optimisation. We can differentiate two different approaches in the use of DNNs with SIMP. The first approach consists in generating with the classical algorithms a dataset of optimised shapes and train a DNN on this dataset to produce new optimal shapes [3, 31]. Variations of this approach use Generative Adversarial Networks (GAN) [21, 28] to effectively reproduce classical topology optimisation.
In the second approach, the density is generated pointwise by a DNN, which is trained with gradient descent to optimise the density field with respect to the physical constraints, as proposed in [11] to use the power of deep models without giving up exact physics. We focus on the approach of [7, 6] where the density field is generated by a Fully-Connected Neural Network (FCNN) taking the coordinates of a grid as inputs. Surprisingly, [7] observes that the DNN-generated density fields do not feature checkerboard artifacts, which are common in vanilla SIMP. A traditional method to avoid checkerboard patterns is to add a filter [29, 5], but it is not needed for DNN-generated density fields.
In this paper, we analyse theoretically how the use of a DNN to generate the density field affects the learning. Our main theoretical tool is the Neural Tangent kernel (NTK) introduced in [13] to describe the dynamics of wide neural networks [13, 2, 17, 12].
While this paper focuses on linear elasticity and SIMP, our analysis can extended to other physical problems such as heat transfer [19], or any model where an image is generated by a DNN taking the pixel coordinates as inputs (like in [20]).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 1.1 Our contribution
text: In this paper we study topology optimisation with neural networks. The physical density is represented by a neural network taking an embedding of spatial coordinates as inputs, i.e. the density at a point x ∈ Rd is given by fθ(ϕ(x)) for θ the parameters of the network and ϕ an embedding. We use theoretical tools, in particular the Neural Tangent Kernel (NTK), to understand how the architecture and hyperparameters of the network affect the optimisation of the density field:
• We show that in the infinite width limit (when the number of neurons in the hidden layers grows to infinity), topology optimisation with a DNN is equivalent to topology optimisation with a density filter equal to the “square root” of the NTK. Filtering is a commonly used technique in topology optimisation, aimed to remove checkerboard patterns.
• In topology optimisation as in other physical optimisation problems, it is crucial to guarantee some spatial invariance properties. If the coordinates are taken as inputs of the network directly, the NTK (and the corresponding filter) is not translation invariant, leading to nonoptimal shapes and visual artifacts. We present two methods to ensure the spatial invariance of the NTK: embedding the coordinates on the (hyper-)torus or using a random Fourier features embedding (similar to [32]).
• In traditional topology optimisation, the filter size must be tuned carefully. When optimising with a DNN, the filter size depends on the embedding of the coordinates and the architecture of the network. We define a filter radius for the NTK, which plays a similar role as the classical filter size and discuss how it is affected by the choice of embedding, activation function, depth and other hyperparameters like the importance of bias in the network. This tradeoff can also be analysed in terms of the spectrum of the NTK, explaining why neural networks naturally avoid checkerboard patterns.
We confirm and illustrate these theoretical observations with numerical experiments. Our implementation of the algorithm will be made public at https://github.com/benjiDupuis/DeepTopo.

heading: 2 Presentation of the method
text: In this paper, we use a DNN to generate the density field used by the Solid Isotropic Material Penalisation (SIMP) method. Our implementation of SIMP is based on [1] and [18]. In this section we introduce the traditional SIMP method and our neural network setting.

heading: 2.1 SIMP method
text: We consider a regular grid of N elements where the density of element i is denoted yi ∈ [0, 1], informally the value yi represents the presence of material at a point i. Our goal is to optimise over the density y ∈ RN to obtain a shape that can withstand forces applied at certain points, represented by a vector F .
The method uses finite element analysis to define a stiffness matrix K(y) ∈ S++N (R) from the density y and computes the displacement vector U(y) (which represent the deformation of the shape at all points i as a result of the applied forces F ) by solving a linear system K(y)U(y) = F . In our implementation, we performed it either by using sparse Cholesky factorisation [9, 8] or BICGSTAB method [33] (this last one can be used for a high number of pixels).
The loss function is then defined as the compliance C(y) = U(y)TK(y)U(y), under a volume constraint of the form ∑N i=1 yi = V0, with 0 ≤ V0 ≤ N (see [1, 18]).

heading: 2.2 A modified SIMP approach
text: Several methods exist to optimise the density field y ∈ RN , such as gradient descent or the so-called Optimality Criteria (OC) [34]. We propose here an optimisation method inspired from [11] which we will refer as the Modified Filtering method (MF). The advantage of this method is that it can be used with or without DNNs, hence allowing comparison between these two approaches. We first present here the model without DNNs.
In our method, the densities yMFi are given by:
∀i ∈ {1, ..., N}, yMFi = σ(xi + b̄(X)), with b̄(X) such that N∑ i=1 yMFi = V0, (1)
for X = (x1, ..., xN ) ∈ RN and the sigmoid σ(x) = 11+e−x . We will denote this operation as: Y MF = Σ(X). The sigmoid ensures that densities are in [0, 1] and the choice of the optimal bias b̄(X) ensures that the volume constraint is satisfied.
Filtering: If the vector X is optimised directly with gradient descent, SIMP often converges toward checkerboard patterns, i.e. some high frequency noise in the image, which is a common issue with SIMP [1]. To overcome this issue a common technique is to use filtering [29]. In this paper, we consider low-pass density filters of the form: X = TX̄ where T represents a convolution on the grid, X̄ are the design variables and X is the vector in equation 1. The loss function of this method is then naturally defined as: X̄ 7−→ C(Σ(TX̄)). The gradient∇Y C is easily obtained by the self-adjointness of the variational problem [34, 15]. We recover∇XC from∇Y C using an implicit differentiation technique [10]. The following proposition is a consequence of implicit function theorem and chain rules:
Proposition 2.1. Let Ṡ be the vector with entries σ̇(xi + b̄(X)). We have ∇XC = DX∇Y C with:
DX := − 1
|Ṡ|1 ṠṠT + Diag(Ṡ). (2)
where |.|1 denotes the l1 norm of a vector. Furthermore DX is a symmetric positive semi-definite matrix whose null-space is the space of constant vectors and has eigenvalues smaller than 14 .

heading: 2.3 Proposed algorithm: SIMP with Neural networks
text: Fully-Connected Neural Networks (FCNN) are characterised by the number of layers L + 1, the numbers of neurons in each layer (n0, n1, ..., nL) and an activation function µ : R −→ R, here we will use the particular case nL = 1. The activations al ∈ Rnl and preactivations ãl ∈ Rnl are defined recursively for all layers l, using the so-called NTK parameterisation [13]:
a0(x) = x, ãl+1(x) = α √ nl W lal(x) + βbl, al+1(x) = µ
( ãl+1(x) ) , (3)
for some hyperparameters α, β ∈ [0, 1] representing the contribution of the weights and bias terms respectively. The parameters θ = (θp)p, consisting in weight matrices W l and bias vectors bl are drawn as i.i.d. standard normal random variables N (0, 1). We denote the output of the network as fθ(x) = ã L(x).
Remark: To ensure that the variance of the neurons at initialization is the equal to 1 at all layers, we choose α and β such that α2+β2 = 1 and use a standardised non-linearity, i.e. EX∼N (0,1)[µ(X)2] = 1 ([14]).
In our approach, the pre-densities XNN(θ) = (xNN1 , ..., x NN N ) are generated by a neural network as xNNi = fθ(zi) where zi ∈ Rn0 is either the coordinates of the grid elements (in this case n0 = d) or
an embedding of those coordinates. We then apply the same transformation Σ to obtain the density field Y NN(θ) = Σ(XNN(θ)). Our loss function is then defined as:
θ 7−→ C(Y NN(θ)) = C ( Σ(X(θ)) ) .
The design variables are now the parameters θ of the network. The gradient∇θC w.r.t. to the parameters is computed by first using Proposition 2.1 to get∇Y NNC followed by traditional backpropagation. Remark: Note the absence of filter T in the above equations, indeed we will show how neural networks naturally avoid checkerboard patterns, making the use of filtering obsolete.
Initial density field: The SIMP method is usually initialised with a constant density field [1]. Since the neural network is initialized randomly, the initial density field is random and non-constant. To avoid this problem, we subtract the initial density field and add a well-chosen constant:
∀i ∈ {1, ..., N}, xi(θ) = f̄θ(t)(zi) = fθ(t)(zi)− fθ(t=0)(zi) + log (
V0 N − V0
) . (4)
We used equation 4 to compute X(θ) in our numerical experiments.

heading: 3 Theoretical Analysis

heading: 3.1 Analogy between the Neural Tangent Kernel and filtering techniques
text: In our paper, we use the Neural Tangent Kernel (NTK [13]) as the main tool to analyse the training behaviour of the FCNN. In our setting (where nL = 1) the NTK is defined as:
∀z, z′ ∈ Rn0 , ΘLθ (z, z′) = ∑ p ∂fθ ∂θp (z) ∂fθ ∂θp (z′) = (∇θfθ(z)|∇θfθ(z′)).
This is a positive semi-definite kernel. Given some inputs z1, ..., zN we define the NTK Gram matrix as: Θ̃Lθ := ( ΘL(zi, zj) ) 1≤i,j≤N ∈ R N×N .
Assuming a small enough learning rate, the evolution of the network under gradient descent is well approximated by the gradient flow dynamics ∂tθ(t) = −∇θC(t). The evolution of the output of the network XNN(θ) can then easily be expressed in terms of the NTK Gram matrix [14] for a loss L:
∂tX NN(θ(t)) = −Θ̃Lθ(t)∇XNNL.
From this equation we can derive the evolution of the physical density field Y NN in our algorithm: Proposition 3.1. If the network is trained under this gradient flow, then by applying chain rules, we can prove that the density field follows the equation:
∂tY NN(θ(t)) = −DX(t)Θ̃Lθ(t)DX(t)∇Y C(Y NN(θ(t))). (5)
The analogy between the NTK and filtering techniques comes from the following observation. With Modified Filtering with a filter T , we show similarly that the density field Y MF evolves as
∂tY MF(t) = −DX(t)TTTDX(t)∇Y C(Y MF(t)). (6)
We see that the NTK Gram matrix and the squared filter TTT play exactly the same role. An important difference however is that the NTK is random at initialisation and evolves during training.
This difference disappears for large widths (when n1, . . . , nL−1 are large), since the NTK converges to a deterministic and time independent limit Θ̃L∞ as n1, . . . , nL−1 → ∞ [13]. Furthermore, in contrast to the finite width NTK (also called empirical NTK), we have access to a closed form formula for the limiting NTK Θ̃L∞ (given in the appendix).
In the infinite width limit, the evolution of the physical densities is then expressed in terms of the limiting NTK Gram matrix Θ̃L∞:
∂tY NN(θ(t)) = −DX(t)Θ̃L∞DX(t)∇Y C(Y NN(θ(t))). (7)
From now on we will focus on this infinite-width limit, comparing the NTK Gram matrix Θ̃L∞ and the squared filter TTT . Recent results [17, 2, 12] suggest that this limit is a good approximation when the width of the network is sufficiently large. For more details see the appendix, where we compare the empirical NTK with its limiting one and plot its evolution in our setting.

heading: 3.2 Spatial invariance
text: In physical problems such as topology optimisation, it is important to ensure that certain physical properties are respected by the model. We focus in this section on the translation and rotation invariance of topology optimisation: if the force constraints are rotated or translated, the resulting shape should remain the same (up to rotation and translation), as in Figure 4 (b.1 and b.2).
In Modified Filtering method, this property is guaranteed if the filter T is translation and rotation invariant. In contrast the limiting NTK is in general invariant under rotation [13] but not translation. As Figure 4 shows, this leads to some problematic artifacts. The NTK can be made translation and rotation invariant by first applying an embedding ϕ : Rd −→ Rn0 with the properties that for any two coordinates p, p′, ϕ(p)Tϕ(p′) only depends on the distance ‖p− p′‖2. Since the rotation invariance of the NTK implies that ΘL∞(z, z
′) depends only on the scalar products zT z′, zzT and z′z′T , we have that ΘL∞(ϕ(p), ϕ(p
′)) depends only on ‖p− p′‖ as needed. The issue is that for finite n0 there is no non-trivial embedding ϕ with this property: Proposition 3.2. Let ϕ : Rd → Rn0 for d > 2 and any finite n0. If ϕ satisfies ϕ(x)Tϕ(x′) = K(‖x− x′‖) for some continuous function K then both ϕ and K are constant.
To overcome this issue, we present two approaches to approximate spatial invariance with finite embeddings: an embedding on a (hyper)-torus and a random feature [23] embedding based on Bochner theorem [26].

heading: 3.2.1 Embedding on a hypertorus
text: In this subsection we consider the following embedding of a nx × ny regular grid on a torus:
R2 3 p = (p1, p2) 7−→ ϕ(p) = r(cos(δp1), sin(δp1), cos(δp2), sin(δp2)), (8)
where δ > 0 is a discretisation angle (our default choice is δ = π2max(nx,ny) ). One can use similar formulas for d > 2 (leading to an hyper-torus embedding), we used d = 2 in equation 8 for simplicity.
This embedding leads to an exact translation invariance and an approximate rotation invariance: ϕ(p)Tϕ(p′) = r2(cos(δ(p1−p′1)) + cos(δ(p2−p′2))) = r2 ( 2− δ 2
2 ‖p−p′‖22
) +O ( δ4‖p−p′‖44 ) .
As a result, the limiting NTK Θ∞(ϕ(p), ϕ(p′)) is translation invariant and approximately rotation invariant (for small δ and/or when p, p′ are close to each other). Moreover, if we look at the limiting NTK on the whole torus, we obtain that the gram matrix Θ̃∞ is a discrete convolution on the input grid, with nice properties summed up in the following proposition:
Proposition 3.3. We can always extend our nx × ny grid and choose δ such that the embedded grid covers the whole torus (typically δ = π2max(nx,ny) and take a n× n grid with n = 4 max(nx, ny)).
Then the Gram matrix Θ̃∞ of the limiting NTK is a 2D discrete convolution matrix. Moreover the NTK Gram matrix has a positive definite square root √ Θ̃∞ which is also a discrete convolution matrix.
As we know, the eigenvectors of such a convolution matrix are the 2D Fourier vectors. The corresponding eigenvalues are the discrete Fourier transforms of the convolution kernel.
The square root of the NTK Gram matrix √
Θ̃∞ then corresponds to the filtering matrix T in our analogy. Figure 3 shows that on the full torus, the matrix square root √ Θ̃θ indeed looks like a typical smoothing filter.
As Figure 4 shows, the torus embedding method gives good numerical results and respect the symmetry of the applied forces F .

heading: 3.2.2 Random embeddings for radial kernels
text: Another approach to approximate a rotation and translation invariant embedding is to use random Fourier features [23], which is a general method to approximate shift invariant kernels of the form k(x, y) = k(x− y). By Bochner theorem [26], any continuous non-zero radial kernel k(x− y) = K(‖x− y‖) can be written as the the (scaled) Fourier transform of a probability measure Q on Rd:
k(r) = k(0) ∫ Rd eiω.rdQ(ω).
For radial kernels, we formulate random Fourier features embeddings ϕ : Rd → Rn0 as follows:
ϕ(p)i = √ 2k(0) sin(wTi p+ π
4 + bi),
for i.i.d. samples w1, ..., wn0 ∈ Rd from Q (which is also invariant by rotation) and i.i.d. samples b1, ..., bn0 ∈ Rd from any symmetric probability distribution (or uniform laws on [0, 2π]). By the law of large numbers for large n0, we have the approximation 1n0ϕ(p) Tϕ(p′) ' k(p− p′).
Gaussian embedding: Depending on the kernel k that we want to approximate, it may be difficult to sample from the distribution Q. The simplest case is for a Gaussian kernel k(d) = e− 1 2`2
d2 , where the distribution Q of the weights wi is N (0, 1`2 Id), i.e. the entries wij are all i.i.d. N (0, 1 `2 ) Gaussians. For this reason this is the embedding that we will use in our numerical experiments. Note the similarity between this type of embedding and an untrained first layer of a FCNN with sine activation function, weights wi and bias bi.
Moreover, the following result shows that we can still define a ”square root” of the NTK with those types of embedding and thus complete the analogy with equation 6.
Proposition 3.4. Let ϕ be an embedding as described above for a positive radial kernel k ∈ L1(Rd) with k(0) = 1, k ≥ 0. Then there is a filter function g : R→ R and a constant C such that for all p, p′:
lim n0→∞
Θ∞(ϕ(p), ϕ(p ′)) = C + (g ? g)(p− p′), (9)
where Θ∞ is the limiting NTK of a network with a Lipschitz, non-constant and standardised activation function µ. (Here ? denotes the convolution product).
As the matrix DX in equation 7 cancels out the constant frequency (proposition 2.1), the constant C doesn’t matter, i.e. DXΘ̃ (L) ∞ DX = DX ( Θ̃ (L) ∞ − C ) DX .

heading: 4 Experimental analysis

heading: 4.1 Setup
text: Most of our experiments were conducted with a torus embedding or a gaussian embedding. For the SIMP algorithm, we adapted the code described in [1, 18]. Here are the hyperparameters used in the experiments.
For the Gaussian embedding, we used n0 = 1000 and a length scale ` = 4. This embedding was followed by one hidden linear layer of size 1000 with standardized ReLu (x 7→ √ 2 max(0, x)) and a bias parameter β = 0.5. For the torus embedding we set the torus radius to r = √
2 (to be on a standard sphere) and the discretisation angle to δ = π2max(nx,ny) (to cover roughly half the torus, which is a good trade-off between rotation invariance and kernel size), where nx×ny is the size of the grid. It was followed by 2 linear layers of size 1000 with β = 0.1. The ReLu activation is not well-suited in this case because it induces filters that are too wide. The large radius of the NTK kernel can be understood in relation with the order/chaos regimes [27, 22], as observed in [14] the ReLU lies in the ordered regime when β > 0, leading to a “wide” kernel, a narrower kernel can be achieved with non-linearities which lie in the chaotic regime instead. We used a cosine activation of the form x 7→ cos(ωx), which has the advantage that the width of the filter can be adjusted using the ω hyperparameter, see Section 7. When not stated otherwise we used ω = 5.
Even though our theoretical analysis is for gradient flow, we obtain similar results with other optimizers such as RPROP [24] (learning rate 10−3) and ADAM [16] (learning rate 10−3). RPROP gave the fastest results, possibly because it is well-suited for batch learning [25]. Vanilla gradient descent can be very slow due to the vanishing of the gradients when the image becomes almost binary (due to the sigmoid), we therefore gradually increased the learning rate during training to compensate.

heading: 4.2 Spectral analysis
text: In SIMP convolution with a low pass filter ensures that low frequencies are optimised faster than high frequencies, to avoid checkerboards.
With the embeddings proposed in the last two subsections, the limiting NTK takes the form of a convolution over the input space Rd. Figure 5 represents the eigenvalues and eigenimages of the
NTK Gram matrix Θ̃θ(t). Even though this plot is done for a finite width network and a finite random embedding, we see that the eigenimages look like 2D Fourier modes. The fact that the low frequencies have the largest eigenvalues supports the similarity between the NTK and a low pass filter.
This may explain why neural networks naturally avoid checkerboard patterns: the low frequencies of the shape are trained faster than the high frequencies which lead to checkerboard patterns.

heading: 4.3 Filter radius
text: formula for the square root of the NTK is more difficult. For Gaussian filters the radius of the squared filter is √ 2 times that of the original, suggesting that the filter radius is well estimated by 1√
2 R̂1/2.
The quantity R̂1/2 is a function of the hyperparameters of the network (α,β, L, see appendix) and of the embedding (the lengthscale `). Using the formula for R̂1/2, these hyperparameters can be tuned to obtain a specific filter radius.
With the Gaussian embedding, the radius of the filter can easily be adjusted by changing the lengthscale ` of the embedding. As illustrated in Figure 7.
With the torus embedding, we instead have to change the hyperparameters of the network to adjust the radius of the filter. With the ReLU activation function, the radius is very large which makes it impossible to obtain precise shape. The solution we found is to use a cosine activation x 7→ cos(ωx) with hyper-parameter ω. Figure 6 shows how the radius decreases as ω increases. The β parameter has the opposite effect, as increasing it increases the radius. For different values of ω and β, we obtain a variety of radius and plot the resulting shapes. This plot also illustrates the role of the radius in the determination of the resulting shape. The fact that cosine activation leads to an adjustable NTK radius could explain why periodic activation function help in the representation of high frequency signal as observed in [30].
The effect of depth is more complex. For large depths L the NTK either approaches a constant kernel in the so-called order regime (with infinite radius) or a Kronecker delta kernel in the so-called chaos regime (with zero radius) [22, 27, 14]. Depending on whether we are in the order or chaos regime (which is determined by the activation function µ and the parameters α, β), increasing the depth can either increase or decrease the radius.
We conducted an experimental study of the influence of this parameter on the geometry of the final shape. We observed that its complexity (number of holes, high frequencies) is highly controlled by R̂1/2. We see in Figure 7 and 6 some examples of shape obtained for several values of R̂1/2.

heading: 4.4 Up-sampling
text: Since the density field is generated by a DNN, it can be evaluated at any point in Rd, hence allowing upsampling. As Figure 8 shows, with our method we obtain a smooth and binary shape. Something interesting happens when the network is trained without an embedding: when upsampling we observe some visual artifacts plotted in Figure 9. We believe that it is due to the lack of spatial invariance.
Note that this second experiment was done with batch norm, as described in [7], since for this problem it was difficult to obtain a good shape with a vanilla ReLU-FCNN. With our embeddings, we can achieve complex shapes without batch-norm.

heading: 5 Conclusion
text: Using the NTK, we were able to give a simple theoretical description of topology optimisation with DNNs, showing a similarity to traditional filtering techniques. This theory allowed us to identify a problem: since the NTK is not translation invariant, the spatial invariance of topology optimisation is not respected, leading to visual artifacts and non-optimal shapes. We propose a simple solution to this problem: adding a spatial invariant embedding to the coordinates before the DNN.
Using this method, our models are able to learn efficient shapes while avoiding checkerboard patterns. We give tools to adjust the implicit filter size induced by the hyperparameters, to give control over the complexity of the final shape. Using the learned network, we can easily perform good quality
up-sampling. The techniques described in this paper can easily be translated to any other problem where spatial invariance is needed.
The NTK is a simple yet powerful tool to analyse a practical method such as SIMP when combined with a DNN. Morover it can be used to make informed choices of the DNN’s architecture and hyperparameters.
Acknowledgments and Disclosure of Funding
There is no funding or competing interests associated to this work.

Review summaries:
Built upon the NTK, this paper provides a theoretical understanding of using deep neural networks in the SIMP method for topology optimization in the infinite-width regime. The authors show that: 1) Topology optimization with a DNN is equivalent to topology optimization with the square root of NTK as the density filter. 2) They also propose two embeddings to ensure the spatial invariance of the NTK. 3) They study how the choice of embedding, activation function, depth, etc., affects the radius of the NTK. 

The paper studies topology optimization with neural networks. In particular, they use Deep neural networks to generate the density at each point used by Solid Isotropic Material Penalisation (SIMP) methods. The deep neural network has the input as the coordinates and the output is density (the amount of material) at that coordinate.    

The paper introduces a deep fully-connected neural network (FCNN) approach to topology optimization, in which a neural network learns to predict density as a function of spatial location, with gradient feedback based on Solid Isotropic Material Penalization (SIMP). The authors draw a connection between the NTK Gram matrix of this FCNN and the spatial low-pass filtering usually done in non-neural topology optimization, and propose two input embeddings that induce shift-invariance in the NTK. Instead of tuning the filtering radius of an explicit low-pass filter, with the proposed FCNN approach the effective filtering can be tuned by adjusting parameters of the embedding and the network architecture to vary the width of the NTK and control its spectrum.