DNN-based Topology Optimisation:  Spatial Invariance and Neural Tangent Kernel
Authors: Benjamin Dupuis
Arthur Jacot
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Tangent Kernel
Topology optimization
DNN
Neural Networks
positional encoding

Abstract: We study the Solid Isotropic Material Penalization (SIMP) method with a density field generated by a fully-connected neural network, taking the coordinates as inputs. In the large width limit, we show that the use of DNNs leads to a filtering effect similar to traditional filtering techniques for SIMP, with a filter described by the Neural Tangent Kernel (NTK). This filter is however not invariant under translation, leading to visual artifacts and non-optimal shapes. We propose two embeddings of the input coordinates, which lead  to (approximate) spatial invariance of the NTK and of the filter. We empirically confirm our theoretical observations and study how the filter size is affected by the architecture of the network. Our solution can easily be applied to any other coordinates-based generation method. 

DNN-based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
Benjamin Dupuis
Arthur Jacot
Chair of Statistical Field Theory
Chair of Statistical Field Theory
Ecole Polytechnique Fédérale de Lausanne
Ecole Polytechnique Fédérale de Lausanne Lausanne, Switzerland arthur. jacbbure.actt
Lausanne, Switzerland www.isdeppl..ch berjjamiud.uuuuss@eppp...cc
jenjamin.
Abstract
We study the Solid Isotropic Material Penalisation (SIMP) method with a density field generated by a fully-connected neural network, taking the coordinates as inputs. In the large width limit, we show that the use of DNNs leads to a filtering effect similar traditional filtering techniques for SIMP, with filter described by the Neural Tangent Kernel (NTK). This filter is however not invariant under translation, leading to visual artifacts and non-optimal shapes. We propose two embeddings of the input coordinates, which lead to (approximate) spatial invariance the NTK and the filter. We empirically confirm our theoretical observations and study how the filter size is affected by the architecture of the network. Our solution can easily be applied to any other merdiintes-based generation method.
Introduction
Topology optimisation 4 also known as structural optimisation, is method find optimal shapes subject some constraints. It has been widely studied in the field of computational mechanics. Here we are interested in the particular case of the Solid Isotropic Material Penalisation (SIMP) method 18 I which is a very common method in this field.
Recently some authors have used Deep Neural Networks (DNNss to perform topology optimisation. We can differentiate two different approaches in the use of DNNs with SIMP. The first approach consists in generating with the classical algorithms dataset of optimised shapes and train DNN on this dataset produce new optimal shapes 3 31. Variations of this approach use Generative Adversarial Networks (GAN) 28 to effectively reproduce classical topology optimisation.
In the second approach, the density is generated pointwise by DNN, which is trained with gradient descent optimise the density field with respect the physical constraints, as proposed in [I]] to use the power of deep models without giving up exact physics. We focus on the approach of where the density field is generated by a Fully-Connected Neural Network (FCNN) taking the
coordinates of grid as inputs. Surprisingly. 7 observes that the DNN-generated density fields do
feature checkerboard artifacts, which are common in vanilla SIMP. A traditional method avoid checkerboard patterns is to add filter 29 [] but it is not needed for DNN-generated density fields.
In this paper, we analyse theoretically how the use of a DNN generate the density field affects the
learning. Our main theoretical tool is the Neural Tangent kemel (NTK) introduced in B to describe the dynamics of wide neural networks
12].
While this paper focuses on linear elasticity and SIMP, our analysis can extended to other physical problems such as heat transfer [19) any model where image is generated by DNN taking the pixel coordinates as inputs (like 20
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1 Our contribution
In this paper we study topology optimisation with neural networks. The physical density is represented by neural network taking embedding of spatial coordinates inputs, i.. the density point I E Rd is given by fo(()()) for 9 the parameters of the network and yp an embedding. We use theoretical tools, in particular the Neural Tangent Kernel (NTK), to understand how the architecture and hyperparameters of the network affect the optimisation of the density field:
We show that in the infinite width limit (when the number of neurons in the hidden layers grows to infinity), topology optimisation with DNN is equivalent topology optimisation with a density filter equal to the "square root" the NTK. Filtering is a commonly used technique in topology optimisation, aimed to remove checkerboard patterns.
In topology optimisation as in other physical optimisation problems, is crucial to guarantee some spatial invariance properties. If the coordinates are taken as inputs of the network directly, the NTK (and the corresponding filter) is not translation invariant, leading to non- optimal shapes and visual artifacts. We present two methods to ensure the spatial invariance of the NTK: embedding the coordinates on the (hyper-) or using a random Fourier features embedding (similar to (32).
In traditional topology optimisation, the filter size must be tuned carefully. When optimising with a DNN, the filter size depends on the embedding of the coordinates and the architecture of the network. We define filter radius for the NTK, which plays similar role as the classical filter size and discuss how it is affected by the choice of embedding, activation function, depth and other hyperparameters like the importance of bias the network. This tradeoff can also be analysed in terms of the spectrum of the NTK, explaining why neural networks naturally avoid checkerboard patterns.
We confirm and illustrate these theoretical observations with numerical experiments. Our implemen- tation of the algorithm will be made public at //ttps://giiiie thub. com/ben comm/bejjiuupli wwwpuis/Deepepppeeeeoeo
2 Presentation of the method
In this paper, use DNN to generate the density field used by the Solid Isotropic Material Penalisation (SIMP) method. Our implementation of SIMP is based on [I] and [8]] In this section introduce the traditional SIMP method and our neural network setting.
2.1
SIMP method
We consider a regular grid of N elements where the density of element i denoted y, E [0, 1]. informally the value y represents the presence of material at point i. Our goal is to optimise over the density y E I obtain a shape that can withstand forces applied at certain points, represented by vector F.
The method uses finite element analysis define stiffness matrix K(y) E SN (R) from the density y and computes the displacement vector Uly) (which represent the deformation of the shape at all points i as a result of the applied forces F)) by solving linear system K K(y)U(y) F. In our implementation, we performed it either by using sparse Cholesky factorisation method 33 (this last one be used for high number of pixels).
or BICGSTAB
The loss function is then defined as the compliance C(y) = K(yU under a volume constraint of the form EN y: Vo. with 0 < Vo S N (see 188..
2.2 A modified SIMP approach
Several methods exist to optimise the density field y E NRN, such as gradient descent or the so-called
Optimality Criteria (OC) 344. We propose here an optimisation method inspired from [[]] which we will refer as the Modified Filtering method (MF). The advantage of this method is that it can be used with or without DNNs, hence allowing comparison between these two approaches. We first present
here the model without DNNs.

---

In our method, the densities thth are given by:
Vi E (I,..,,) Vi = oa; + b(X)), with b(X) such that
MF y;
Vo:
(1)
for X = (f1, RR and the sigmoid (ar) Ite-r- We will denote this operation as: YMF [(X). The sigmoid ensures that densities are in [0, 1] and the choice of the optimal bias b(X) ensures that the volume constraint is satisfied.
Filtering: If the vector X is optimised directly with gradient descent, SIMP often converges toward checkerboard patterns, i.e. some high frequency noise in the image, which is a common issue with SIMP 0. To overcome this issue common technique is use filtering (29) In this paper, consider low-pass density filters of the form: X = 7'X where I represents a convolution on the grid, X are the design variables and X is the vector in equation
The loss function of this method is then
naturally defined as: X + C()(TX)).
The gradient Vy C' is easily obtained by the self-ad eelf-adjointness of the variational problem 34 15 We recover V C' from V using an implicit differentiation technique [10]. The following proposition is consequence of implicit function theorem and chain rules:
Proposition 2.1. Let be the vector with entries ox, + b(X)). We have
with:
Dx
SST + Diag($).
15
where 1-l2 denotes the 1 norm of vector. Furthermore Dx is symmetric positive semi-definite matrix whose mull-space is the space of constant vectors and has eigenvalues smaller than 4.
2.3 Proposed algorithm: SIMP with Neural networks
Conneeeddd Neural Networks (FCNN) are characterised by the number of layers L + 1, the numbers of neurons in each layer (No, Tl. and an activation functtion R - R, here we will use the particular case n = 1. The activations e Th and preactivations â E Ih are defined recursively for all layers using the so-called NTK parameterisation (13):
a' x, a ax)
a'dd(x) 3t, (=p(=)((aaa m
for some hyperparameters a, 3 E [0. 1] representing the contribution of the weights and bias terms respectively. The parameters 0 p. consisting weight matrices Wi and bias vectors // are drawn as i..dd. standard normal random variables (0, 1). We denote the output of the network as fo(x) = (x))
Remark: To ensure that the variance of the neurons at initialization the equal to I at all layers, we choose a and B such that a² 3 1 and use standardised non-linenn linearittty, i.e. Ex~Nio,n
1 ([14))
grid G of N points
Embedding
Physical solver
Density field
SIMP
Pointwise FCNN
(xi)....N
(yi)....
KU-F F
Backpropagation
V.C
VyC
C=ETEE
Figure 1: Illustration of our method
In approach, the pre-densities X (N) (N ..... Ey NN are generated by neural network as
NN fo(z)) where 2, E 0go is either the coordinates of the grid elements (in this case no d)

---

an embedding of those coordinates. We then apply the same transformation Σ to obtain the density field YNN = =I(X E(XXNN (0)). Our loss function then defined as:
0 C(Y (6)) C()(X
The design variables are now the parameters of the network. The gradient V&C w.r.t. to the parame- ters is computed by first using Proposition 2.1 to get y. C followed by traditional backpropagation.
Remark: Note the absence of filter T in the above equations, indeed will show how neural networks naturally avoid checkerboard patterns, making the use of filtering obsolete.
Initial density field: The SIMP method is usually initialised with constant density field [1. Since
the neural network is initialized randomly, the initial density field is random and non-constant. To avoid this problem, we subtract the initial density field and add well-chosen constant:
Vo
Vi E NJ, x; (0) = [e(o)(z)) = [ece) - So(e=o) (zi) + log
(4)
N - Vo
We used equation 4 to compute X x(0) our numerical experiments.
Resulting donsity fldd
3420
3200
3322
230
200
50
20
Figure 2: Example of result of our method with applied forces (red arrow) and a fixed boundary (green). Here we used Gaussian embedding (see section for details).
Theoretical Analysis
3.1 Analogy between the Neural Tangent Kernel and filtering techniques
In our paper, we use the Neural Tangent Kernel (NTK [3] as the main tool to analyse the training behaviour of the FCNN. In our setting (where n1 = 1) the NTK defined as:
Σ ff
(2) 0fe (z') (Ve.o())
V2, 2' E Ro (2,
00,-
00,
This is positive semi-definite kernel. Given some inputs 21;-.; AN we define the NTK Gram matrix as: 6 (0 2,)) ISisEN E NNN.
Assuming small enough learning rate, the evolution of the network under gradient descent is well approximated by the gradient flow dynamics 0,6(t) = Ct)(t). The evolution of the output of the network N-NN can then easily be expressed in terms of the NTK Gram matrix [14 for a loss L:
a, (0(t)) - V X.
From this equation we can derive the evolution of the physical density field NN in our algorithm:
Proposition 3.1. If the network is trained under this gradient flow, then by applying chain rules, can prove that the density field follows the equation;
NN (0(1)) - -Dx P
(5)
The analogy between the NTK and filtering techniques comes from the following observation. With Modified Filtering with a filter T, we show similarly that the density field y evolves
MT (t)
-Dx(t)TT y (t)).
(6)

---

We see that the NTK Gram matrix and the squared filter TT play exactly the same role. An important difference however is that the NTK is random initialisation and evolves during training.
This difference disappears for large widths (when
are large), since the NTK converges to a deterministic and time independent limit θ% as .1..... nL-1 + o 13]. Furthermore, in contrast the finite width NTK (also called empirical NTK), we have access closed form formula for the limiting NTK 6 (given in the appendix).
In the infinite width limit, the evolution of the physical densities is then expressed in terms of the limiting NTK Gram matrix O%:
(OAN(((()) = - (xxxx()
From now on we will focus on this infinite- width limit, comparing the NTK Gram matrix θ% and the squared filter TT". Recent results 17 2 12 suggest that this limit is a good approximation when the width of the network is sufficiently large. For more details see the appendix, where we compare the empirical NTK with its limiting one and plot its evolution in our setting.
3.2 Spatial invariance
In physical problems such as topology optimisation, it is important to ensure that certain physical properties are respected by the model. We focus in this section the translation and rotation invariance of topology optimisation: if the force constraints are rotated or translated, the resulting shape should remain the same (up to rotation and translation), as in Figure 4 (b.1 and b.2).
In Modified Filtering method, this property guaranteed if the filter T is translation and rotation invariant. In contrast the limiting NTK is in general invariant under rotation K3 but not translation. As Figure 4 shows, this leads to some problematic artifacts. The NTK can be made translation and rotation invariant by first applying
embedding y Rd - ReO with the properties that for any two coordinates p. p', (pp)" (pp) only depends on the distance |p 2/1l2- Since the rotation invariance of the NTK implies that O (z, z²) depends only the scalar products 2 z', zz" and z'
have that @ (<((), (p')) depends only on ||p - p'll || as needed.
The issue is that for finite there is no non-trivial embedding & with this property:
Proposition 3.2. Let Rd + R0 for d > 2 and any finite no- If ip satisfies ((x)" =
[[|=- 1'l)) for some continuous function K then both y and K are constant.
To overcome this issue, we present two approaches to approximate spatial invariance with finite embeddings: an embedding on a (hyper)-torus and a random feature 23 embedding based on Bochner theorem 226
3.2.1 Embedding on a hypertorus
In this subsection we consider the following embedding of n2 X ny regular grid on a torus:
R² 3 p- (2),P2) + ((p) = r(cossóp)) sin((p)) cos(ōpp)) sin (6p2)),
(8)
where 8 > 0 is a discretisation angle (our default choice is 8 max( .t.y) One can use similar formulas for d (leading an hyper-torus embedding), we used d = equation g for simplicity. This embedding leads to an exact translation invariance and an approximate rotation invariance:
8² 2 lp r'11 + 0 (/r'11)
p(p)) = (cooss(5(p) - -1)) cos(0(p2 pb))) = ,
As a result, the limiting NTK Oxx (p)ppp(pp))) is translation invariant and approximately rotation invariant (for small and/or when p, p/ are close each other). Moreover, if we look the limiting NTK the whole torus, obtain that the gram matrix @. is discrete convolution the input grid, with nice properties summed up in the following proposition:
Proposition 3.3. We can always extend our ng X ny grid and choose 8 such that the embedded grid covers the whole torus (typically ò = m and take a X n. grid with n = 4 max Rly ).
5

---

terus embedding
NTK torus
NTK gausS
100
603
0.73
0.50
202
1/
0.25
embedding
0.2
10
b.2
10-
102
10-
ReLe-FCNN
0.02
20
50
b.3
1.03
0.73
0.90
Loss evolution
0.23
tanh-FCNN
0.00
b.4
Tonus entedking
10" Gauesinn enibedding
19th
0.02
20
30 50
Figure 4: Left: empirical NTK of FCNNs with both embedding (a.1, 1, a.2, see Section 4.1 for details) or without embedding (a.3 with ReLu, a.4 with tanh). Right: Corresponding shape obtained after training. Note that methods without spatial invariance particularly struggles with this symmetric load case (b.3, b.4) while both "embedded methods" respect the symmetry 1b. b.2). We also observed that training with non-embedded methods is very unstable
Then the Gram matrix 2 of the limiting NTK is 2D discrete convolution matrix. Moreover the NTK Gram matrix has positive definite square root / e which is also discrete convolution matrix.
As we know, the cigenvectors of such a convolu- tion matrix the 2D Fourier vectors. The cor- responding eigenvalues are the discrete Fourier transforms of the convolution kernel.
Xg
30
The square root of the NTK Gram matrix V @.x then corresponds to the filtering matrix T in our analogy. Figure 3 shows that on the full torus, the matrix square root / Og indeed looks like typical smoothing filter.
25
20
Figure 3: Representation of one line θn on the full torus and of its square root. We used / 0.2 and w = 3 (see Section[4.1) here to make the filter visible the whole torus.
As Figure 4 shows, the torus embedding method gives good numerical results and respect the symmetry of the applied forces F.
3.2.2 Random embeddings for radial kernels
Another approach to approximate rotation and translation invariant embedding is to use random Fourier features [23]. which is general method approximate shift invariant kernels of the form k(x, y) = k(x y). By Bochner theorem [26), any continuous non-zero radial kernel k(r - y) K [|l - y/ll can be written as the the (scaled) Fourier transform of a probability measure Q on Rd;
(rr) k(0) [ wuir (d..))
For radial kernels, we formulate random Fourier features embeddings & RRd - R as follows:
p(p):
22k(0) sin ( p 4
π
b,),

---

for i..dd samples Wi.... Wno ER' from Q (which is also invariant by rotation) and i.idd samples i1 ... Uno E Rd from any symmetric probability distribution (or uniform laws on [0, 2T)). By the law of large numbers for large no. we have the approximation ni ==)) (((') k(p p').
Gaussian embedding: Depending on the kernel k that we want to approximate, it may be difficult
to sample from the distribution Q The simplest case is for a Gaussian kernel k(d) = 277 de where the distribution Q of the weights w; is (0, la). i.e. the entries wij are all i.i.d. (0, #) Gaussians. For this reason this is the embedding that we will use in our numerical experiments. Note the similarity between this type of embedding and untrained first layer of a FCNN with sine activation function, weights wy and bias b;
Moreover, the following result shows that we can still define a "square root" of the NTK with those types of embedding and thus complete the analogy with equation 6
Proposition 3.4. Let yp be an embedding as described above for a positive radial kernel & E L'' (RR)) with (())= = 1, k 2 0. Then there is filter function g R - R and constant C such that for all p.p':
lim 4 ( (DD1) C (g (g* pp)),
70-+00
where Oo is the limiting NTK of a network with a Lipschitz, non-constant and standardised activation function μ. (Here denotes the convolution product).
As the matrix Dx in equation | cancels out the constant frequency (proposition 2.1.. the constant C doesn't 't matter, i.e. 6
(L) Dx = Dx (o% C) Dx-
4
Experimental analysis
4.1 Setup
Most of our experiments were conducted with torus embedding or a gaussian embedding. For the SIMP algorithm, adapted the code described in [8] Here are the hyperparameters used in the experiments.
For the Gaussian embedding, we used no 1000 and length scale é= = 4. This embedding was followed by one hidden linear layer of size 1000 with standardized ReLu (x + v2 max(0, x)) and bias parameter 3 = 0.5.
For the torus embedding we set the torus radius discretisation angle to 8= 2 m,, (to cover roughly half the torus, which is a good trade-off between rotation invariance and kernel size), where ntz X My is the size of the grid. It was followed by 2 linear layers size 1000 with 33 = 0.1. The ReLu activation is not well-suited in this case because it induces filters that too wide. The large radius of the NTK kernel can be understood in relation with the order/chaos regimes [27] 22). as observed in [4] the ReLU lies in the ordered regime when 3 >> 0, leading to "wide" kernel, a narrower kernel can be achieved with non- linearities which lie in the chaotic regime instead. We used a cosine activation of the form x + cos(wz), which has the advantage that the width of the filter can be adjusted using the hyperparameter, see Section 7 When not stated otherwise we used w = 5.
V2 (to be standard sphere) and the
Even though our theoretical analysis is for gradient flow, we obtain similar results with other optimizers such as RPROP (24) (learning rate 10-3) and ADAM [66 (learning rate 10-3). RPROP gave the fastest results, possibly because it is well-suited for batch learning (25). Vanilla gradient descent can be very slow due to the vanishing of the gradients when the image becomes almost binary (due to the sigmoid), we therefore gradually increased the learning rate during training to compensate.
4.2 Spectral analysis
In SIMP convolution with a low pass filter ensures that low frequencies are optimised faster than high frequencies, to avoid checkerboards.
With the embeddings proposed in the last two subsections, the limiting NTK takes the form of a convolution over the input space ıdd. Figure represents the eigenvalues and eigenimages of the
7

---

- 12 59187
A 65 4744744
1-2752289
Figure 5: Sorted eigenvalues of the empirical NTK with some eigenvectors (reshaped im- ages). Obtained with a Gaussian embedding.
Figure 6: Colormap of R /2 in the (3, w) plane, torus embedding. Level lines and shapes ob- tained for different radius are represented.
NTK Gram matrix θote)- Even though this plot is done for a finite width network and a finite random embedding,
that the cigenimages look like 2D Fourier modes. The fact that the low frequencies have the largest eigenvalues supports the similarity between the NTK and low pass filter.
This may explain why neural networks naturally avoid checkerboard patterns: the low frequencies of the shape are trained faster than the high frequencies which lead to checkerboard patterns.
4.3 Filter radius
In the classical SIMP algorithm, the choice of the radius of the filter T is critical. It controls the appearance of checkerboards or intermediate densities.
R12 0.49
Rı 0.98
When using DNNs, there is no explicit choice of filter radius, since the filter depends on the embedding and the architecture of the network. In Section 3.2 we have shown that the NTK is approximately invariant, it can hence be expressed as:
R1 1.37
R1 1.96
(vPl),
4 (1p p'l))
where &x can be analytically expressed with the embed- ding and the limiting NTK (see appendix for detailed example).
The kernels we consider do not have compact support in general, we therefore focus instead on the radius at half-maximmn of &
Figure 7: Shape obtained for different val- ues of R2/2 with Gaussian embedding for different values of & E (0.5. 1, 1.4.2).
1 00 ( (0) inf (r)
L (R2
Note that for simplicity we are computing here the ra- dius of the squared filter, since obtaining closed form formula for the square root of the NTK is more difficult. For Gaussian filters the radius of the squared filter is / times that of the original, suggesting that the filter radius is well estimated by 1 R1/2-
v2

---

The quantity /2 is function of the hyperparameters of the network (a.5, L, see appendix) and of the embedding (the lengthscale f). Using the formula for Ra,2. these hyperparameters can be tuned to obtain specific filter radius.
With the Gaussian embedding. the radius of the filter can easily be adjusted by changing the length- scale of the embedding. As illustrated in Figure 7
With the torus embedding, instead have change the hyperparameters of the network adjust the radius of the filter. With the ReL.U activation function, the radius very large which makes impossible obtain precise shape. The solution found is how the radius decreases as w increases. The / parameter
cosine activation æ cos(wxr
with sypee-paaraeter w. T has the opposite effect, increasing it increases the radius. For different values of w and 3, obtain a variety of radius and plot the resulting shapes. This plot also illustrates the role of the radius in the determination of the resulting shape. The fact that cosine activation leads an adjustable NTK radius could explain why periodic activation function help in the representation of high frequency signal as observed in 30).
The effect of depth is more complex. For large depths L the NTK either approaches a constant kernel in the so-called order regime (with infinite radius) Kronecker delta kernel in the so-called chaos regime (with zero radius) [2 27 14. Depending on whether we are in the order or chaos regime (which is determined by the activation function and the parameters o; B), increasing the depth can either increase or decrease the radius.
We conducted an experimental study of the influence of this parameter on the geometry of the final shape. We observed that its complexity (number of holes, high frequencies) highly controlled by R1/2-
We see in Figure and 6 some examples of shape obtained for several values of R1
4.4 Up-sampling
Since the density field is generated by DNN, it can be evaluated any point in Rdd, hence allowing upsampling. As Figure 8 shows, with our method we obtain a smooth and binary shape. Something interesting happens when the network is trained without an embedding: when upsampling observe We believe that it is due to the lack of spatial invariance.
some visual artifacts plotted in Figure 9
Note that this second experiment was done with batch norm, as described in 77. since for this problem was difficult obtain good shape with a vanilla ReLU-FCNN. With our embeddings, we can achieve complex shapes without batch-norm.
Resutting density Reld
Up-sampling
Figure 8: Density field obtained with Torus embedding (left) and up sampling of factor of the same network (right).
Figure 9: Exemple of up-sampling of a FCNN (ReLu FCNN with batchnorms) without embed- ding, exhibing typical visual artifacts.
5 Conclusion
Using the NTK, we were able to give a simple theoretical description of topology optimisation with DNNs, showing a similarity to traditional filtering techniques. This theory allowed to identify problem: since the NTK is not translation invariant, the spatial invariance topology optimisation is not respected, leading to visual artifacts and non-optimal shapes. We propose simple solution to this problem: adding a spatial invariant embedding to the coordinates before the DNN.
Using this method, our models are able to learn efficient shapes while avoiding checkerboard patterns. We give tools to adjust the implicit filter size induced by the hyperparameters, to give control over the complexity of the final shape. Using the learned network, we can easily perform good quality

---

up-sampling The techniques described in this paper can easily be translated to any other problem where spatial invariance is needed.
The NTK is a simple yet powerful tool to analyse practical method such as SIMP when combined with a DNN. Morover it can be used to make informed choices of the DNN's architecture and hyperparameters.
Acknowledgments and Disclosure of Funding
There is no funding or competing interests associated to this work.
References
[1] Erik Andreassen, Anders Clausen, Mattias Schevenels, Boyan Lazarov, and Ole Sigmund. Effi- cient topology optimization in matlab using 88 lines of code. Structural and Multidisciplinary Optimization, -3:1-16, 2011.
[2] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. CoRR, 0bss/9044.11555 2019.
[3] Saurabh Banga, Harsh Gehani, Sanket Bhilare, Sagar Patel, and Levent Kara. 3d topology optimization using convolutional neural networks. CoRR, abs/1808.07440. 2018.
[4] Bendsoe and Sigmund. Topology optimization: Theory, methods and applications. Springer Science and Business, April 2013.
[5] Martin Bendsøe. Bendsoe, m.p.: Optimal shape design as material distribution problem. structural optimization 1, 193-202. Structural Optimization,
01 1989.
[6] Aaditya Chandrasekhar and K. Suresh. Length scale control in topology optimization using fourier enhanced neural networks. 2020.
[7] Aaditya Chandrasekhar and Krishnan Suresh. Tounn: Topology optimization using neural
networks. Structural and Multidiscipllnary Optimization, 2020.
[8] Yanqing Chen, Timothy A. Davis, and William W. Hager. Algorithm 887: Cholmod, supernodal
sparse cholesky factorization and update/downdate. ACM Transactions on Mathematical Software, pages 1-14, 2008.
[9] Timothy A. Davis. User guide for cholmod: sparse cholesky factorization and modification package. 2009.
[10] Andreas Griewank and Christèle Faure. Reduced functions, gradients and hessians from fixed- print iterations for state equations. Numerical Algorithums, 30:113 13-139, 06 2002.
[11] Stephan Hoyer, Jascha Sohl-Dickstein, and Sam Greydanus. Neural reparameterizztioo improves structural optimization. CoRR, abs/1 909.9999000 909.04240, 2019.
[12] Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. CoRR, abs/1909.08156, 2019.
[13] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. CoRR, 806.07572. 2018.
[14] Arthur Jacot, Franck Gabriel, and Clément Hongler. Order and chaos: NTK views DNN normalization, checkerboard and boundary artifacts. CoRR, abs/1907.05777 05515, 2019.
[15] Jiaqi Jiang and Jonathan A. Fan. Global optimization of dielectric metasurfaces using a physics- driven neural network. Nano Letters, 19(8):536--533 Jul 2019.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2017.
[17] Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2020( 124022, Dec 2020.
[18] Kai Liu and Andres Tovar. An efficient 3d topology optimization code written in matlab. Structural and Multidisciplinary Optimization, 50, 12 2014.
10

---

[19] Gilles Marck. Maroun Nemer, Lean-Luc Harion. Serge Russeil, and Daniel Bougeard. Topology optimization using the simp method for multiobjective conductive problems. Numerical Heat Transfer Part B-fundamentals- NUMER HEAT TRANSFER PT B- B-FUND, 61 :439-470, 06 2012.
[20] Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parameterizations. Distill, 2018. https://distill.pubb// ottt:
[21] Zhenguo Nie, Tong Lin, Haoliang Jiang, and Levent Burak Kara. Topologygan: Topology optimization using generative adversarial networks based on physical fields over the initial domain. CoRR, abs/2003.046835 2020.
[22] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnetr, editors, Advances in Neural Information Processing Systems 29, pages 3360-3368. Curran Associates, Inc., 2016.
[23] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008.
[24] M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: the rprop algorithm. In IEEE International Conference on Neural Networks, pages 586-591 vol.1, 1993.
[25] Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster backpropagation learning: The rprop algorithm. pages 586-591, 1993.
[26] W. Rudin. Fourier Analysis on Groups. Wiley Classics Library. Wiley, 1990.
[27] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep infor- mation propagation. 2017.
[28] M.-H. Herman Shen and Liang Chen. A new CGAN technique for constrained topology design optimization. CoRR, abs/1901.07675, 2019.
[29] Ole Sigmund. Morphology based black and white filters for topology optimization. Structural and Multidisciplinary Optimization, 33:401-424, 04 2007.
[30] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gor- don Wetzstein. Implicit neural representations with periodic activation functions. CoRR, abs/2006.09661, 2020.
[31] Ivan Sosnovik and Ivan V. Oseledets. Neural networks for topology optimization. CoRR,
abs/1709.09578, 2017.
[32] Matthew Taneik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. CoRR, abs/2006.10739, 2020.
[33] H. Vorst. Bi-cgstab: A fast and smoothly converging variant of bi-cg for the solution of nonsymmetric linear systems. SIAM J. Sci. Comput., 13:631-644, 1992.
[34] Luzhong Yin and Wei Yang. Optimality criteria method for topology optimization under multiple constraints. Computers and Structures, 79(20): 1811118800,, 2001.


Reviews:
[{"review_id": "tHsZ64kd3Q-", "reviewer": "Reviewer_r379", "summary": "The paper introduces a deep fully-connected neural network (FCNN) approach to topology optimization, in which a neural network learns to predict density as a function of spatial location, with gradient feedback based on Solid Isotropic Material Penalization (SIMP). The authors draw a connection between the NTK Gram matrix of this FCNN and the spatial low-pass filtering usually done in non-neural topology optimization, and propose two input embeddings that induce shift-invariance in the NTK. Instead of tuning the filtering radius of an explicit low-pass filter, with the proposed FCNN approach the effective filtering can be tuned by adjusting parameters of the embedding and the network architecture to vary the width of the NTK and control its spectrum. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is clear and interesting. The authors demonstrate that their proposed embeddings induce shift-invariance and the ability to tune the bandwidth of the resulting NTK. These adjustments to the embedding (and in some cases, network architecture) translate to interpretable changes in the resulting optimized shapes, and a marked improvement over an FCNN without an input embedding. However, I have some questions/concerns, especially regarding prior work and limitations, that make me question how novel and how useful the proposed approach really is.\n\nPrior work:\n\nThe discussion of prior work is limited to the introduction, and focuses almost exclusively on the problem of topology optimization. There has been a wealth of recent progress on different problems e.g. in graphics, in which the goal is again to model a structure, using a FCNN to map spatial coordinates to density (and sometimes other properties). These works also make use of an input embedding (e.g. https://arxiv.org/abs/2006.10739) and/or a periodic activation function (e.g. https://arxiv.org/abs/2006.09661), quite similar to the proposed Gaussian random Fourier feature embedding and torus embedding with cosine activations, respectively. Although to my knowledge the application of this technique to the problem of topology optimization is novel, the embeddings themselves are less so.\n\nAdditionally, improvement is only shown with respect to a FCNN with no embedding; no other prior techniques are compared against. Is there a benefit to using the proposed approach instead of e.g. the classical filtering method?\n\nLimitations/other questions:\n\nAlthough the experiments are compelling, they are limited to two-dimensional problems. Is this standard in topology optimization? An experiment in three dimensions would be even more convincing.\n\nWhat are the strengths and weaknesses of each of the two proposed embedding methods? Why introduce both unless there are some situations where either is preferable to the other?\n\nIn section 1.1 (first bullet point), doesn’t the equivalence to the NTK also require an assumption of gradient flow (infinitesimal learning rate), in addition to the assumption of infinite width?\n\nIf the torus embedding uses a periodic activation function, how can you separate the effects of using the embedding from the effects of the activation function?\n\n\nMinor/low-level comments/questions:\n\nSIMP is mentioned in the abstract, without being defined until the introduction. \n\nIn a few places (equations 2, 5, and 6 in particular), a formula is introduced with no explanation or derivation. Although derivations are available in the supplement, it would be helpful to have a bit more intuition/explanation provided in the main text. \n\nLine 72: An equality condition on the sum of the densities is described as a volume constraint. Would it be more accurate to call this a mass constraint?\n\nLine 83: The distinction between X and \\bar X is not clear. \n\nLine 87-88: This might be more clearly worded by reversing the order, i.e. D_X is PSD with all eigenvalues less than ½, and its nullspace is… (especially since the word “kernel” is a bit overloaded).\n\nLine 91: Having a scalar output is a feature of the FCNNs used in this paper, but not a general property of all FCNNs (as is implied).\n\nFigure 1: The figure and caption use X and Y, but the surrounding text includes superscripts NN. Are these the same quantities?\n\nLine 138-139: Do the z’s need to be normalized for this to hold?\n\nLine 149: Where does the 4-norm in the error term come from? \n\nLast line on page 6: Why the need to add pi/4? Isn’t this subsumed by b_i?\n\nEquation 9: What does star denote?\n\nTypos: \n\nLine 63-64: comma splice\n\nLine 77: “it can used” -> “it can be used”\n\nFigure 4 caption: Left and right are reversed\n\nLine 195: “lenghtscale” -> “length scale”\n\nLine 238: “hyperparameters network” -> “hyperparameters of the network”\n\nLine 243: “illustrate” -> “illustrates”\n\nLine 255, 257: bad figure reference (should be figures 8, 9)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper would benefit from a discussion of limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KT1ODMH7Rou", "reviewer": "Reviewer_XBF1", "summary": "Built upon the NTK, this paper provides a theoretical understanding of using deep neural networks in the SIMP method for topology optimization in the infinite-width regime. The authors show that: 1) Topology optimization with a DNN is equivalent to topology optimization with the square root of NTK as the density filter. 2) They also propose two embeddings to ensure the spatial invariance of the NTK. 3) They study how the choice of embedding, activation function, depth, etc., affects the radius of the NTK.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper provides a theoretical understanding of using DNNs in the SIMP method for topology optimization and offers practical guidance on choosing the activation function, the choice of embedding, the depth of the neural networks for adjusting the radius of the NTK. The results in the paper can be good references for practitioners working in this field. However, the theoretical analysis primarily builds on the NTK theories developed in [11], limiting the theoretical novelty. Conversely, the proposed two embeddings for inducing spatial invariance are novel, to my knowledge. These two embeddings can also be useful for other areas where spatial invariance is required.  In the meantime, I have the following questions:\n\n\n1) Since the approximation error for the torus embedding depends on the grid size, I wonder how to choose the grid size in practice? Also, how does the approximation error propagate? Will the later steps (e.g., the pointwise FCNN) amplify it?  \n\n\n2) For the two embeddings, the torus embedding, and the Gaussian embedding, what's the advantage of each other?  It would be great to discuss this and provide suggestions on the choice of embedding for some specific scenarios. By doing so, it can better relate the theories to practical problems in topology optimization.\n\n\nI am not familiar with the research in topology optimization, so my assessment is based on the theoretical analysis in the paper. Overall, I am neutral to this paper but more on acceptance due to the above reasons.\n\n====Post rebuttal=====\n\nThanks for the response! I would suggest authors include these discussions in the appendix. My score remains unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation was discussed in the paper. The authors did not discuss the negative social impact considering that their work is theoretical.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "HL1kiynvvn", "reviewer": "Reviewer_cu3P", "summary": "The paper studies topology optimization with neural networks. In particular, they use Deep neural networks to generate the density at each point used by Solid Isotropic Material Penalisation (SIMP) methods. The deep neural network has the input as the coordinates and the output is density (the amount of material) at that coordinate.  \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n1) In equation 1, how is it guarantees that there is a fix bias terms $\\bar{b}$ for all the elements in the grid that would satisfy the conservation of $V_{0}$ for any arbitrary $x_{i}$'s and $V_{0}$? Is it the reasoning for that is line 13 of your supplement? I think it would be good to refer in the main text that you answer to that question in the supplement. Also don't you need a condition that $V_{0} \\leq N$? In the proof you used $\\bar{b}$ as both the unique bias and also the graph function that is the consequence of Implicit Theorem, it is better to different this two notions. \n\n\n2) Proposition 2.1 proof: I don't see why $F(\\lambda) = 0$ only happens in open intervals $(a_{j},a_{i})$? Furthermore, how does this prove that the eigenvalues are between $[0,0.5]$?\n\n\n3) In the experimental setup how did you choose the parameters for your neural network and the embedding? is there any sort of cross-validation involved?\n\n\n4) Could've you not fix that translational invariancy via using equivariant networks, such networks exist, \"Scaling-Translation-Equivariant Networks with Decomposed Convolutional Filters\", it would've been nice to see the result in comparison with NTK?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the application of deep neural networks to the "SIMP" method for topology optimization. It proposes a novel approach for doing this and analyzes it using NTK theory.

This application is well outside the field of expertise of the reviewers, or indeed probably everyone associated with NeurIPS, including myself. That being said, the reviewers seem to think this is a good well-written contribution, with theoretical insights that might generalize beyond this particular application.
Decision: Accept (Poster)