DNN-based Topology Optimisation:  Spatial Invariance and Neural Tangent Kernel

1 Introduction . Topology optimisation [ 4 ] , also known as structural optimisation , is a method to find optimal shapes subject to some constraints . It has been widely studied in the field of computational mechanics . Here we are interested in the particular case of the Solid Isotropic Material Penalisation ( SIMP ) method [ 18 , 1 ] , which is a very common method in this field . Recently some authors have used Deep Neural Networks ( DNNs ) to perform topology optimisation . We can differentiate two different approaches in the use of DNNs with SIMP . The first approach consists in generating with the classical algorithms a dataset of optimised shapes and train a DNN on this dataset to produce new optimal shapes [ 3 , 31 ] . Variations of this approach use Generative Adversarial Networks ( GAN ) [ 21 , 28 ] to effectively reproduce classical topology optimisation . In the second approach , the density is generated pointwise by a DNN , which is trained with gradient descent to optimise the density field with respect to the physical constraints , as proposed in [ 11 ] to use the power of deep models without giving up exact physics . We focus on the approach of [ 7 , 6 ] where the density field is generated by a Fully-Connected Neural Network ( FCNN ) taking the coordinates of a grid as inputs . Surprisingly , [ 7 ] observes that the DNN-generated density fields do not feature checkerboard artifacts , which are common in vanilla SIMP . A traditional method to avoid checkerboard patterns is to add a filter [ 29 , 5 ] , but it is not needed for DNN-generated density fields . In this paper , we analyse theoretically how the use of a DNN to generate the density field affects the learning . Our main theoretical tool is the Neural Tangent kernel ( NTK ) introduced in [ 13 ] to describe the dynamics of wide neural networks [ 13 , 2 , 17 , 12 ] . While this paper focuses on linear elasticity and SIMP , our analysis can extended to other physical problems such as heat transfer [ 19 ] , or any model where an image is generated by a DNN taking the pixel coordinates as inputs ( like in [ 20 ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Our contribution . In this paper we study topology optimisation with neural networks . The physical density is represented by a neural network taking an embedding of spatial coordinates as inputs , i.e . the density at a point x ∈ Rd is given by fθ ( ϕ ( x ) ) for θ the parameters of the network and ϕ an embedding . We use theoretical tools , in particular the Neural Tangent Kernel ( NTK ) , to understand how the architecture and hyperparameters of the network affect the optimisation of the density field : • We show that in the infinite width limit ( when the number of neurons in the hidden layers grows to infinity ) , topology optimisation with a DNN is equivalent to topology optimisation with a density filter equal to the “ square root ” of the NTK . Filtering is a commonly used technique in topology optimisation , aimed to remove checkerboard patterns . • In topology optimisation as in other physical optimisation problems , it is crucial to guarantee some spatial invariance properties . If the coordinates are taken as inputs of the network directly , the NTK ( and the corresponding filter ) is not translation invariant , leading to nonoptimal shapes and visual artifacts . We present two methods to ensure the spatial invariance of the NTK : embedding the coordinates on the ( hyper- ) torus or using a random Fourier features embedding ( similar to [ 32 ] ) . • In traditional topology optimisation , the filter size must be tuned carefully . When optimising with a DNN , the filter size depends on the embedding of the coordinates and the architecture of the network . We define a filter radius for the NTK , which plays a similar role as the classical filter size and discuss how it is affected by the choice of embedding , activation function , depth and other hyperparameters like the importance of bias in the network . This tradeoff can also be analysed in terms of the spectrum of the NTK , explaining why neural networks naturally avoid checkerboard patterns . We confirm and illustrate these theoretical observations with numerical experiments . Our implementation of the algorithm will be made public at https : //github.com/benjiDupuis/DeepTopo . 2 Presentation of the method . In this paper , we use a DNN to generate the density field used by the Solid Isotropic Material Penalisation ( SIMP ) method . Our implementation of SIMP is based on [ 1 ] and [ 18 ] . In this section we introduce the traditional SIMP method and our neural network setting . 2.1 SIMP method . We consider a regular grid of N elements where the density of element i is denoted yi ∈ [ 0 , 1 ] , informally the value yi represents the presence of material at a point i . Our goal is to optimise over the density y ∈ RN to obtain a shape that can withstand forces applied at certain points , represented by a vector F . The method uses finite element analysis to define a stiffness matrix K ( y ) ∈ S++N ( R ) from the density y and computes the displacement vector U ( y ) ( which represent the deformation of the shape at all points i as a result of the applied forces F ) by solving a linear system K ( y ) U ( y ) = F . In our implementation , we performed it either by using sparse Cholesky factorisation [ 9 , 8 ] or BICGSTAB method [ 33 ] ( this last one can be used for a high number of pixels ) . The loss function is then defined as the compliance C ( y ) = U ( y ) TK ( y ) U ( y ) , under a volume constraint of the form ∑N i=1 yi = V0 , with 0 ≤ V0 ≤ N ( see [ 1 , 18 ] ) . 2.2 A modified SIMP approach . Several methods exist to optimise the density field y ∈ RN , such as gradient descent or the so-called Optimality Criteria ( OC ) [ 34 ] . We propose here an optimisation method inspired from [ 11 ] which we will refer as the Modified Filtering method ( MF ) . The advantage of this method is that it can be used with or without DNNs , hence allowing comparison between these two approaches . We first present here the model without DNNs . In our method , the densities yMFi are given by : ∀i ∈ { 1 , ... , N } , yMFi = σ ( xi + b̄ ( X ) ) , with b̄ ( X ) such that N∑ i=1 yMFi = V0 , ( 1 ) for X = ( x1 , ... , xN ) ∈ RN and the sigmoid σ ( x ) = 11+e−x . We will denote this operation as : Y MF = Σ ( X ) . The sigmoid ensures that densities are in [ 0 , 1 ] and the choice of the optimal bias b̄ ( X ) ensures that the volume constraint is satisfied . Filtering : If the vector X is optimised directly with gradient descent , SIMP often converges toward checkerboard patterns , i.e . some high frequency noise in the image , which is a common issue with SIMP [ 1 ] . To overcome this issue a common technique is to use filtering [ 29 ] . In this paper , we consider low-pass density filters of the form : X = TX̄ where T represents a convolution on the grid , X̄ are the design variables and X is the vector in equation 1 . The loss function of this method is then naturally defined as : X̄ 7−→ C ( Σ ( TX̄ ) ) . The gradient∇Y C is easily obtained by the self-adjointness of the variational problem [ 34 , 15 ] . We recover∇XC from∇Y C using an implicit differentiation technique [ 10 ] . The following proposition is a consequence of implicit function theorem and chain rules : Proposition 2.1 . Let Ṡ be the vector with entries σ̇ ( xi + b̄ ( X ) ) . We have ∇XC = DX∇Y C with : DX : = − 1 |Ṡ|1 ṠṠT + Diag ( Ṡ ) . ( 2 ) where |.|1 denotes the l1 norm of a vector . Furthermore DX is a symmetric positive semi-definite matrix whose null-space is the space of constant vectors and has eigenvalues smaller than 14 . 2.3 Proposed algorithm : SIMP with Neural networks . Fully-Connected Neural Networks ( FCNN ) are characterised by the number of layers L + 1 , the numbers of neurons in each layer ( n0 , n1 , ... , nL ) and an activation function µ : R −→ R , here we will use the particular case nL = 1 . The activations al ∈ Rnl and preactivations ãl ∈ Rnl are defined recursively for all layers l , using the so-called NTK parameterisation [ 13 ] : a0 ( x ) = x , ãl+1 ( x ) = α √ nl W lal ( x ) + βbl , al+1 ( x ) = µ ( ãl+1 ( x ) ) , ( 3 ) for some hyperparameters α , β ∈ [ 0 , 1 ] representing the contribution of the weights and bias terms respectively . The parameters θ = ( θp ) p , consisting in weight matrices W l and bias vectors bl are drawn as i.i.d . standard normal random variables N ( 0 , 1 ) . We denote the output of the network as fθ ( x ) = ã L ( x ) . Remark : To ensure that the variance of the neurons at initialization is the equal to 1 at all layers , we choose α and β such that α2+β2 = 1 and use a standardised non-linearity , i.e . EX∼N ( 0,1 ) [ µ ( X ) 2 ] = 1 ( [ 14 ] ) . In our approach , the pre-densities XNN ( θ ) = ( xNN1 , ... , x NN N ) are generated by a neural network as xNNi = fθ ( zi ) where zi ∈ Rn0 is either the coordinates of the grid elements ( in this case n0 = d ) or an embedding of those coordinates . We then apply the same transformation Σ to obtain the density field Y NN ( θ ) = Σ ( XNN ( θ ) ) . Our loss function is then defined as : θ 7−→ C ( Y NN ( θ ) ) = C ( Σ ( X ( θ ) ) ) . The design variables are now the parameters θ of the network . The gradient∇θC w.r.t . to the parameters is computed by first using Proposition 2.1 to get∇Y NNC followed by traditional backpropagation . Remark : Note the absence of filter T in the above equations , indeed we will show how neural networks naturally avoid checkerboard patterns , making the use of filtering obsolete . Initial density field : The SIMP method is usually initialised with a constant density field [ 1 ] . Since the neural network is initialized randomly , the initial density field is random and non-constant . To avoid this problem , we subtract the initial density field and add a well-chosen constant : ∀i ∈ { 1 , ... , N } , xi ( θ ) = f̄θ ( t ) ( zi ) = fθ ( t ) ( zi ) − fθ ( t=0 ) ( zi ) + log ( V0 N − V0 ) . ( 4 ) We used equation 4 to compute X ( θ ) in our numerical experiments .

Summary: Built upon the NTK, this paper provides a theoretical understanding of using deep neural networks in the SIMP method for topology optimization in the infinite-width regime. The authors show that: 1) Topology optimization with a DNN is equivalent to topology optimization with the square root of NTK as the density filter. 2) They also propose two embeddings to ensure the spatial invariance of the NTK. 3) They study how the choice of embedding, activation function, depth, etc., affects the radius of the NTK.