Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Built upon the NTK, this paper provides a theoretical understanding of using deep neural networks in the SIMP method for topology optimization in the infinite-width regime. The authors show that: 1) Topology optimization with a DNN is equivalent to topology optimization with the square root of NTK as the density filter. 2) They also propose two embeddings to ensure the spatial invariance of the NTK. 3) They study how the choice of embedding, activation function, depth, etc., affects the radius of the NTK.

main_review: 
This paper provides a theoretical understanding of using DNNs in the SIMP method for topology optimization and offers practical guidance on choosing the activation function, the choice of embedding, the depth of the neural networks for adjusting the radius of the NTK. The results in the paper can be good references for practitioners working in this field. However, the theoretical analysis primarily builds on the NTK theories developed in [11], limiting the theoretical novelty. Conversely, the proposed two embeddings for inducing spatial invariance are novel, to my knowledge. These two embeddings can also be useful for other areas where spatial invariance is required.  In the meantime, I have the following questions:


1) Since the approximation error for the torus embedding depends on the grid size, I wonder how to choose the grid size in practice? Also, how does the approximation error propagate? Will the later steps (e.g., the pointwise FCNN) amplify it?  


2) For the two embeddings, the torus embedding, and the Gaussian embedding, what's the advantage of each other?  It would be great to discuss this and provide suggestions on the choice of embedding for some specific scenarios. By doing so, it can better relate the theories to practical problems in topology optimization.


I am not familiar with the research in topology optimization, so my assessment is based on the theoretical analysis in the paper. Overall, I am neutral to this paper but more on acceptance due to the above reasons.

====Post rebuttal=====

Thanks for the response! I would suggest authors include these discussions in the appendix. My score remains unchanged.
limitations_and_societal_impact: The limitation was discussed in the paper. The authors did not discuss the negative social impact considering that their work is theoretical.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies topology optimization with neural networks. In particular, they use Deep neural networks to generate the density at each point used by Solid Isotropic Material Penalisation (SIMP) methods. The deep neural network has the input as the coordinates and the output is density (the amount of material) at that coordinate.  


main_review: 

1) In equation 1, how is it guarantees that there is a fix bias terms $\bar{b}$ for all the elements in the grid that would satisfy the conservation of $V_{0}$ for any arbitrary $x_{i}$'s and $V_{0}$? Is it the reasoning for that is line 13 of your supplement? I think it would be good to refer in the main text that you answer to that question in the supplement. Also don't you need a condition that $V_{0} \leq N$? In the proof you used $\bar{b}$ as both the unique bias and also the graph function that is the consequence of Implicit Theorem, it is better to different this two notions. 


2) Proposition 2.1 proof: I don't see why $F(\lambda) = 0$ only happens in open intervals $(a_{j},a_{i})$? Furthermore, how does this prove that the eigenvalues are between $[0,0.5]$?


3) In the experimental setup how did you choose the parameters for your neural network and the embedding? is there any sort of cross-validation involved?


4) Could've you not fix that translational invariancy via using equivariant networks, such networks exist, "Scaling-Translation-Equivariant Networks with Decomposed Convolutional Filters", it would've been nice to see the result in comparison with NTK?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a deep fully-connected neural network (FCNN) approach to topology optimization, in which a neural network learns to predict density as a function of spatial location, with gradient feedback based on Solid Isotropic Material Penalization (SIMP). The authors draw a connection between the NTK Gram matrix of this FCNN and the spatial low-pass filtering usually done in non-neural topology optimization, and propose two input embeddings that induce shift-invariance in the NTK. Instead of tuning the filtering radius of an explicit low-pass filter, with the proposed FCNN approach the effective filtering can be tuned by adjusting parameters of the embedding and the network architecture to vary the width of the NTK and control its spectrum. 

main_review: Overall, the paper is clear and interesting. The authors demonstrate that their proposed embeddings induce shift-invariance and the ability to tune the bandwidth of the resulting NTK. These adjustments to the embedding (and in some cases, network architecture) translate to interpretable changes in the resulting optimized shapes, and a marked improvement over an FCNN without an input embedding. However, I have some questions/concerns, especially regarding prior work and limitations, that make me question how novel and how useful the proposed approach really is.

Prior work:

The discussion of prior work is limited to the introduction, and focuses almost exclusively on the problem of topology optimization. There has been a wealth of recent progress on different problems e.g. in graphics, in which the goal is again to model a structure, using a FCNN to map spatial coordinates to density (and sometimes other properties). These works also make use of an input embedding (e.g. https://arxiv.org/abs/2006.10739) and/or a periodic activation function (e.g. https://arxiv.org/abs/2006.09661), quite similar to the proposed Gaussian random Fourier feature embedding and torus embedding with cosine activations, respectively. Although to my knowledge the application of this technique to the problem of topology optimization is novel, the embeddings themselves are less so.

Additionally, improvement is only shown with respect to a FCNN with no embedding; no other prior techniques are compared against. Is there a benefit to using the proposed approach instead of e.g. the classical filtering method?

Limitations/other questions:

Although the experiments are compelling, they are limited to two-dimensional problems. Is this standard in topology optimization? An experiment in three dimensions would be even more convincing.

What are the strengths and weaknesses of each of the two proposed embedding methods? Why introduce both unless there are some situations where either is preferable to the other?

In section 1.1 (first bullet point), doesn’t the equivalence to the NTK also require an assumption of gradient flow (infinitesimal learning rate), in addition to the assumption of infinite width?

If the torus embedding uses a periodic activation function, how can you separate the effects of using the embedding from the effects of the activation function?


Minor/low-level comments/questions:

SIMP is mentioned in the abstract, without being defined until the introduction. 

In a few places (equations 2, 5, and 6 in particular), a formula is introduced with no explanation or derivation. Although derivations are available in the supplement, it would be helpful to have a bit more intuition/explanation provided in the main text. 

Line 72: An equality condition on the sum of the densities is described as a volume constraint. Would it be more accurate to call this a mass constraint?

Line 83: The distinction between X and \bar X is not clear. 

Line 87-88: This might be more clearly worded by reversing the order, i.e. D_X is PSD with all eigenvalues less than ½, and its nullspace is… (especially since the word “kernel” is a bit overloaded).

Line 91: Having a scalar output is a feature of the FCNNs used in this paper, but not a general property of all FCNNs (as is implied).

Figure 1: The figure and caption use X and Y, but the surrounding text includes superscripts NN. Are these the same quantities?

Line 138-139: Do the z’s need to be normalized for this to hold?

Line 149: Where does the 4-norm in the error term come from? 

Last line on page 6: Why the need to add pi/4? Isn’t this subsumed by b_i?

Equation 9: What does star denote?

Typos: 

Line 63-64: comma splice

Line 77: “it can used” -> “it can be used”

Figure 4 caption: Left and right are reversed

Line 195: “lenghtscale” -> “length scale”

Line 238: “hyperparameters network” -> “hyperparameters of the network”

Line 243: “illustrate” -> “illustrates”

Line 255, 257: bad figure reference (should be figures 8, 9)
limitations_and_societal_impact: The paper would benefit from a discussion of limitations.
review_title: null

Meta-review: This paper considers the application of deep neural networks to the "SIMP" method for topology optimization. It proposes a novel approach for doing this and analyzes it using NTK theory.

This application is well outside the field of expertise of the reviewers, or indeed probably everyone associated with NeurIPS, including myself. That being said, the reviewers seem to think this is a good well-written contribution, with theoretical insights that might generalize beyond this particular application.
Decision: Accept (Poster)