Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose Simulation Transformer (SiT), a transformer based approach for particle-based fluid simulations (in contrast to all the existing approaches which are overwhelmingly based on Graph Convolutional Networks). Specifically, in their paper the authors augment the vanilla transformers by incorporating sub-networks for richer modeling particle interactions (i.e., as opposed to particle interactions being modeled as dot products and being reduced to a single number, sub-networks allow richer modeling of interactions), as well as material specific properties. Authors evaluate the proposed SiT model on diverse environments and compare against several state of the art fluid simulation models and also demonstrate generalization across different materials.
main_review: **Strengths:**
1. Paper is well written, well organized, and narrative is coherent.

2. Experimentation for comparisons with previous state of the art models are convincing as authors compare with 4 recent related approaches and demonstrate that SiT improves performance over other models for fluid simulation in different contexts using standard datasets. 

3. Specifically, an interesting contribution of the paper is the use of explicit (disentangled) "abstract tokens" that encode material specific properties. The authors demonstrate (through multi-material simulations like a solid cube floating in a fluid) that the SiT method is able to leverage the material specific properties learned in the "abstract tokens" to better simulate the properties of the solid interacting with the fluid. For example: other methods (without abstract tokens) which don't explicitly differentiate between the solid and the fluid simulate the solid disintegrating and merging with the flowing fluid but SiT maintains the solid shape and is able to simulate the motion of the solid over the moving fluid.

4. Demonstrated SiT models are much more light-weight (i.e., far fewer parameters). Training methodology further explicitly prioritizes learning properties for different material types by adopting a weighted MSE loss.


**Weaknesses:**

1. Some of the initial claims made by the authors of disadvantages of GCNs (as they presuppose particle interaction neighborhoods where interaction occurrs with all particles in neighborhood) seem to be part of the proposed method as well? For example, even in the SiT model, the particle token interactions are defined by a window function governed by a radius `R`, isn't the particle token interacting with all other particles within radius `R`? 

2. Incomplete ablation analysis, authors do not demonstrate the importance of abstract tokens.

3. In the "few shot generalization" case, it is unclear what the goal of the experiment is? Also, in in Fig. 5b what does "robust" mean? i.e., What does it mean to claim that the SiT model (i.e., without abstract tokens) is "robust even at 60% training data" ? Secondly, the claim of "few-shot" learning while the model uses 60% of training data is questionable. 


**Questions:**

1. How is the parameter governing the extent of the particle window to inform particle connections (`R`) selected? 

2. How sensitive is SiT to the selection of `R` ?

3. Also, in the case of pair-wise interaction components i.e., particle interaction tokens, how is setting allowable connections (using Eq 5) any different from setting connections in the GCN works (that authors describe as inferior) earlier in their paper?

4. Authors claim that GCN based mechanisms are unable to provide selection mechanisms? Would employing attention in the GCN context not serve a a selection mechanism? 

5. All abstract tokens of a particular material interact with each other? How scalable is this operation for large fluid simulations?
summary_of_the_review: The paper makes a contribution of being the first work to employ transformer to model particle based simulations where the model explicitly accounts for particle, particle interaction as well as material properties during the simulation. The authors have performed a rigorous comparison with state of the art models and the achieved improvements and the explanations rendered for the improvements are sensible. Also, authors demonstrate also through qualitative and quantitative comparisons, the generalizability of the SiT model in various multi-material settings which are also convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proses the Simulation Transformer (SiT) to simulate particle dynamics, using the Tranformers' attention mechanism to attend to critical particle interactions. They demonstrate qualitative and quantitative improvements over prior work with various architectures in multiple environments. 
main_review: SiT uses the Transformer architecture to model particle dynamics. In contrast to a vanilla Transformer, SiT incorporates interaction tokens and abstract tokens to better model particle interactions and material properties, respectively. SiT is evaluated in four simulated environments: FluidFall, FluidShake, BoxBath, RiceGrip, though it would be nice to see environments with more non-liquid multi-object interactions. 
summary_of_the_review: SiT provides a novel Transformer-based architecture for predicting particle dynamics that outperforms previous graph-convolutional and Transformer-based architectures.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to learn particle dynamics of physical systems by the means of a transformer. In particular, the paper investigates using a vanilla transformer as well as a more customized variant called 'Simulation Transformer'. Both networks are validated based on variety of experiments of simulations of fluids, rigid and deformable objects, as well as through comparisons to existing approaches.  
main_review: - In the abstract the sentence 'most existing particle-based simulators adopt graph convolutional networks to model the underlying physics of particles' seems off. For most learning-based methods this might be true, but there of course exists a lot of simulators that are not learning-based. 
- In the abstract it is not clear what is meant by 'material semantics', 'particle tokens', 'interaction tokens'.
- S1: 'Different from traditional simulators ...'. Here it would help to provide a few examples and references. 
- S1: 'Consequently, they follow the same pipeline ...'. It is not clear which pipeline is meant by 'the same pipeline'. Does it mean the same pipeline is used to model different material properties? Please clarify. 
- S1: 'where they can robustly estimate the dynamics of a system with varying number and configuration of particles'. This is a strong claim, please provide a reference or tone-down this statement.
- S1 'rich semantics'. Please define what is meant by this. 
- S1 'deploys an abstract token'. Not clear what this means. Please clarify. 
- S1 The second to last paragraph about the contributions of the proposed paper remains too vague. The text refers to several concepts (e.g. like interaction tokens, trainable abstract tokens, intrinsic material characteristics, system-specific semantics, good balance between efficiency and complexity, particle interactions etc.) that have all not been introduced up to this point and it is not clear what the contributions of the proposed paper are compared to existing concepts. 
- S3.1 'The goal of a simulator is to learn a model ...' Does this refer to a learning-based simulator? In other literature simulators are commonly not described to 'learn a model'. It might be useful to more carefully define what is meant by a simulator in the context of this paper. 
- S3.2 'a selective mechanism is thus in great need to help the simulator'. This sentence is not clear. Is this supposed to mean 'a selective mechanism is needed to help the simulator'?
- 'is naturally a good backbone of building'. Why? It would help to add a few more details here. Are transformers 'good' because particles can be interpreted as discrete tokens? 
- S3.2 'considering all possible interactions is computationally redundant' This is not clear. Why would that be the case?
- S3.3 'rich semantics': up to this point the text has mentioned the term 'semantics' several times without clearly defining what is meant by it. 
- S3.3 'the semantics of a system is scattered in the tokens'. Without a clear definition of what is meant by 'semantics', sentences like these are difficult to comprehend. 
- S3.3 'To improve generalization ability of SiT and disentangle system-specific semantics from its system-agnostic counterparts'. This sentence is not clear. What is meant by 'disentangle system-specific semantics'? What is meant by 'system-agnostic counterparts'? Please provide more details. 
- S3.3 'Once abstract tokens capture system-agnostic semantics'? At this point 'system semantics' has not be defined, therefore this sentence is not clear. 

W.r.t the related work, I consider the following recent work on simulating rod-dynamics with GNNs missing: 

H. Shao, T. Kugelstadt, W. Pałubicki, J. Bender, S. Pirk, D. L. Michels, Accurately Solving Rod Dynamics with Graph Learning, Conference on Neural Information Processing Systems (NeurIPS), 2021
summary_of_the_review: 
Overall, I am very supportive of the topic this paper addresses and generally also of using transformers for solving physical systems. Using transformers for solving physics is an interesting research direction. However, in its current form I am not enthusiastic about this paper which is due to the following main reason: 

(1) While the paper is generally well-written, several sentences are left to vague and unclear. In the introduction several concepts and terms are used without carefully introducing them. This makes understanding the text difficult and blurs what the contribution is. Also, some sentences almost read like a collection of buzzwords which makes is very difficult to comprehend what the actual meaning is. Consequently, while the experiments and results are technically sound, I got the impression more care should be taken in carefully explaining and formalizing the concepts that are aimed to be discussed in the paper. 

(2) With 12 pages the Appendix provides a significant amount of additional material, some of which can be considered quite relevant for the main contribution of the paper (e.g. Table 4). This indicates that the contribution of this paper may better be discussed in a format that has less restrictive page constraints. There are reasons why papers have page limits and I have the impression that for this work the authors are trying to circumvent them by providing a lot of material in the Appendix. 

(3) The paper does not discuss any limitations of the introduced method. 

Therefore, I don't think the paper is ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the use of transformers for particle-based physics simulation. The main contributions are as follows.

C1. the paper proposes a specific transformer-inspired form of message passing, which, unlike transformer, still does explicit message computation.

C2. the paper proposes a very specific way to encode material type information as abstract tokens.

C3. the paper compares to several non-transfomer based baselines, as well as a pure transformer.

C4. comparisons are performed on a subset of 4 environments including some generalization settings. As presented in the paper, the proposed model seems to perform better than the baselines in most settings.

main_review: Strengths:

S1: Testing the ability of transformers on physical simulation, as opposed to message passing, is indeed an interesting question that needed to be asked. I suspect many groups have probably tried this, and generally found worse performance (Consistent with the findings for the GraphTrans baseline in this paper), but it is good to actually show some published results.

S2: Given that a pure transformer does not seem to work very well, there is another question worth asking: Is there a hybrid between message passing and transformer that could get the best of both worlds?

While I am very supportive of studying those two questions, I am afraid the current experimental setup is not doing a great job to support the claims that are being made due to some weaknesses:

W1: The proposed model, (equation 10) contains alpha and beta terms, and actually the conclusion from the authors is that alpha=0 beta=1 is the best working configuration. Now, if the alpha term is ignored, all that is left is the beta term, and the beta term is mostly a weighted aggregation of equation (8), what the authors call "Interaction Tokens". Looking closely at this function, this is pretty much a high dimensional message function, as used by the GNN baselines (DPI-Net and GNS), the only difference is that in this case this message plays two roles, the role of a high dimensional message, and the role of an edge specific query, which is combined with a receiver node specific key to build an additional attention weight for the aggregation. First of all I would probably not call this a transformer, because the mechanism is neither cross attention, or even self attention, since each edge key is used for only one edge, and never shared for more than one key/query product. Also, the mechanism does not offer the main advantage of transformers which is not having to require to compute specialized edge level embeddings, which are expensive. Instead, it pretty much is a standard message passing model, with an additional scalar attention component.

W2: Now, even if we leave aside whether the model should be called a transformer or not, then the remaining question is: how important is the contribution of the softmaxed "\omega" attention weights in the right term of equation (10). This this is the only major difference between this SiT and the other message passing baselines (e.g. DPI-Net and GNS), the easiest way to study that difference in a clean way would have been to take one of the most competitive message passing baselines (e.g. GNS), and added the proposed attention mechanism (e.g. omega) to the very same implementation, while keeping the rest exactly the same. Still, I tried to gave it my best shot at using evidence presented in the paper to actually find an answer to that question by looking at the comparisons to the closest best performing message passing baseline, but several experimental choices made this very hard (See W3, W4, W5).

W3: It is hard to take much from the comparisons in the BoxBath environments. Specifically, because, as highlighted by comparing the MSE and WMSE losses, the main differences in performance are just depending on whether the model keeps the box together or not. However predictions for the box particles are made with completely different approaches. In DPI-Net, GraphTrans, SiT, and SiT+, a single solid translation and rotation prediction is made for the whole box (it is by design impossible for the box to break apart), while for CConv and GNS the model is asked to produce independent predictions for each particle in the box (so the box can actually break). This difference seems to be the main dominant factor in the comparison between those two groups of models, and specifically between GNS and SiT, however, the explanation of this difference in the text does not seem clear enough. Furthermore, comparison about how to make predictions for the box is completely orthogonal to the topic in the paper, and just makes the results harder to interpret. There does not seem to be a good reason not to use the same approach (either model the box as a whole, or make per particle predictions) for all baselines. Otherwise the results presented in Table 1, and Figure 3 don’t actually tell much about how the extra attention weight contributes to the predictions, and the usage of the WMSE metric seems to just highlight this unfair advantage that some models have over the others when modeling the box. This also seems to directly oppose the main claim at the end of the abstract “Without bells and whistles…achieves superior performance … than existing methods”, as this is precisely a “bell and whistle” that is only used for a subset of the models.


W4: Comparison in FluidFall is also hard to interpret. “though the MSE results are close to each other. SiT can better maintain the shape of drops while DPI-Net and GNS fail due to the wrong prediction of viscosity”. This statement seems to make a quite general claim of why the DPI-Net and GNS fail on this domain, but there is not actual evidence supporting this explanation. First in Figure 3 frames differences are not actually that obvious, but instead we can look at Supplementary Figure 8, where the problem is more apparent. Second, in the animations from the original DPI-Net paper (http://dpi.csail.mit.edu/, “Rollout from our model”), the model predictions for this same domain look qualitatively much better than the results shown on the same domain for DPI-Net. Similarly, the GNS paper also models goop like materials with viscosity (https://sites.google.com/corp/view/learning-to-simulate#h.p_JUaMhhhLYDkV), and this particular artifact that not seem to happen. So not sure if there is any difference in the implementations that could explain this. Third, from my experience, these sort of artifacts appear when (1) the training distribution does not have a lot of variance (like it seems to be the case for this dataset) and (2) a model that has a lot of capacity per message passing layer does not have enough depth to learn effective long range interactions/correlations, so it learns bad short range correlations that exploit the biases in the training dataset and make the model overfit. The reason this happens for DPI-Net and GNS, but not SiT, may be dependent on the choice of hyperparameters so it is unclear, if for a different choice of hyperparameters, results would have told a different story. Note, these biases / failure modes are also more typical of models that are not translation equivariant and try to exploit biases on the absolute positions, but not sure whether the implementations of the models are translation equivariant or not. 

W5: GNS is not implemented for FluidShake or Rice Grip. From FluidFall and BoxBath, GNS seems to be the most competitive model with SiT, so it is unclear why no comparison is provided for FluidShake and for RiceGrip. This would probably be ok if it was not because of W3, and W4, but it seems crucially important considering the potential issues with those comparisons.

W6: The role of the abstract tokens in the model is unclear. The results clearly show that SiT+ is better than SiT performance-wise, however, it is unclear why. The authors intent is that they “abstract tokens capture system-agnostic semantics, they can be reused by SiT when generalizing to systems that have same materials but vary in particle amount and configuration.”,  however I don’t think this hypothesis is tested explicitly. Abstract tokens seem to help on FluidFall (which has a single material), and the training distribution on box, batch, for which the whole training datasets consists of 4x4x4 boxes, which cannot be explained by that hypothesis. On RiceGrip, the difference between SiT and SiT+ is larger, however, actually in that domain SiT that worse than DPI-Net, which I find suspicious. On the other hand if I understand abstract tokens correctly, they actually connect to all nodes of the same type, in a bidirectional way, and if this is the case they actually are sort of a global node that allow communication between any two pairs of particles of the same material in two hops, regardless of how far they are. How do we know that the differences observed are not just due to this extra range of communication? To test this, an ablation where “dummy abstract nodes” without the material embedding but with the same connectivity are added should be included. Authors also say “baselines … may regard the latter as part of the intrinsic material semantics and fail to generalize to systems with the same materials but different particle amounts and configurations.”, but this is not backed by evidence, and in fact several of the baselines in the paper have already shown generalization to different shapes at test time.

W7: (Referring to the baselines) “First, it forces each particle to interact with all its nearby particles without providing a selection mechanism, which leads to computational redundancy and prevents the discovery of inherent patterns of particle interaction.” 
“They require the particles to interact with all nearby neighbors without selections, which will aggregate redundant and irrelevant neighbors”. This is slightly misleading. Currently, all methods in the paper are using a fixed radius of connectivity to select a number of candidate neighbors (if this claim is True, this should not be necessary for SiT). Second, the SiT model still has to consider all interactions within the radius of connectivity, and learn to discard some by downweighting them. This is something that the baselines could also learn to do, by just outputting messages that are 0 for unnecessary interactions, and a similar plot like Sup. Fig. 7 (b) could be made by showing the norm of the messages. It is true that the explicit scalar weight is a nicer way to disable full edges, rather than disabling all element wise message components, but for the other reasons indicated above, there is no evidence showing that SiT is actually empirically better than the baselines at doing this.


Other observations comments:

O1: I would recommend using the more general term GNN (Graph Neural Network) instead of GCN (Graph Convolutional Network) to refer to the corresponding baselines, as most readers will think of node update rules as done “Kipf et. al” model when reading the term GCN.

O2: The transformer description (eqs. 2, 3, 4), seems to be missing something, as none of these equations that are part of the iterative update of the nodes actually contain learnable weights. I think the computation of separate keys, queries, and values may be missing. Not sure if this is a bug in the math, or if it is actually implemented that way. Perhaps this is also missing in some of the equations (10, 11, 12, 13, 14). 

O3: More explanation is needed about specifics of the model, for example, does the encoder uses absolute positions, or does it (like several or the baselines) use relative positions in the encoder, to become translation equivariant? There is also some explanation in the supplementation about number of blocks, number of MLP layers etc, but this is ambiguous and the main hyperparameters (such as number of message passing steps), should be more prominent, and probably in the main text. Also some explanation about how the model and baselines were fine tunes should be included.

O4: Not sure if the definition of WMSE is correct (eq 16). My understanding is that the goal is is to increase the relative weight of particles with materials that appear less often when there is an imbalance of number of particles of different materials. Currently, if you replace equation (15), into equation (16), if is unclear what happens with the N in equation (15). If it gets replaced by an N_k, then it would cancel out with the N_k in equation (16), which means no rebalancing happens, since K is just a constant (number of material types). If it remains as N, then you get N_k/N and the weighting goes into the opposite direction, materials with less particles, get even less weight on the sum on average. It would be useful to just fully expand equation (15).

O5: Not sure if there is something wrong with the top row of figure 3, since the camera position seems to be slightly different for each frame.

O6: “Notice that CConv is not suitable for rigid box, we only report result on BoxBath for reference of simulation on fluid parts.” Why?

O7: In the related work it may be worth mentioning “CConv” before “GNS” as it was published earlier.

summary_of_the_review: The intent of the paper is good, and the results could bring interesting insights about whether attention inspired architectures could bring to the simulation domain. However the empirical evaluation makes it very hard to know which improvements are a result of the specific technical additions added to the model (extra scalar attention for the explicit messages and abstract tokens), and which improvements are just due to other small, but possibly very relevant, differences (including different input encoders/decoders, different featurizations, different outputs, etc.) between SiT and the baselines. My recommendation would be to perform the same investigations by adding the extra attention to the existing baselines one by one (or at least to the most promising ones) while keeping other parameters identical and test if the form or proposed attention helps universally. Similarly, the idea of adding extra abstract nodes for material types, and its impact on generalization, could be applied to existing baselines too and tested and studied orthogonally to the attention contribution. This would make the paper conclusions much more insightful and useful than in their current form.

EDIT POST REBUTTAL: I want to thank the authors for the additional experiments and extensive replies. While many interesting ideas are introduced in the paper, the way the comparisons with baselines is performed makes it very hard to really understand which aspects of the proposed architectures is making the model perform better than baselines on some settings. 

Since the authors are proposing a new neural network layer, it would be more useful if, rather than compare to different baselines each with their original hyper-parameters from their papers, etc, the authors took the approach of integrating the different parts of the newly proposed layer into existing architectures (possibly including non-simulation settings), and try to understand better that way how the new layer may help in a more apples-to-apples comparison.
review_title: null

Meta-review: The submission received split reviews: two reviewers recommended weak accepts, and the other two weak rejects.  The AC went through the reviews, responses, and discussions carefully.  The AC agrees that this paper is well-written and has demonstrated the possibility of using transformers for particle-based physical simulation.  The AC also believes that the authors have addressed the concerns of reviewer dYcg, despite that the reviewer didn't engage in the discussions.  The contributions are however not most exciting, and none of the reviewers would like to champion the submission.

Further, the AC agrees with the knowledgeable and responsible reviewer ZsKn that the presentation and experiments can be better positioned to highlight the key contribution. As reviewer ZsKn has summarized, it's recommended that "the authors took the approach of integrating the different parts of the newly proposed layer into existing architectures (possibly including non-simulation settings), and try to understand better that way how the new layer may help in a more apples-to-apples comparison."

The recommendation is reject, and the authors are encouraged to revise the paper for the next venue.
Decision: Reject