PhysGNN: A Physics--Driven Graph Neural Network Based Model for Predicting Soft Tissue Deformation in Image--Guided Neurosurgery
Authors: Yasmin Salehi
Dennis Giannacopoulos
Venue: NeurIPS 2022 Accept (2022)
Keywords: PhysGNN
Physics-Driven
Graph Neural Network
Tissue Deformation
Image-Guided Systems
GraphSAGE
GraphConv
Jumping Knowledge
Physical Simulation
Mechanical Simulation
Finite Element Method
Finite Element Analysis

Abstract: Correctly capturing intraoperative brain shift in image-guided neurosurgical procedures is a critical task for aligning preoperative data with intraoperative geometry for ensuring accurate surgical navigation. While the finite element method (FEM) is a proven technique to effectively approximate soft tissue deformation through biomechanical formulations, their degree of success boils down to a trade-off between accuracy and speed. To circumvent this problem, the most recent works in this domain have proposed leveraging data-driven models obtained by training various machine learning algorithms---e.g., random forests, artificial neural networks (ANNs)---with the results of finite element analysis (FEA) to speed up tissue deformation approximations by prediction. These methods, however, do not account for the structure of the finite element (FE) mesh during training that provides information on node connectivities as well as the distance between them, which can aid with approximating tissue deformation based on the proximity of force load points with the rest of the mesh nodes. Therefore, this work proposes a novel framework, PhysGNN, a data-driven model that approximates the solution of the FEM by leveraging graph neural networks (GNNs), which are capable of accounting for the mesh structural information and inductive learning over unstructured grids and complex topological structures. Empirically, we demonstrate that the proposed architecture, PhysGNN, promises accurate and fast soft tissue deformation approximations, and is competitive with the state-of-the-art (SOTA) algorithms while promising enhanced computational feasibility, therefore suitable for neurosurgical settings.

PhysGNN: A Physics-Driven Graph Neural Network Based Model for Predicting Soft Tissue Deformation in Image-Guided Neurosurgery
Yasmin Salehi, Dennis Giannacopoulos
Department of Electrical and Computer Engineering
McGill University
Montreal, QC, Canada
yasmin. salehi @mail mwgill.ccmmmoooo.cco... ca, dennis dennis. hiangsgggammcoooooseenggg......
Abstract
Correctly capturing intraoperative brain shift in image-guided neurosurgical proce- dures is critical task for aligning preoperative data with intraoperative geometry for ensuring accurate surgical navigation. While the finite element method (FEM) is a proven technique to effectively approximate soft tissue deformation through biomechanical formulations, their degree of success boils down trade-off between accuracy and speed. To circumvent this problem, the most recent works in this domain have proposed leveraging data-driven models obtained by train- ing various machine learning algorithms-e.g. random forests, artificial neural networks (ANNs) wiith the results of finite element analysis (FEA) speed up tissue deformation approximations by prediction. These methods, however, do not account for the structure of the finite element (FE) mesh during training that provides information on node connectivities as well as the distance between them, which can aid with approximating tissue deformation based on the proximity force load points with the rest of the mesh nodes. Therefore, this work proposes a novel framework, PhysGNN, data-driven model that approximates the solution of the FEM by leveraging graph neural networks (GNNs), which are capable of accounting for the mesh structural information and inductive learning over un- structured grids and complex topological structures. Empirically, we demonstrate that the proposed architecture, PhysGNN, promises accurate and fast soft tissue deformation approximations, and is competitive with the state-of-the- (SOTA) algorithms while promising enhanced computational feasibility, therefore suitable for neurosurgical settings.
Introduction
Exploiting high-quality preoperative images surgical planning common practice among neurosurgeons. With the recent advances in surgical tools, medical imagery techniques have especially made their way into operating rooms to enhance surgical visualization and navigation, giving rise to image-guided neurosurgical techniques, where correctly aligning patient's intraoperative anatomical geometry with preoperative images has a significant impact on the accuracy surgical navigation [Hagemann al. 1999 Warfield al. [2000 Ferrant et al.! 2002 Sun et al. 2004 Vigneron et al. 2004 Choi et al. 2004a
Miller et al. [007]] Wittek et al. 2009 Joldes et al. 2009 Vigneron et al.
2010 Wittek et al. 2010 ei al. 2014 Miller et al. [ This task, however, poses challenge.
During the neurosurgical procedure, the brain continuously undergoes a series anatomical changes
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

for variety of reasons such as gravity, craniotomy tissue retraction and resection, and anesthetics [Roberts et al. 1998|, making surgical navigation based on preoperative data alone error- prror-prone and unreliable [Wittek et al. 2010
To account for intraoperative brain shift registration technology, by which the coordinate system of preoperative data and surgical site are brought into spatial alignment, has emerged as an essential tool in image-guided surgical frameworks for maintaining surgical navigation accuracy and depth
perception [Crum el al. 2004, [Cleary and Peters 2010 Risholm et al. 2011]. In a registration framework, data alignment is achieved by leveraging interpolation and transformation functions, which manipulate preoperative data to realize tissue deformations and are regularized by constraint
(Ferrant et al.] (2002).
At large, registration approaches may be categorized into image-based and model-based methods, which differ in the type of prior information they use as a constraint for deforming preoperative data with respect to intraoperative geometry. To define, image- registration methods are a set of techniques which mainly use pixel/voxel information and solely rely on image processing techniques
to realize tissue deformation [Beauchemin and Barron, 1995 Hata et al., 2000 Ji et al. 2014
Iversen et al. 2018 Tu et al. 2019]. While they may be effective in some settings e. g., registering deformations of hard tissues such as bone-they are prone to failing in effectively capturing soft tissue deformation for various reasons, such as contrast variations between preoperative and intraoperative images resulted from use of different image acquisition techniques. disappearance of boundaries between tissues upon surgical cuts, and presence of contrast agents [Ferrant al., [002]. J But most importantly, image-basee based techniques neglect prior information about the biomechanical behaviour of soft tissue, which an essential knowledge for creating accurate volumetric mappings NNabav et al., 2001 Ferrant et al. 2002].
On the other hand, model-based methods are a set of registration techniques that treat images as a deformable volume-a notion first introduced by BBoit [1981] better allow for presenting elastic and plastic deformations. These approaches address intraoperative registration challenges by imposing additional constraints based on mathematical or physical characterization of soft tissue İZhang et al., [017] to compute soft tissue deformation, resulting in mathematical- and physics-based solution formulations. In particular, studying physics- methods has become an emerging area of research in recent years [Choi et al., 2004a| Joldes et al. 2009 Malone et al.. 2010
Wittek et al., 2010 Joldes et al. 2010 Basafa and Farahmand] [201]] Miller et al.) 2019] for their potential to accurately model the biomechanics of the brain and predicting soft tissue deformation. Indeed, recent advances in this domain have led to the innovation of advanced image-guided surgical systems that are equipped with augmented reality (AR), which is established by fusing virtual reality (VR) with high-quality preoperative images in the form of deformable volume (Cleary and Peters 2010]. The goal of integrating AR in this framework is to facilitate tracking predefined surgical access routes without having to look away from the screen, and provide surgeons with visual tool-tissue interaction [Touttii et al. 2017].
In the literature, the finite element method (FEM) is a powerful technique for generating 3-dimensional (3D) deformable volumes from high-quality preoperative images, which can simulate intraoperative brain deformations [Wittek et al., 2010]. However, their degree of success boils down to trade-off between accuracy and speed [Tonutti et al., 2017]. In most cases, benefiting from a finite element (FE) model, which accounts for all types of material and geometric non-linearities. is not always feasible for requiring significant computational resources [Wittek et al. 2010 Tonutti et al.. 2017]. To address this limitation, recent works have proposed deriving deformable models by training machine learning algorithms with the results of precomputed FEM simulations that are based on preoperative images [De et al., 2011 Tonutti et al. 2017 Lorente et al.) 2017 Pfeiffer et al.) 2019 Liu et al.) 2020]. However, while promising, they all lack a key aspect for effectively learning over mesh data: they do not account for graph structure e.g., node connectivities
is an important piece of information for learning nodal displacements within each FE resulted from prescribed loads. In fact, wanting to incorporate node connectivities by feeding the adjacency matrix as set of features to the aforementioned algorithms will result in an ill-posed problem as the number of features would then exceed the number of data points. Additionally, these methods are arguably computationally inefficient and hard to train for high quality weshes-whied contain many nodes-at as the number of
'eemporarlyy removing part of the hone from skull to access the brain tissue.
VVolumetric brain deformations induced by surgical operations Nabavi et al. 2001

---

parameters needed to train the models would be in order of O((V)) where [ is the number of mesh nodes. Although the later problem has been previously addressed by training only on a subset of the mesh data [Tonutti et al., 2017]. or capturing lower dimensional embedding of the deformation space through Principle Component Analysis (PCA) [Liu et al., 2020]. an important question is raised on whether we can enhance computational feasibility while also benefiting from the structural information of the FE mesh.
I which approximates soft tissue deformation under prescribed forces by leveraging graph neural networks (GNNs) (Scarselli et al., 2008]. Specifically, PhysGNN is a physics-driven framework for having been trained with the results of Finite Element Analysis (FEA) (Fht) is based on the physical properties of soft tissue such as Young's 's modulus and Poisson ratio and for approximating tissue deformation i.e. displacement of the FE mesh nodes- by summing up the forces applied to the prescribed load nodes and their neighbours using GraphSAGE (Hamilton et al. 2017] and GraphConv [Morris et al.) 2019] layers, analogous to the way displacement caused by applied force(s) is computed according to the Newton's first law of motion. To the best of our knowledge, we are the first ones to be leveraging GNNs for approximating soft tissue behavior upon prescribed loads, and propose combination of
Present Work In this work, we propose novel data-driven framework, named PhysGNN
GraphSAGE and GraphCony layers as well as Jumping Knowledge (JK) connections [Xu et al. 2018
for this purpose. Our motivation for using GNNs for this task is their ability to enable generalizatee cinttoliiation of convolution on graph data through the notion of neural message passing [Gilmer et al. 2017].
by which the information of nodes within neighbourhood of arbitrary size transformed and aggregated producing a new value [Hamilton] 2020]. The message passing framework enables inductive learning over graphs through parameter sharing, a property that ensures learning constant number of parameters independent of the mesh size. Therefore, contrary to the previous methods, not only is PhysGNN capable of incorporating mesh structural information-ias nodes with each other and the Euclidean distance of neighbouring nodes from one another-which which can aid with effective prediction of tissue deformation, but also can enhance computational feasibility when learning from high quality meshes that consist of many nodes.
proximity of mesh
We evaluate the performance of various PhysGNN models- from different combination of GraphSAGE and GraphConv layers and aggregation functions- on two datasets distinguished by the number of force load points and the total force Empirically, we demonstrate that PhysGNN is capable of successfully predicting tissue deformation upon prescibed forces under 0.005 and 1.9 seconds using GPU and CPU, respectively. In fact, the absolute displacement error of the best-performing PhysGNN model for 94% to 97% of the nodes was found to be under mm-wwich is the precision in neurosurgery (Miller al. 2010
the results of FEA (our baseline).
2 Related Work
Approximating Tissue Deformation with Machine Learning The predictive power of various machine learning algorithms in approximating tissue deformation with respect to physics laws has been assessed in number of studies. These approaches either implement machine learning techniques an alternative to the FEM-e.g. representing mass points as cellular neural networks to assimilate elastic deformations [Zhong et al. 2006 Zhang et al., (019) or as a subsequent step to preoperative FEA in which data-driven models are resulted by training machine learning models with patient-generated content that are intrasubject as shown in Figure [ For examplc, while De al. [201T) proposed a physics-driver framework, named PhyNN consisting of radial basis function to reconstruct the deformation fields of liver and Penrose drain models, Lorente et al. [2017] investigated the success of linear regression, decision trees, random forests, and extremely randomized trees in simulating the biomechanical behavior of human liver under large deformations and during breathing in real time. Furthermore, [Lu et al. (2020 exploited ANNs to reconstruct tissue deformation of a head and neck tumour, where the dataset was encoded into low-dinensional representation using PCA prior to training to facilitate the learning process. Meanwhile, similar techniques have been employed to approximate tissue deformation in neurosurgical settings. In [Toutti et al. 2017], the authors explored and compared the effectiveness of ANNs and support vector regression (SVR) in predicting deformation of a brain tumour caused by applying forces the brain tissue. Generalizing the notion of deep learning based soft tissue
paatient-specffie to predict soft tissue deformation,
'The data and implementation are available
3

---

Finite Element
Volume Mesh
Data
Image-Guided Neurosurgical System
Medical
Finite Element Analysis
Imagery
Learning
Registration Technology
Live Imaging
Visualization
Module
Tmined Model
Tracking Devices
Software
Figure I: A generic image-guided neurosurgical system framework (Cleary and Peters 2010]. where patient's anatomy is aligned with preoperative images through deforming a deformable volume generated from preoperative imagery. which its biomechanical behavior under prescribed loads is predicted by machine learning methods.
simulation frameworks to different types of organs, Pfeiffer et al. [2019] proposed a convolutional neural network (CNN) based soft tissue simulation system to predict an organ 's internal deformation from known surface deformation. In their study, they accounted for all tissue types within an organ entirely synthetic data of random organ-like meshes to enhance the learning process by using more data than otherwise available. They stated their method can be adopted by the neurosurgical community capture brain shift, or be used in motion compensation radio-therapy.
and trained CNNs
Learning Mesh-Based Simulations Using GNNs Recently, Sanchez-Gonzalez et al. (2020) and Pfaff ct al. [2200] have proposed Graph Network-buss Simulators (GNS) and MeshGraphNets, respectively, to simulate interaction of fluids, rigid solids, and deformable materials with one another (Sanchez-Gonzalee et al., 2020], and complex physical systems such as acrodynamics, structural mechanics, and cloth |faff et al. 2020). Both GNS and MeshGraphNets generative architectures that use GraphNet Sanchez-Gonnzin et al. 2018 as their building block.
3 The Physics-Driven Graph Neural Network Based (PhysGNN) Model
PhysGNN is a data-driven model that leverages GNNs to learn the nonlinear behavior of soft tissue under prescribed loads from FE simulations. In this work, GNNs are deemed as the most suitable deep learning candidates for capturing tissue deformation for the reasons stated below.
First, GNNs enable incorporating mesh structural information by forming a computation graph- a neural network architecture-for for every node through unfolding the neighbour- hood around them via neural message passing, which refers to the exchange of feature and structural information between nodes that within the same k-hop neighbourhood [Hamilton] 2020). This characteristic of GNNs can ensure more accurate prediction of tissue deformation that is especially in close proximity of prescribed forces and susceptible to undergoing larger displacements, since the distance between prescribed force nodes and free-boundary condition nodes be accounted for.
2. Second, GNNs learn the same weight and bias parameters across the FE mesh through parameter sharing. In other words, the number of parameters which need to be learned are constant and independent of the mesh size, a property that also enables generalization of GNNs to unseen parts of the graph. This makes training PhysGNN with high quality meshes- meshes-which which inherently consist of many nodes computatioon less expensive than the methods presented in the previous studies.
3.1 Architecture
PhysGNN is a data-driven model that captures tissue deformation-ia. displacement of the FE mesh nodes-caused by prescribed forces microscopically by summing up taking the maximum of forces applied to their k-hop neighbours at each & for K iterations, where & E {1,..., K}. The idea of force summation or taking the maximum for when force is applied only one of the mesh nodes-is iss based on the way net displacement due to force is computed according to Newton's 's first law of motion, and inspired by the force propagation model presented (Choi al. 2004b). For this purpose, PhysGNN incorporates two types of GNNs, namely GraphSAGE Hamilton et al. 2017 and GraphConv- graph convolution operator k-GNNs in [Morris al. 2019 where the later GNN allows for incorporating nodal distances. The message passing architecture of GraphSAGE

---

JK
JK2
PROLUTONNIX)
PReLU(GNNNP))
PRoLU(GNNGY))
PRoI.LCGNNN(KKKO ((g)))
Brain Mesh Data X
GNN
PReL.
GNN
PReL.U
GNN
PRel.U
GNN
PRel.U
GNN
PRol.U
GNN
PRee.U
PRel.U
Linear
Output
PRoLU(GNNizg)
PReLL((GN(((za)
PRoLUULinaan((KK))g))) dropout 0.10 Liiear((g))
Figure 2: Architectural diagram of PhysGNN where the "GNN" layers may be GraphCony and/or
GraphSAGE layers that aggregate messages by summation and/or maximum functions. The best-
performing PhysGNN models in our study are shown in Figure 4a and 4b
and GraphCony for a graph G = (V,8), with V and being the vertex and edge sets, respectively, U, E V, and matrix E [gdl[V] representing the node features, are defined as:
(k) (u)
(( ((((-1)) Ve E (N(u))) ( h o W())) h (k-11) @ N(u) MNu) + b(b))
AGGREGATE
(1)
and
(R) m ((u) AGGREGATE (Feev h ( Vo N(u)))
D
(k) o (MERGE(() (W ((-11) (())) m mw(w)) + bba)))
N(u)
respectively, where. (u) the neighbourhood of node u, mx) represents the aggregated messages from the neighbours of node u at iteration k; where k E (1,..., K}, h, and h, are the embedding for nodes and v, respectively, Eun the edge weight between node and v, T is the concatenation operator, AGGREGATE is an arbitrary permutation-invaaian,, differentiable function, MERGE is either sum, column-wise vector concatenation or long short-term memory (LSTM) style update step, is a component-win non-lineat function (e.g., a sigmoid, or a ReLU), and the weights, ((A) (+) E RBd xx(-11 and the bias term, (A) R.444) are trainable parameters. Moreover, (())
(k)
the input to the GNN is simply the node features, expressed as h x., Yu E V. while the output of the final layer for each node is zu h Yu [Hamilton al. 2017
Morris al. 2019
Hamilton 2020
In this work, K is set to 6-i.e.. the mesh nodes capture information from their 6-hop neighborhood. This decision was made by noting performance improvement after increasing the number of GNN layers. Although deeper neural network architecture may have yielded better performance, increasing the number of GNN layers beyond 6 increased training time while may have also resulted in over-smoothing for aggregating over the entire FE mesh. Moreover, the implemented AGGREGATE functions are addition and maximum for summing up or taking the maximum of the applied forces, respectively, while the MERGE function used for the GraphConv layers is concatenation. PhysGNN also benefits from a variation of JK connections [u et al. 2018 to combat over-smoothing, which generates a weighted sum of embeddings at different iterations from bi-directional LSTM layer as:
where a are the attention scores. A full architectural diagram of PhysGNN is depicted in Figure 2
It should be noted that different PhysGNN models can be obtained by varying the combination of GraphConv and GraphSAGE layers and aggregation functions. The performance of various PhysGNN models on two different datasets is presented in the Results and Discussion section.
5

---

150
BC support preserihed
BC support
BC preserihed
100
140
140
N
100
100
50
100
150 100
50
Y
X 20
(a)
(b)
(c)
Figure 3: (a) Cut view of the FE volume mesh, where cyan represents the healthy brain tissue, while magenta represents the tumour tissue; prescribed and fixed boundary condition nodes, denoted as BC prescribed and BC support, respectively, of (b) Dataset and (c) Dataset
Table 1: Material properties of different parts of the brain [Joles et al.
2010
Wittek et al. 2010
Density (kg/m²)
Young's Modulus (Pa)
Poisson
Dataset
Ratio
Healthy Tissue
1000
3000
0.49
Tumour Tissue
1000
7500
0.49
4 Experimental Setting
4.1 Data Acquisition
Generating the Finite Element Mesh The FE volume mesh for rumning FE simulations was gener- ated by running TetGen [Si] 2015) on the surface mesh files provided by Tonutti al. [2017), which is based on MRI scans taken from the Repository of Molecular Brain Neoplasia Data |Madhavan jet al. 20099. Figure Sa is illustrative of the FE volume mesh, which consists of 91 nodes. More details on the FE volume mesh may be found in the supplementary material.
Material Properties Based on the work done in [Joldes al. 2010 and Wittek et al., 2010) the mechanical behavior of healthy brain and tumour tissues were characterized as hyper-elastee material obeying the Neo-Hookean constitutive law [Maas al. 2012). Additionally, similar to the experimental setting in [Tonutti et al. 2017], the tumour and healthy brain tissues have identical density and Poisson ratio, and the tumour is assumed to be stiffer than the healthy brain tissue, thus having a larger Young's 's modulus. The numerical value of material parameters are listed in Table
Finite Element Simulations To assess the effectiveness of PhysGNN in predicting brain shift imposed by forces during surgery and especially craniotomy- wwhih is modeled by applying forces the surface of the brain [Wittek et al., 2009 Miller al. 2019] two different datasets were generated by running FE simulations on the FE mesh in FEBio Maas et al.) 2012 using the GIBBON [Moerman] 2018 toolbox. The datasets are distinguished by the total force applied and the number of prescribed load nodes. Specifically, Dataset was created by applying 1.35 N to I node at a time in 30 time steps and directions for total of different nodes. This choice was motivated by the results of the experimental study done by Gan et al.) (205)) who measured 70% of induced forces during neurosurgery to be less than 0.3 N and a maximal peak 1.35 N during tissue dissection. On the other hand, Dataset 2 was created by applying 20 N to 100 nodes at time in 30 time steps and 165 directions to further capture the non-linear behavior soft tissue under large forces. Therefore, the amount of force applied to each prescribed node in Dataset at time i is:
F rotal
F
E f1....,30),
(4)
30

---

Table Characteristics of Datasets and 2. B.C. stands for Boundary Condition. Pres. stands for Prescribed.
No.
No.
No. Pres. Load Node(s) per
Max.
Fixed
Pres.
No. Force
Force Applied
No. Directions
No. Simulations
Dataset
B.C.
Load
Magnitudes
Nodes
Nodes
Simulation
(N)
1
502
11
30
1.35
15
4950
2
502
100
100
30
20
165
4950
Table 3: PhysGNN features and outputs.
Features (X)
Output (Z)
Fz. Fy. F., Fp. Fo, Fo, Physical Property
óx, dy, 8x
whereas the amount of force applied to one of the prescribed nodes in Dataset 2 at time i is:
total
F
30 x 100 xi i (1.....30).
(5)
The direction of forces applied to each prescribe load node features its surface normal direction, as well as directions that are randomly sampled from a hemisphere with radius 1. Lastly, to train split of 70:20:10 illustrative of the prescribed load nodes in Datasets and 2, respectively, which are located near the tumour site, well the fixed houndary condition nodes, which mimic the skull bone that prevents tissue from moving. Further details on the generated datasets can be found Table 2
PhysGNN,
applied to both datasets. Figures /b)r and 3c are
4.2 Features and Output
The features inputted into PhysGNN are the value of prescribed forces applied to the mesh nodes in Cartesian and spherical coordinates, denoted as (Fa, Fy, Fz) and (Fp, Fg, Fo), respectively, as well as constant value, named Physical Property, which for nodes with free boundary condition set to 0.4 the node belongs to tumour tissue and if the node belongs to healthy brain tissue, and () for nodes with fixed boundary condition. This value determines how much node can undergo displacement based on its boundary condition and Young's modulus given that brain tissue susceptible to undergoing larger displacements for its smaller Young's 's modulus compared to the tumour tissue. In particular, the value of 0.4 was calculated as the ratio of healthy tissue's 's Young's 's modulus (3000 Pa) over tumour tissue's Young's modulus (7500 Pa). On the other hand, the output of PhysGNN are the displacement value of the mesh nodes in the x, y. and directions, denoted as calculated the inverse
òx, dy. dz. Lastly, for the GraphConv models, the edge weights, Cu,es of the Euclidean distance between adjacent nodes u and v with V E V as:
Cu,v
(6)
ViZ 2,)² (Mh yo (Zu, =, )2) A summary of the features and output of PhysGNN can be found in Table
4.3 Baseline
Similar the previous studies Tonutti al. 2017 Lorente al. 2017 Liu et al. 2020), the predictions made by the proposed data-driven model, namely PhysGNN in this work, are compared against the results approximated by the FEM.
4.4
Loss Function, Optimization, and Regularization
The loss function used for learning the trainable parameters the mean Euclidean error computed as:
==
Σ
(yv.i 20.
V
PEV
i=

---

Table 4: Best performance PhysGNN on Datasets and
Mean
Absolute
Mean Euclidean
Euclidean
MAE (óx:
МАE
MAE
Absolute Position
Position
Error mm (%)
Dataset
(öy)
(ôz)
Error
Error
(mm)
(mm)
(mm)
Error
mm
(mm)
(mm)
(%)
Dataset
0.1145 + 0)28822
0.1051 +
0. =
0.2105 +
0.1690 + 0.4200
94.99
95.91
Validation
0.2544
0.2650
0.4530
Dataset 0.1117 =
0.0999 +
0.0999
0.2049 +
0.1612 +
95.11
96.18
Test
0.2750
0.2318
0.2601
0,4329
0.3961
Dataset
0.1478
0.1879
0.1283
0,3116 +
0.2106 +
94.25
96.26
Validation
0.3825
0,5527
0.2364
0,6958
0.6129
Dataset 0.1393
0,1842
0.1249
0,3023 +
0,2023 +
94.60
96.59
Test 0.3402
0.5479
0.2272
0.6671
0,5920
Table 5: The test set statistics of Datasets and 2, where is the displacement, and Max. Euclidean Errormean computed as the average of the maximum Fuclidean error observed for each data element i.e., each simulation.
Dataset 5ymax (mm)
8y mcan (mm)
Max. Euclidean Errormean (mm)
24.5864
11.6229 + 5.7681
2.5924 + 1.5665
2
47,8233
16.4170 # 10.4650
4.1952 + 2.5161
where [)] is the number of mesh nodes, y E TR is the displacement in the y, and directions approximated by the FEM, and I Vix3 is the displacement predicted by PhysGNN, To minimize the loss value, AdamW optimizer was used with an initial learning rate 0.005, which was reduced by factor of 0 to minimum value of le-8 if no improvement on the validation loss was noticed after 5 epoccs. To prevent over-fitting, early stopping was one of the regularization methods implemented by which learning stopped after no decrease in validation loss occurred after 15 epochs. Moreover, dropout rate of 0.1 was applied to the second last layer of PhysGNN to improve generalization to unseen data.
5 Results and Discussion
5.1 Performance of PhysGNN in Predicting Tissue Deformation
Table indicative of the performance the PhysGNN models that resulted in the lowest validation loss on Datasets 1 and 2. illustrated in Figures 4a and 4b) respectively. According to the results, PhysGNN capable of effectively predicting tissue deformation under prescribed loads, especially highlighted by 96% and 97% of absolute position errors being under mm in Datasets and 2, respectively. Notably, approximating tissue deformation using the FEM on a quad-core Intel @ 2.9 GHz CPU took 11.5099 + 6.8004 seconds on average for each FE simulation, with minimum and maximum computation times of 5.4191 and 32.3798 seconds, respectively, whereas prediction of tissue deformation per each FE simulation on dual-core Intel(R) Xeon(R) @ 2.20GHz CPU took 1.8294 + 0.0322 seconds on average, while on an NVIDIA P-100 GPU took 0.0043 0.0005 seconds on average. Based on these observations and the fact that the accuracy in neurosurgery is not better than 1 mm [iller et al. 2010]. PhysGNN manifests itself as a promising deep learning module for predicting tissue deformation in neurosurgical settings for its accurate approximations, computation speed, and case of implementation. Additional results on the median errors may be found in the supplementary material.
5.2 Comparison to Similar Studies
Comparing our results with the most recent works in predicting tissue deformation with machine learning methods, our approach compares favourably. In (Tonutti et al. 2017 the authors investigated

---

JKg
Jk
7x112
112x112
112
112x112
112
112x112
112
Graph 112x112
112
112x112
112
112x63
63
63x3
Mesh Date
Beain
GruphCont
agg agg-aad
PRoLU
GruphComn agg max
PRoLU
Gmphh agg max
PRRL.U
Graph agg max
PRcLU
PRel.U
GraphSAGE
aggs max
PReLU
Lincar
PRoLU
Lintar
Output
agg
7x112
112
112x112
112
112x112
112
112x112
112
112x112
112
112x112
112
112x63
63
63x3
GraphiCony
ayg uudd
PReLU
GraphCom Conv agg-aad
PReLU
GraphiCon agg max
PReLU
GraphS agg-
PReLU
GrapiS
PRel.U
GraphSAGG
agg maX
PReL.U
PReLU
Linear
Output
X
egg
Mean Euclidean Error (mm)
Mon Eucliden Enver
0.35
0.30
Danasct
Dinset 2
0.25
0.25
No. Layers with Add Agg.
Function
No. GraphCony Layers
(c)
(d)
Figure 4: Best-performing PhysGNN models (a) Dataset 1, and (b) Dataset 2. The effect of (c) consecutively replacing the maximum with the addition aggregation function in a PhysGNN model consisting of GraphConv layers followed by GraphSAGE layers, and (d) consecutively replacing GraphSAGE AGE with GraphCony layers while using the best- pestt-erforming aggregation function combination determined from Figure Agg. stands for aggregation.
the effectiveness of their approach in approximating brain tumour tissue deformation caused by applying forces less than 1 N to single node on the brain surface and reported a mean absolute position error of 0.191 + 0.201 mm. While absolute position error for Dataset 1 (where forces s 1.35 N are applied to a single node) seems to only be marginally smaller than theirs, we argue that result implicitly much better as our both the brain and tumour tissues. Similarly, our proposed method appears to be competitive with the results in [Lorente al. 2017 where they reported 100% of the Fuclidean errors be mm, for dataset with a maximum displacement of 15 mm. Lastly, [iu et al. [2020] reported an average Euclidean error of 0.129 mm and 0.392 mm, and an average maximum Euclidean error of 0.483 mm and 1.011 mm for two different test sets that exhibited maximum displacement of 30 mm on mesh with 9118
contains larger deformations since consider for
mesh with 1158 nodes. By referring to Tables and [ our results (achieved nodes) are shown to be competitive without requiring PCA to reduce the subspace dimensionality. Indeed, GNNs are inherently graph embedding models that encode graphs into a lower dimensional subspace to facilitate data manipulation for carrying out machine learning tasks on graphs. Therefore, we deem our method as not only effective for predicting tissue deformation but also simpler one to implement.
5.3 Ablation Study
To find the best-performing PhysGNN model on different datasets, we assessed the performance of various PhysGNN models resulted from different combinations of GNN layers and aggregation functions. First, to find best-performing aggregation function combination, we started with PhysGNN model consisting of three GraphConv layers followed by three GraphSAGE layers with all six layers using maximum aggregation function, which we consecutively replaced with the addition aggregation function. Afterwards, initialized a PhysGNN model with one

---

GraphConv layer followed by five GraphSAGE layers and replaced GraphSAGE with GraphConv layers consecutively while using the best-performing combination of aggregation functionn.
Initializing the PhysGNN model with three GraphCony layers followed by three GraphSAGE layers, consecutive replacement of maximum with addition aggregation function for two iterations reduced the mean validation Euclidean loss of Dataset 2 more noticeably, while such replacement for five iterations only marginally changed the performance of PhysGNN on the validation set of Dataset 1, as shown in Figure 4c Similarly, consecutive replacement of GraphSAGH with GraphConv layers while using the best-performing combination of aggregation functions resulted in a more pronounced decrease in the validation loss of Dataset 2 --where using three GraphCony layers yielded the best performance- while caused over-smoothing beyond a certain number for both datasets, as depicted
in Figure
These observations are justified by realizing that nodal displacements in Dataset 1 resulted from applying forces to only one node per simulation, contrary to nodal displacements in Dataset 2, which occur from multiple prescribed force nodes per simulation. Therefore, for Dataset 2, not only is using the summation aggregation function a more reasonable choice, but also weighting the impact of multiple force load points on their neighboring nodes by multiplying their message with the inverse of the nodal distance is indeed expected to produce more accurate results. The later observation sheds light on the importance of incorporating nodal distance and proper formulation of edge weights to effectively capture tissue deformation.
It is worthy note that the best-performing PhysGNN model Dataset 2, depicted in Figure 4b resulted in a validation and test mean Fuclidean error of 0.21 14 mm and 0.2058 mm, respectively, for Dataset 1, which is only marginally greater than the errors resulted from the best-performing model for Dataset 1, depicted in Figure 4a| Accordingly, the PhysGNN model depicted in Figure 4b may be regarded as robust for both datasets.
5.4 Limitations and Future Directions
One of the limitations of this study is that comparing our method against other works was done on empirical grounds due to either lack of accessibility to other studies' data and/or implementation, or not having sufficient computational power to use their data/models. However, our work presents itself as great benchmark for future studies as data and implementations publicly accessible and reproducing our results only requires a Google Colab Pro server.
Another limitation of this work is using the FEA results as baseline, which is dependent on the quality of MRI image segmentation, and the selected soft tissue constitutive laws that may not hold true for different patients (Tonutti et al. 2017 Moreover, tetrahedral elements are prone to undergoing volumetric locking for almost incompressible materials, such as soft tissue, and should not be used to simulate tissue deformation [Joldes et al. 2019). To alleviate these problems, Meshless Total Lagrangian Explicit Dynamics (MTLED) algorithms are emerging as a solution to better simulate soft tissue deformation, which also be trained with empirical results (Miller al., 2019 Joldes et al. 2019). As PhysGNN may also be trained with the results of Lagrangian meshless methods, the presented framework is foreseen to yield highly accurate and computationally efficient data-driven deformable volumes in the future.
6 Conclusion
The purpose of this work was to introduce novel data-driven framework, PhysGNN, combat the limitations of using the FEM for approximating tissue deformation-ane especially craniotomy- induced brain shift-in neurosurgical settings. Through our empirical results, we demonstrated the effectiveness of PhysGNN in accurately predicting tissue deformation. Compared to similar studies, our method is competitive with data-driven SOTA approaches for capturing tissue deformation while promising enhanced computational feasibility, especially for large and high-quality meshes.
Acknowledgments and Disclosure of Funding
The authors would like to thank the reviewers for their helpful and constructive feedback. This work was supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada.
10

---

References
Ehsan Basafa and Farzam Farahmand. Real-time simulation of the nonlinear visco-elastic defor- mations of soft tissues. International journal of computer assisted radiology and surgery, 6(3): 297-307, 2011.
Steven S. Beauchemin and John L. Barron. The computation of optical flow. ACM computing surveys (CSUR), 273))4333-66, 1995.
Chaim Broit. Optimal registration of deformed images. 1981.
Kup-Sze Choi, Hanqiu Sun, and Pheng-Ame Heng. An efficient and scalable deformable model for virtual reality-based medical applications. Artificial intelligence in medicine, 32(1):51-6)) 2004a.
Kup-Sze Choi, Hanqiu Sun, Pheng-Ann Heng, and Jun Zou. Deformable simulation using force propagation model with finite element optimization. Computers & Graphics, 28(4)5559556, 2004b.
Kevin Cleary and Terry M Peters. Image-guided interventions: technology review and clinical applications. Annual review of biomedical engineering, 12:119-142, 2010.
William Crum, Thomas Hartkens, and DLG Hill. Non-rigid image registration: theory and practice. The British journal of radiology, 77(suppl_ (actuupp_2):S14000 2):$140-1155, 2004.
Suvranu De, Dhannanjay Deo, Ganesh Sankaranarayanan, and Venkata Arikatla. A physics- driven neural networks-based simulation system (phynness) for multimodal interactive virtual environments involving nonlinear deformable objects. Presence: Teleoperators and Virtual Environmmnts, 20(4):289-300 2011.
Matthieu Ferrant, Arya Nabavi, Benort Macq, Peter M Black, Ferenc A Jolesz, Ron Kikinis, and Simon K. Warfield. Serial registration of intraoperative mr images of the brain. Medical image analysis, 6(4):337-359, 2002.
Liu Shi Gan, Kourosh Zareinia, Sanju Lama, Yaser Maddahi, Fang Wei Yang, and Garnette R Sutherland. Quantification of forces during neurosurgical procedure: A pilot study. World neurosurgery, 84(2):537-548, 2015.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 2263-1272 PMLR, 2017.
Alexander Hagemann, Karl Rohr, Siegfried Stiehl, Uwe Spetzger, and Joachim M Gilsbach. Biomechanical modeling of the human head for physically based, nonrigid image registration. IEEE transactions on medical imaging, 18(10)8875-884 1999.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pages 1024-1034, 2017.
William Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1-153 2020.
Nobuhiko Hata, Arya Nabavi, William M Wells III, Simon K Warfield, Ron Kikinis, Peter McL Black, and Ferene A Jolesz. Three-dimensional optical flow method for measurement of volumetric brain deformation from intraoperative mr images. Journal of Computer Assisted Tomography, 24(4): 531-538, 2000.
Daniel Høyer Iversen, Wolfgang Wein, Frank Lindseth, Geirmund Unsgård, and Ingerid Reinertsen. Automatic intraoperative correction of brain shift for accurate neuronavigation. World neurosurgery, 120:c1071-c1073 2018.
Songbai Ji, Xiaoyao Fan, David W Roberts, Alex Hartov, and Keith D Paulsen. Cortical surface shift estimation using stereovision and optical flow motion tracking via projection image registration. Medical image analysis, 18(7): 2014.

---

Grand Joldes, George Bourantas, Benjamin Zwick. Habib Chowdhury, Adam Wittek, Sudip Agrawal, Konstantinos Mountris, Damon Hyde, Simon K Warfield, and Karol Miller. Suite of meshless algorithms for accurate computation of soft tissue deformation for surgical simulation. Medical image analysis, 56: 152-171, 2019.
Grand Roman Joldes, Adam Wittek, and Karol Miller. Suite of finite element algorithms for accurate computation of soft tissue deformation for surgical simulation. Medical image analysis, 13(6): 912-919, 2009.
Grand Roman Joldes, Adam Wittek, and Karol Miller. Real-time nonlinear finite element computa- tions gpu-application to neurosurgical simulation. Computer methods in applied mechanics and engineering, 199(49-52)33305-3333) -3144, 2010.
Ping Li, Weiwei Wang, Zhijian Song, Yong An, and Chenxi Zhang. framework for correcting brain retraction based on extended finite element method using a laser range scanner. International journal of computer assisted radiology and surgery, 9(4):669-61,, 2014.
Haolin Liu, Ye Han, Daniel Emerson, Houriych Majditchran, Qi Wang, Yoed Rabin, and Levent Burak Kara. Real-time prediction of soft tissue deformations using data-driven nonlinear presurgical simulations. arXiv preprint arXiv:2010. 18833, 2020.
Delia Lorente. Francisco Marttnez-Martinez, María José Rupérez, MA Lago, Marcelino Martínez- Sober, Pablo Escandell- José María Martinez- Sandra Martínez-Sanchie Anto- nio J Serrano-L-ppez, C Monserrat, et al. A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Systems with Applications, 71:342-357, 2017.
Steve A Maas, Benjamin Ellis,, Gerard A Ateshian, and Jeffrey A Weiss. Febio: finite elements for biomechanics. Journal of biomechanical engineering, 134(1), 2012.
Subha Madhavan, Jean-Claude Zenklusen, Yuri Kotliarov, Himanso Sahni, Howard A Fine, and Kenneth Buetow. Rembrandt: helping personalized medicine become a reality through integrative translational research. Molecular cancer research, 7(2):157-167, 2009.
Hani R Malone, Omar N Syed, Michael S Downes, Anthony L D' D'Ambrosio, Donald O Quest, and Michael G Kaiser. Simulation in neurosurgery: a review of computer-based simulation environments and their surgical applications. Neurosurgery, 67(4): 2010.
Karol Miller, Grand Joldes, Dane Lance, and Adam Wittek. Total lagrangian explicit dynamics finite element algorithm for computing soft tissue deformation. Communications in numerical methods in engineering, 23(2):121-134, 2007.
Karol Miller, Adam Wittek, and Grand Joldes. Biomechanics of the brain for computer-integrated surgery. ACTA of Bioengineering and Biomechanics, 12(2):25-37, 2010.
Karol Miller, Grand R Joldes, George Bourantas, Simon K. Warfield, Damon E Hyde, Ron Kikinis, and Adam Wittek. Biomechanical modeling and computer simulation of the brain during neurosurgery. International journal for numerical methods in biomedical engineering, 35(10)ee250, 2019.
Kevin M Moerman. Gibbon: the geometry and image-based bioengineering add-on. Journal of Open Source Software, 3(22):506, 2018
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2019.
Arya Nabavi, Peter Mcl. Black, David T Gering, Carl-Fredrik Westin, Vivek Mehta, Richard S Pergolizzi Jr, Mathieu Ferrant, Simon K Warfield, Nobuhiko Hata, Richard B Schwartz, al. Serial intraoperative magnetic resonance imaging of brain shift. Neurosurgery, 48(4):787-798, 2001.
Tobias Pfaff, Meire Fortunato, Alvaro Gonzaez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference Learning Representations, 2020.
12

---

Micha Pfeiffer, Carina Riediger, Jürgen Weitz, and Stefanie Speidel. Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks. International journal of computer assisted radiology and surgery, 1477)1147-1155,, 2019.
Petter Risholm, Alexandra J Golby, and William Wells. Multimodal image registration for preopera- tive planning and image-guided neurosurgical procedures. Neurosurgery Clinics, 22(2):197-206,
2011.
David W Roberts, Alexander Hartov, Francis Kennedy, Michael Miga, and Keith D Paulsen. Intraoperative brain shift and deformation: a quantitative analysis of cortical displacement in 28 cases. Neurosurgery, 43(4):749-758, 1998.
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physics engines for inference and control. In International Conference on Machine Learning, pages 4470-4479. PMLR, 2018.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. International Conference on Machine Learning, pages 8459-8468. PMLR, 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-8,, 2008
Hang Si. Tetgen, delaunay-based quality tetrahedral mesh generator. ACM Transactions Mathematical Software (TOMS), -1(2):1-36, 2015.
Hai Sun, Francis E Kennedy, Erik J Carlson, Alex Hartov, David W Roherts, and Keith D) Paulsen. Modeling of brain tissue retraction using intraoperative data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 225-233. Springer, 2004.
Michele Tonutti, Gauthier Gras, and Guang-Zhong Yang. A machine learning approach for real-time modelling of tissue deformation in image- guided neurosurgery. Artificial intelligence in medicine, 80:39-47, 2017.
Zhigang Tu, Wei Xie, Dejun Zhang, Ronald Poppe, Remeo C Veltkamp. Baoxin Li, and Junsong Yuan. A survey of variational and enn-based optical flow techniques. Signal Processing: Image Communication, 72:9-24, 2019.
Lara M Vigneron, Jacques G Verly, and Simon K Warfield. Modelling surgical cuts, retractions, and resections via extended finite element method. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 311-318. Springer, 2004.
Lara M Vigneron, Romain C Boman, Jean- -han-Philippe Ponthot, Pierre A Robe, Simon K Warfield, and Jacques G Verly. Enhanced fem-based modeling of brain shift deformation in image-guided neurosurgery. Journal of Computational and Applied Mathematics, 234(7):0046-255, 2010.
Simon K Warfield, Matthieu Ferrant, Xavier Gallez, Arya Nabavi, Ferene A Jolesz, and Ron Kikinis. Real-time biomechanical simulation of volumetric brain deformation for image guided neuro- surgery. In SC'00: Proceedings of the 2000 ACM/IEEE Conference Supercomputing, pages 23-23. IEFF, 2000.
Adam Wittek, Trent Hawkins, and Karol Miller. On the unimportance of constitutive models in computing brain deformation for image-guided surgery. Biomechanics and modeling in mechanohi- ology, 8(1 87)),77884 2009.
Adam Wittek Grand Joldes, Mathieu Couton, Simon K Warfield, and Karol Miller. non-linear finite element modelling for predicting soft organ deformation in real-time; application to non-rigid neuroimage registration. Progress in biophysics and molecular biology, 103(2-3): 292-303, 2010.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pages 5453-5462. PMLR, 2018.
13

---

Jiaxuan You, Rex Ying. and Jure Leskovec. Position-awand graph neural networks. In International conference on machine learning, pages 7134-7143. PMLR, 2019.
Jinao Zhang, Yongmin Zhong, and Chengfan Gu. Deformable models for surgical simulation: survey. IEEE reviews in biomedical engineering, 11:143-164, 2017.
Jinao Zhang, Yongmin Zhong, and Chengfan Gu. Neural network modelling of soft tissue deformation for surgical simulation. Artificial intelligence in medicine, 97:61-70, 2019.
Yongmin Zhong. Bijan Shirinzadeh, Gürsel Alici, and Julian Smith. A cellular neural network methodology for deformable object simulation. IEEE Transactions on Information technology in Biomedicine, 10(4):749-762, 2006.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] The standard deviation of the errors are reported in Tables Ta S Regarding the error bars with respect to the random seed, the random seed was initialized
for all experiments allow for reproducible results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The code used towards generating the datasets was adapted from the examples provided in the Gibbon [Moerman] 2018 documentation. Additionally, one of the python functions used towards creating the PyTorch Geometric datasets was adapted from the code provided in (You et al. 2019), which is referred to in the ReadMe file of the supplementary material that has been published the GitHub repository The surface mesh models taken from (oonutii et al. 2017). which are based on MRI scans in MMadhavan et al.) 2009] are also cited in the paper.
(b) Did you mention the license of the assets? [Yes] The licences of the aforementioned code assets are included in the license document that is published in the GitHub repository.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] The code used towards generating the datasets, as well as all the PhysGNN models have been included the supplementary material, which are also available at the public GitHub repository.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] All the data used from other sources-i.e. the surface mesh of the brain model taken from Tonutti et al. [207]] that was generated from the MRI images taken from the Repository of Molecular Brain Neoplasia Data Madhavan et al.
22099 have been cited in the paper.
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "uziCewb0N3", "reviewer": "Reviewer_8xpx", "summary": "In this paper, the authors present a method for soft tissue deformation of the brain using GNNs. The GNN is trained with the results of a FEM simulation. Compared to other methods that were trained on FEM results, the authors use a GNN to account for the graph structure of the FEM. The results show that the presented method is on par with other methods for soft tissue deformation of the brain.", "questions": "1) Is there a specific reason to use two different aggregation functions (mean and max)?\n2) Are the node features that are used as input the same for all nodes?\n3) Why is the input provided in polar and Cartesian coordinates?\n4) Is the output of the last linear layer 63x3? \n", "limitations": "Yes, the authors have addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written and organized, especially the introduction and the prior work section is very detailed. The method part is missing some details and is difficult to understand. For example, the description (177 – 186) of Eq. 1 and 2. Having a detailed description of each element of Eq. 1 and 2 would improve the understanding. The evaluation could be more detailed as only the mean and std error is provided. A more detailed evaluation of the different regions of the brain would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c4MigVP8XKc", "reviewer": "Reviewer_BHqj", "summary": "The authors present a series of graph neural networks (GNNs) trained on finite element analysis (FEA) data to model deformations of brain tissue during surgery. The authors evaluate these networks on two separate datasets, and compare against baseline FEA results. The results show that the GNN framework is capable of milisecond inference on GPU hardware, whilst still retaining a clinically feasible level of accuracy.", "questions": "1. In section 5.1, you quote the time taken for the model to produce predictions on a GPU & CPU. Do you have the time taken to produce the  results using the original baseline data (i.e. the calculated FEA results)? Including this, as well as a quote of the speedup factor would be very useful as one of the main motivations of the model is the relative compute efficiency boost you would gain from this method.\n\n2. You mention in section 5.4 that you lacked \"accessibility to other studies’ data and/or implementation\". Did you attempt to contact the authors to get access to the aforementioned datasets & implementations to validate your method against theirs?\n\n3. You also mentioned in section 5.4 that in some cases you lacked \"sufficient computational power to use their data/models\". Could you elaborate on this please?", "limitations": "The authors appropriately address the limitations of the study, including the lack of validation datasets present within other studies, and the inherent limitation of using FEA as a baseline.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is clearly written and concise. The work presents a novel approach to modelling FEA data using a GNN whilst clearly outlining why a GNN would be a suitable architecture for this problem. The introduction and related information clearly outlines the prior work and motivations relevant to this topic. \n\n**Weaknesses**\n\nThe validation of the method is fairly limited, as is discussed in the limitations section 5.4. Without direct comparison to other methods data it is difficult to draw meaningful conclusions about the performance relative to similar methods.\n\nIn addition, the data used for validation is generated from finite element simulations, which is not as rigorous a source of comparison as compared to Lorente et al. (2017) which measure tissue deformations from ex-vivo data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YvU4OmKgKsO", "reviewer": "Reviewer_PQLo", "summary": "In this paper, the authors present the creation of a surrogate FEM model of the biomechanical behavior of the brain. The creation of this surrogated model is based on the use of Graph Neural Network (GNN) for its ability to allow introducing not only the positions of the nodes but also the connections between them (by sending messages). Compared to similar approaches, the model is competitive in terms of speed and accuracy. In addition, the authors state that their approach allows the generation of models surrogated to different approximations by means of finite elements. ", "questions": "*.- How could this approximation be applied in real scenarios?\n*.- Could it be applied to the simulation of anisotropic tissue behavior? \n*.- The images are a little bit poor. They should be improved.\n", "limitations": "Authors have adequately addressed the limitations of their approach. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n*.- This approximations allow to generate fast and accurate surrogate models to FEM based on GNN. \n*.- Due to the use of GNN, it can consider connexions between nodes. \n*.- It is general enough to be applied to other Finite Element approximations. \n\nWeaknesses:\n*.- In this approximation (as other ones) only forces are considerer when (f.e. in case of brain-shift) most of the time only the displacement of the nodes can be obtained. \n*.- It is only compared to synthetic models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept