GRAM: Graph Regularizable Assessment Metric

Authors:   com
Venue: MICCAI_org_2024_Workshop_MSB
Published: 2024-06-29 15:28:00

Abstract: Here, we propose the Graph Regularizable Assessment Metric ($GRAM$), a customizable tool for evaluating the quality of generated brain graphs. Current geometric deep learning methods often lack robust quantification techniques for assessing the synthetic brain graphs integrity. $GRAM$ addresses this gap by proportionally combining a set of existing graph metrics to establish a linear correlation between distortions' levels and metric values of ground-truth graphs. To evaluate the performance of our model, we generated a synthetic dataset of structural brain connectomes which was derived from an existing dataset and used to simulate a set of predicted connectomes from a generative model with controlled levels of distortions. Our results show that $GRAM$ outperforms single metrics in quantifying the distortion between generated and original graphs. This approach is a significant step towards establishing a universal graph quality index for graph-based predictive studies.

Main paper: 
 introduction: 
 Brain connectomes are crucial for exploring the connectivity patterns underlying cognitive processes [21] . These connectomes provide a framework for predicting the progression of neurodegenerative diseases by integrating connectomic analyses with established neuroscientific knowledge [4] . For instance, [10, 13] introduced geometric deep learning approaches to forecast Alzheimer's disease progression using brain connectome data. Despite their potential, obtaining connectomic data poses significant challenges. One major impediment is the extensive processing required for neuroimages acquired through modalities such as Magnetic Resonance Imaging (MRI). This process is both time-consuming and computationally intensive. Another challenge is the limited availability of sufficient MRI data, which can hinder comprehensive analyses.One approach to address the challenge of limited MRI data involves the use of generative models to produce synthetic neuroimages. Generative Adversarial Networks (GANs) [7] have shown significant capability in generating realistic brain scans. For instance, [14] proposed a GAN model based on a fully convolutional network and an Auto-Context Model to enhance the realism and accuracy of synthetic images. Similarly, [19] developed a GAN model that produces high-quality, realistic images that simulate the ground-truth brain images-to improve the performance of diagnostic models in medical diagnostics. Despite the potential benefits, using GAN-generated MRI data to study brain connectomes introduces two key challenges. First, the generated MRI data must be indistinguishable from real data both quantitatively and qualitatively. Second, the synthetic data requires additional processing to extract connectivity matrices.To address the above-mentioned issues, [25] proposed predicting brain connectivity matrices using a graph GAN-based approach. The authors created representative templates from clustered brain graphs to train models that predict the evolution of connectivities for a given brain disease over time. Their novel few-shot learning framework uses minimal training data and employs clustering and Connectional Brain Templates (CBTs) to handle the diversity within brain connectomic data. This ensures robust model training despite limited data. However, unlike images, brain connectomes are virtually impossible to evaluate qualitatively. Instead, quantitative metrics (e.g., centrality measures) are, thus far, a single way to evaluate the quality of the generated graphs.In this paper, we highlight the limitations of existing metrics for graph quality assessment and propose a novel universal customizable metric to quantify the quality of generated graphs with an application to a simulated prediction of brain connectivities based on an existing dataset. In particular, we propose Graph Regularizable Assessment Metric (GRAM ), a customizable framework designed to learn to proportionally combine a set of existing graph metrics in order to evaluate the generated graph's quality. Drawing inspiration from the universal image quality index by [23] , GRAM could be considered a first step towards a more universal graph quality index. Our contributions are listed as follows:1. We propose a new general assumption for quantitatively interpreting the quality of a generated graph based on the linearity between the amount of distortion and the value of the reported metric. 2. We propose a novel general metric based on the weighted combination of existing metrics. 3. Our proposed metric is adjustable depending on the type of graph as well as the chosen metrics to report. 
methods: 
 In this section, we present in detail the proposed metric GRAM for quantifying the quality of generated graphs.Let G = (V, E) be a directed weighted graph, V denotes the vertices, and E denotes the weighted edges given by w : e → R. Let Ĝ be the simulation of the output of a given generative model F aiming to predict a target brain connectome G such that Ĝ ≈ G. The goal of the simulation is to bypass the problem of finding the optimal F to train as well as to control the amount of distortion d between Ĝ and G, where d ∈]0, 1] with s defined as distortion step.The distortion level between Ĝ and G is measured by the number of edges |E| with differing weights. Specifically, for an edge e ∈ E with weight w in G We train GRAM using an MLP to optimize the metrics' weights αj forming the vector B(q) by minimizing the loss between the predicted surface created by the MLP output A × B and the reference surface created by the vector C(p) across n surfaces.and ŵ in Ĝ, the distortion is defined as the proportion of edges for which w ̸ = ŵ, regardless of the magnitude of the difference |w -ŵ|.As shown in Fig. 1 , we define a set of m distorted graphs, each characterized by a distortion level d denoted as Ĝd = { Ĝ1 , . . . , Ĝm }. The process of generating a suite of distorted graphs across all distortion levels is detailed in Algorithm 1, resulting in the set Ĝ = { Ĝs , . . . , Ĝ1 }. Initially, we set a predefined distortion step (e,g., 0.1). For each increment of the distortion level d, we randomly select | Ê| edges, where| Ê| = d × (|E| -|V |).Here, |E| is the total number of edges in the graph, and |V | is the total number of vertices. The term |E| -|V | represents the total number of non-diagonal edges, as diagonal edges (self-loops) are excluded. Therefore, by subtracting |V | from |E|, we ensure that we only consider nondiagonal edges. At each selected edge e, we replace its weight with a randomly generated value within the range [w min , w max ], where w min and w max are the minimum and maximum weight values in the graph G, respectively. We repeat this process m times to ensure that all the edges are distorted at least once. Initialize Ĝd ← ∅ 6:| Ê| ← d × (|E| -|V |) 7:for i from 1 to m do 8:Ĝ ← G 9:Select | Ê| random edges from E 10:for each selected edge e do 11:ŵ(e) ← random(wmin, wmax) 12:Update edge weight in Ĝ to ŵ(e) 13:end for 14:Add Ĝ to Ĝd 15:end for 16:Add Ĝd to Ĝ 17: end for return ĜAssumption 1: Let G = (V, E) represent a brain graph, where V denotes vertices and E denotes edges, with w : e → R representing the weights of the edges in E. For a metric M that assesses graph quality, we postulate that the variation in y such that:EQUATIONis linearly correlated with the distortion d applied to G. Where Ĝ is the distorted graph, and ρ is the correlation function. We express M as follows:EQUATION)Where d is the distortion level expressed as a ratio (e.g., d = 0.1 for 10% distortion), and k is a constant scaling factor.We introduce the GRAM : an adjustable and learnable measure for evaluating the quality of generated graphs. Unlike existing metrics, such as centrality measurements that separately assess different graph aspects, GRAM provides a linear approximation of the relationship between distortion evolution and its output value taking into consideration multiple aspects of the graph. For instance, a GRAM value of 0.8 indicates that the graph is 80% similar to the original data. The result of our metric is represented by the y value, which indicates the degree of similarity between the generated and original graphs.To do so, as seen in Fig. 1 for each graph, we define a matrix A ∈ R p×q , such that p is the number of distortion levels, q is the number of existing metrics.Within A, each element A i,j represents the correlation between a given metric's output ξ(G) and ξ( Ĝ) applied to G and Ĝ, respectively, at a particular distortion level d. Here i indexes a distinct distortion level, and j refers to a particular metric correlation (e.g., At a distortion increment s = 0.1, A 1,3 corresponds to the third metric correlation at a 10% distortion level). We define C as the reference output of the metric ensuring adherence to Assumption 1. GRAM aims to find the values α j forming a vector B such that:A × B = C.We define GRAM (G) as a weighted sum of the metrics' correlations between the ground truth and distorted graphs. Specifically, let ξ j (G) and ξ j ( Ĝ) denote the j-th metric evaluated on G and the distorted graph Ĝ, respectively. Then:EQUATIONWhere ρ is the correlation function and α j are the learnable weights for each metric's correlation.To solve the vector B, we leverage the universal approximation theorem demonstrated by [8] , which establishes that a feedforward neural network featuring a single hidden layer can approximate any continuous function with sufficient neurons and appropriate parameters (weights and biases). Consequently, our approach involves training a Multi-Layer Perceptron (MLP) to determine the parameters within B, taking A as input and C as output.To train the MLP, we minimize the loss between two surfaces: S r , formed by the intersection of vector C with the X and Y axes, and S p , defined by the curve of predicted weights in B, where the vector A × B intersects the X and Y axes, Fig. 1 (3). The goal is to optimize the predicted weights in B so that the vector A × B closely approximates vector C, aligning surfaces S r and S p .To do so, we use least squares regression [11] to fit a polynomial functionP (x) = a n x n + a n-1 x n-1 + • • • + a 1 x + a 0to the A × B output data. This involves finding a polynomial that minimizes the sum of the squared differences between the MLP output data points and the polynomial's predicted values. This method creates a continuous curve that closely follows the data pattern formed by A × B, which we then use to approximate the integral within a specified range. In our study, the loss is minimized by the following process: first, the total surface created by the MLP values is divided into n distinct parts. Each part is optimized independently to simplify parameter convergence. Finally, we average all the results from the optimizations.Our proposed surface loss can be defined as follows:Surface Loss = 1 n n i=1 xmax xmin |f C (x) -f MLP (x)| dx (4)where n represents the number of distinct surface parts, x min and x max denote the minimum and maximum values of the input range, respectively. The function f C (x) denotes the line defined by C values, while f MLP (x) corresponds to the polynomial approximation function of the MLP output.Training details. We train our model for 250 epochs, using Google Colab. For optimization, we use Adam optimize [9] , with learning rate of 0.01. 
results and discussion: 
 In this section, we evaluate 10 selected individual graph metrics as well as our proposed GRAM . Additionally, we discuss each of the findings.We used a dataset from [20] that contains 88 subjects (48 females, 40 males aged between 18 and 48 years). All subjects are right-handed and healthy. The dataset contains the structural connectomes where each connectome contains 90 brain regions of intrest from the Automated Anatomical Labeling Atlas (AALA) [22] .For this study, we select a step of s = 0.1 and a set of ten widely utilized graph metrics in the literature [16] . These metrics are: Betweenness Centrality [6] , Closeness Centrality [17] , Weighted Degree Centrality [1] , Eigen Centrality [2] , Pagerank Centrality [15] , Katz Centrality [3] , Hub-Authority [5] , Harmony [12] , Average Neighbor Degree [24] , Diversity Index [18] . As a baseline for evaluating our proposed GRAM , Fig. 2 , displays the correlation between the ground truth graphs and the generated ones, for each individual metric across various levels of distortion. Fig. 2 shows the evolution of the independent metrics differs across the studied correlation coefficients. These evolutions are non-linear and could be visually categorized into two distinct patterns. The first pattern includes metrics such as Betweenness, Closeness, Harmony, and Weighted Degree Centralities which exhibit a moderate progression for values of d < 0.8 followed by a rapid decline towards a correlation values of 0. Contrarily, the second pattern shows metrics such as Eigenvector, PageRank, Katz, and Diversity Index. These metrics show a non-linear evolution, characterized by a rapid correlation decline for distortion levels lower than d = 0.3. This decline is then followed by a gradual stabilization of the correlations for levels where d > 0.3. This observation highlights the insufficiency of relying on a single set of metrics to comprehensively assess generated graph quality. All metrics exhibit non-linear correlations compared to the reference line, thus rendering them unreliable due to their under-estimation or over-estimation of distortion levels. We generated distorted graphs using a step value of s = 0.1 and trained GRAM using the ten previously listed metrics. The training process optimizes two separate surfaces. The loss for the first surface is calculated over the range [0.1, 0.5], while the loss for the second surface is calculated over the range [0.5, 1]. Fig. 3 shows GRAM testing results based on the previously mentioned correlations (i.e., Pearson correlation coefficient, etc ...). Visually, the correlational outputs of GRAM seem to closely approximate the target triangular shape created by the reference line and its intersection with x and y axis.Table 1 shows the weight of each metric as produced by GRAM across various correlation coefficients. The average neighbor degree consistently exhibits high weight values across Pearson, Spearman, and Kendall's Tau correlation coefficients. Yet, almost all the other metrics' weights are close to (0 ± 0.1). This disparity in the correlation's values may be due to the significant overlap in the information captured by the average neighbor degree and closeness centrality with other metrics like betweenness or diversity index. The computation redundancy in some metrics could lead to one of these metrics to be over-represented compared to similar metrics. Limitations and Future Directions. This study marks an initial effort to establish a universal metric for assessing the quality of generated brain graphs. One notable limitation is the selection of ten metrics that share similar calculation methods. Another limitation is the evaluation based on a single dataset. Future research should explore a broader range of metrics and evaluate the model across various graph datasets for different applications.Code Availability. All codes used for this study are available in: https: //shorturl.at/xjSPW 
conclusion: 
 This paper introduced the Graph Regularizable Assessment Metric (GRAM ) to evaluate the quality of generated brain graphs that could be used as a universal method in reporting the quality of generated graphs in future predictive studies. It combines multiple metrics in a weighted framework, addressing the limitations of existing graph quality metrics. Our proposed method establishes a general assumption for graph quality based on the linearity between distortion and metric values where we used a multi-layer perceptron to optimize metric weights. We test GRAM using a set of simulated structural connectome data on which it demonstrated reasonable reliability in quantifying graph quality. In future work, we aim to extend GRAM to diverse graph types and datasets. 
 
 Appendix:  


Decision: The paper introduces GRAM, a novel and innovative method for evaluating the quality of synthetically generated brain graphs by aggregating multiple existing graph metrics. This approach is both customizable and adaptable, addressing a significant gap in the current literature by offering a unified metric for graph quality assessment. While the reviewers pointed out potential weaknesses, they acknowledged the paper's clarity, novelty, and potential for significant impact in the field. The authors have thoroughly addressed all feedback, providing detailed responses and committing to revise the work when appropriate.

Reviews: review: main_review: main_review: The metric aim is to evaluate the quality of generated brain graphs. Since the method seems to be evaluated on simulated graphs with know distortions, its application to a real scenario might need to be discussed.
strength_weakness: strength_weakness: The metric seems to be formulated considering the distortion of the generated graph. The proposed approach generates the distortion based on a known graph to simulate the generation process. However, some questions can be raised regarding its application to real scenarios with a generated graph and when the distortion is potentially unknown.

Similarly, some clarification can be appreciated in the Methods section, for example, in the definitions of the variables B and C and the training process of the metric.
review_summary: review_summary: The paper proposes a metric aggregation method to define a unified metric for quality evaluation of generated brain graphs.
main_review: main_review: The paper is written clearly to understand the method. I like the idea of combining different graph metrics to create a universal one. The method flows well and easy to understand. The authors have clearly outlined the limitations and future work. They have also shared the code to reproduce the  results reported on their choice of similarity metrics.
strength_weakness: strength_weakness: The main weakness of the paper is that it does not justify some of the assumptions made in the paper about for example, why is the distortion proportional to just number of edges with w != w^ and not the magnitude of the difference. The difference in the magnitude of edge weight is not amounting to the distortion. This makes me wonder of the metric is practical enough to be called a universal assessment metric. Secondly, the assumption of linearity of distortion with respect to the reported metric is not justified. Since the authors generated their own set of distortion graphs, I am curious about how this extends to GAN generated data. Therefore this paper is a preliminary work. The paper requires further refinement to establish the metric as practical. One comment of writing is that in the introduction section of the paper, the authors just reviewed different GAN methods and not different assessment metrics. If the main contribution is about a novel metric then in the introduction section, there should be a literature review about existing metrics.
review_summary: review_summary: The authors propose a generalized metric to quantify the distortion between two connectivity matrix. The proposed metric is basically a weighted sum of existing similarity/distance graph metrics. They synthesized their own set of distorted graphs and trained MLP to find out the weights.
main_review: main_review: Overall, this was a really good paper with clear novel contributions and strong performance. There are some limitations of concern regarding the constraints and potential scalability/generalizability based on the training dataset used & the single graph metrics.
strength_weakness: strength_weakness: 1. The study uses a small dataset of 88 subjects from a specific age range (18-48 years) and health status (all healthy). This limits the generalizability of the results. It's unclear how GRAM could scale to larger populations or how it would perform on datasets with different characteristics or pathological conditions.

2 The authors acknowledge that the ten chosen graph metrics share similar calculation methods. It would be interesting to evaluate GRAM on metrics that capture global network properties or are more specific to neuroscience (e.g., rich club coefficient). For example, see "Rich-club in the brain’s macrostructure: Insights from graph theoretical analysis" by Kim and Min (2020) and "Trajectory of rich club properties in structural brain networks" by Riedel et al. (2022). 

3. While the paper compares GRAM to individual metrics, it doesn't compare it to other existing methods for assessing graph quality or similarity, such as graph edit distance or graph kernel methods. This makes it difficult to assess GRAM's advantages over current approaches fully.

4. The evaluation relies entirely on simulated distortions of real brain graphs. While this provides controlled conditions for testing, it may not accurately represent the types of differences that would occur in graphs generated by actual predictive models. Testing on graphs produced by real generative models would strengthen the validation.

5. The paper doesn't demonstrate how improvements in graph quality assessment translate to clinical applications. Discussing or demonstrating how GRAM could impact diagnosis or prognosis in neurological disorders would strengthen the paper's relevance.

6. The paper doesn't discuss/explore how GRAM's performance changes with different combinations of metrics or how it performs on different types of graphs (e.g., functional vs. structural connectomes).  

7. The paper doesn't discuss GRAM's computational cost compared to individual metrics. If GRAM is significantly more computationally expensive, its practical applicability could be limited.

9. Using an MLP to optimize metric weights introduces the possibility of overfitting the specific dataset used (especially since it's so small and constrained). The paper doesn't address how this could be mitigated or how well GRAM would generalize to unseen data.

10. There were a few typos in the paper, e.g. "genrated" in Fig 1 and "refrence" in Fig. 3 caption. The bottom pink banner for Fig 1 also has some distortion where text runs over weirdly. Would adjust the font size/words used.
review_summary: review_summary: The paper proposes GRAM, a Graph Regularizable Assessment Metric for evaluating the quality of synthetically generated brain graphs. The authors test GRAM's performance over 10 commonly used graph metrics and noted improved qualitative and quantitative meaningfulness.

Comments: title: Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
comment: We thank the reviewer for the significant effort they made to provide feedback for this manuscript. We addressed each feedback as follows:

Feedback 1: The metric may face challenges in real scenarios where the distortion of the generated graph is unknown.
Response 1:  We value the reviewer's feedback. To address this, users of GRAM would initially create simulated graphs with known distortion levels to fine-tune the weights of the chosen existing metrics that form GRAM. This approach ensures that the metric is approximative even when distortion levels are not explicitly known in practical applications.

Feedback 2: Definitions of variables B and C need to be clarified.
Response 2: We appreciate the reviewer's comment regarding variables B and C. According to assumption 1 in the manuscript: The vector C is a representation of linearly correlated values with distortion levels where for instance, graphs with 0.1 of distortion level their metric values are 0.9. We trained GRAM to approximate this vector C. The vector B is the output of the MLP, it contains the weight of each of the exisitng metric we chosed in this study ensuring the linearity condition stated in assumption 1. GRAM is trained to output a vector B such that AxB approximates C. We highlight the explanation in GRAM section:
"We define C as the reference output of the metric ensuring adherence to Assumption 1. GRAM aims to find the values αj forming a vector B such that: A × B = C."

Feedback 3: The training process of the metric requires more detailed explanation.
Response 3: We acknowledge the reviewer’s comment. Accordingly, we have updated the training explanation to include more details as follows:
"We train our model for 250 epochs, using Google Colab. For optimization, we use Adam optimize [11], with a learning rate of 0.01. We used an 80/20 split, resulting in a training set of 7,040 samples and a testing set of 1,760 samples."

Feedback 4: Is a reference graph required? It seems that the metric compares against a reference/ground truth graph. Considering that the generated graphs in an actual implementation might not be directly associated with ground truth, what can be used as a reference graph?
Response 4: We acknowledge the reviewer’s comment. The reference graphs are needed for computing our metric. Similar to any statistical measures such as MAE, MSE, etc.. GRAM requires a ground truth graph that it can use to measure the distortion level and therefore report the quality of the generated graph. 

Feedback 5: Ambiguity about whether the distortion level needs to be known to compute the metric.
Response 5: We appreciate the reviewer's feedback. In response, we confirm that initially, users of GRAM would generate simulated graphs to fine-tune the weights of the selected existing metrics that constitute GRAM. Therefore, it is essential to know the distortion levels during the training process of GRAM.

Feedback 6: The purpose and objective of using a multi-layered perceptron (MLP) in conjunction with polynomial fitting need to be explained.
Response 6: We value the reviewer's input on this matter. The MLP architecture is used to find the optimal weights for the existing metrics that constitute GRAM, based on the universal approximation theory, with polynomial fitting serving as the loss function to train the MLP.

Feedback 7: Explanation about the surfaces Sr and Sp is required.
Response 7: We appreciate the reviewer's feedback regarding this. Sr denotes the reference surface, defined as the intersection of vector C with the X and Y axes. On the other hand, Sp represents the predicted surface, which is determined by the intersection of vector AxB (where B represents the output of the MLP) with the X and Y axes. Furthermore, we have revised the manuscript accordingly: "C(x) denotes the line defined by C values, while fMLP(x) corresponds to the polynomial approximation function of B (the MLP output) multiplied by A."
"The goal is to optimize the predicted weights in B so that the vector A × B closely approximates vector C, aligning surfaces Sr and Sp."


Please find here our revised paper: https://drive.google.com/file/d/1vaGrsrw6g8Lscdnl5Ehbrwr6eI3y37e_/view?usp=sharing
We thank the reviewer for the significant effort they made to provide feedback for this manuscript. We addressed each feedback as follows:

Feedback 1: The paper does not justify some of the assumptions made, for example, why the distortion is proportional to the number of edges with w \neq w^ and not the magnitude of the difference. The difference in the magnitude of edge weight is not amounting to the distortion.
Response 1: We appreciate the reviewer's remark. We added this sentence in the methods part as follows: "The objective is to detect any distortion in the generated graph, treating any alteration in edge weights as significant."

Feedback 2: The assumption of linearity of distortion with respect to the reported metric is not justified.
Response 2: We acknowledge the reviewer for pointing this out and apologize for the lack of information. To address it, we mentioned updated the GRAM section: "We opt for a linear model due to its ease of interpretation and analytical benefits [8]. In graph distortion context, the linear relationship clarifies how changes in edge weights impact overall metrics, enhancing result communication.

Feedback 3: The author-generated distortion graphs need to extend to GAN-generated data.
Response 3: We thank the reviewer for this observation. In response, this study serves as a proof of concept, and in future studies, we will incorporate GAN-generated data to further validate our approach. We added that in the limitations sections:
"Future studies will also incorporate the use of GAN-generated data to further validate our approach"

Feedback 4: The paper is preliminary work and requires further refinement to establish the metric as practical.
Response 4: We thank the reviewer for highlighting this. We plan to incorporate more real-world data from diverse fields to validate GRAM's applicability.

Feedback 5:  In the introduction section, the authors reviewed different GAN methods but not different assessment metrics. There should be a literature review about existing metrics.
Response 5: We thank the reviewer for poiting this out. Indeed, we mentioned in the introduction some of the single metrics that are used to quantify a given aspect of the graph such as centrality and betweness. We apologize for the lack of clarity.The updated paragraph in the introduction is as follows: " However, unlike images, brain connectomes are virtually impossible to evaluate qualitatively. Instead, quantitative metrics (e.g., centrality measures [7], Aver- age Neighbor Degree [26] and Diversity Index [20]) are, thus far, a single way to evaluate the quality of the generated graphs."

Please find here our revised paper: https://drive.google.com/file/d/1vaGrsrw6g8Lscdnl5Ehbrwr6eI3y37e_/view?usp=sharing
We thank the reviewer for the significant effort they made to provide feedback for this manuscript. We addressed each feedback as follows:

Feedback 1: Small dataset limitation.
Response 1: We appreciate the reviewer's observation. In our study, we generated 100 variations for each graph across all levels, resulting in a total of 8,800 graphs. We apologize for any previous lack of clarity on this point and have updated the manuscript to explicitly state: "We emphasize that our simulations resulted in a total of 8,800 graphs."

Feedback 2: Evaluate GRAM on metrics that capture global network properties or are more specific to neuroscience (e.g., rich club coefficient).

Response 2: We appreciate this suggestion. Future work will involve incorporating different metrics that capture global networks to validate GRAM's applicability. We updated the discussion part as follows:
"Future research should explore a broader range of metrics (e.g., that capture global network properties) and evaluate the model across various graph datasets (e.g., functional vs structural connectomes) for different applications."

Feedback 3: The paper compares GRAM to individual metrics but not to other existing methods for assessing graph quality or similarity, such as graph edit distance or graph kernel methods.

Response 3: We acknowledge the reviewer for pointing this out. To address it, we plan to explore these comparisons in future research efforts. We mentioned that in the manuscript as follows:
"Additionally, future work should include a comparison between GRAM and other methods for assessing graph quality."

Feedback 4: The evaluation relies entirely on simulated distortions of real brain graphs.
Response 4: We value the reviewer's feedback about that. Currently, this evaluation serves as an initial proof of concept. Future studies will expand this by incorporating GAN-generated data to strengthen the validation of our approach.

Feedback 5: The paper doesn't demonstrate how improvements in graph quality assessment translate to clinical applications.

Response 5: We appreciate the reviewer's remark on that. We added this sentence in the conclusion part as follows: "This approach is a significant step towards establishing a universal graph quality index for graph-based predictive studies (e.g., predicting disease progression in Alzheimer's, analyzing brain network development in infants)."

Feedback 6: The paper doesn't explore how GRAM's performance changes with different combinations of metrics or types of graphs (e.g., functional vs. structural connectomes).
Response 6: We recognize the reviewer’s point on that. This exploration is planned for future research efforts. We updated the discussion as follows:
" Future research should explore a broader range of metrics (e.g., that capture global network properties) and evaluate the model across various graph datasets (e.g., functional vs structural connectomes) for different applications."

Feedback 7: The paper doesn't discuss GRAM's computational cost compared to individual metrics.
Response 7: We thank the reviewer's observation regarding that and we apologize for this lack of information. We have now included the following information in the GRAM section: "The training of GRAM on Google Colab took 1 hour and 14 minutes." 

Feedback 8: Using an MLP to optimize metric weights introduces the possibility of overfitting the specific dataset used.

Response 8: We appreciate the reviewer's concern regarding that. It's important to note that we addressed this by generating 100 variations for each graph across all distortion levels, resulting in a total of 8,800 graphs. We apologize for any confusion and have clearly cited this in the manuscript:
"We emphasize that our simulations resulted in a total of 8,800 graphs."

Feedback 9: There were a few typos in the paper, e.g. "genrated" in Fig. 1 and "refrence" in Fig. 3 caption. The bottom pink banner for Fig 1 also has some distortion where text runs over weirdly.

Response 9: We thank the reviewer for bringing this up, we appologize for any confusion. The mentioned typos have been corrected.

Please you find here the revised version of our gram: https://drive.google.com/file/d/1vaGrsrw6g8Lscdnl5Ehbrwr6eI3y37e_/view?usp=sharing