Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation

Abstract: Graph Similarity Computation (GSC) is essential to wide-ranging graph applications such as retrieval, plagiarism/anomaly detection, etc. The exact computation of graph similarity, e.g., Graph Edit Distance (GED), is an NP-hard problem that cannot be exactly solved within an adequate time given large graphs. Thanks to the strong representation power of graph neural network (GNN), a variety of GNN-based inexact methods emerged. To capture the subtle difference across graphs, the key success is designing the dense interaction with features fusion at the early stage, which, however, is a trade-off between speed and accuracy. For slow learning of graph similarity, this paper proposes a novel early-fusion approach by designing a co-attention-based feature fusion network on multilevel GNN features. To further improve the speed without much accuracy drop, we introduce an efficient GSC solution by distilling the knowledge from the slow early-fusion model to the student one for fast inference. Such a student model also enables the offline collection of individual graph embeddings, speeding up the inference time in orders. To address the instability through knowledge transfer, we decompose the dynamic joint embedding into the static pseudo individual ones for precise teacher-student alignment. The experimental analysis on the real-world datasets demonstrates the superiority of our approach over the state-of-the-art methods on both accuracy and efficiency. Particularly, we speed up the prior art by more than 10x on the benchmark AIDS data.


Reviews and author responses:

 I have read the other reviews and author responses. In my own review, the authors clarified a couple of important points regarding student-teacher relation that was a concern for me. Though I think the paper still needs more explanation on this dimension, I find author responses satisfactory and I am leaning towards an acceptance.

The paper studies efficient graph similarity computation and proposes two connected techniques: a complex teacher model slowly learning through the graph and a simpler student model jointly learned and proposed to be used for faster inference. The proposed technique uses well-established Fusion network structures. The joint learning through knowledge distillation enables the simpler student model to learn more accurate latent representations through the teacher network. Both accuracy and run-time results have been provided using benchmark data sets.
  The paper studies an important graph similarity computation problem. It does not propose completely new techniques/methodologies but uses known ones in a reasonably creative way to attack an important problem. It provides a variety of results--mostly adequate with some caveats I explain below. I could easily follow the paper as a first-time reader, it is reasonably well written. 

In short, the paper proposes an interesting solution to an important problem. However, the results are not convincing enough to justify such a complex solution. It is possible that there is not adequate experimentation or explanation of the results. 

1) Sometimes the (significantly simpler) student model beats the teacher model in terms of accuracy (e.g., mse, p). This has not been studied well. In my opinion, this is a very important matter and at the heart of the paper. The reason for the existence of the teacher model is just to help the student model but if it is often not necessary in terms of accuracy, what is the point? How common is this?  Under what conditions/problems/data sets does this happen? Could one just learn the student model and have the fast inference technique.

2) The main motivation for the dual architecture is fast inference. The motivation is well explained (e.g., many online systems are ok for long offline training but require low latency in inference). However, I don’t think that the experiments (e.g., table 3) are fair enough. The difference between student-F and student-R is an implementation detail where embeddings are computed offline and stored (  a fairly important and practical technique commonly used). The shortcoming is that such similar optimizations have not been explored in other techniques. Is it possible at all for other techniques to do at least some partial offline computation and store results? Many of the baseline results are just copied from the relevant papers. 
 The main application of the paper tries to make similarity search, or complex graph search much more convenient. This has a bunch of positive implications from search engines to bioinformatics. However, the proposed techniques could be used in a vast variety of fast retrieval/search systems in which bad actors could do an adversarial search in the complex graphs. A short description of such social/practical implications can be explained further. What could negatively happen if we have very fast graph similarity computation? For example, could hackers make faster password searches?

 Thank you very much for the valuable comments. Your concerns and questions are addressed below:

**Q1**: How well would the proposed method work on larger graphs?

**AQ1**: That is an excellent question. As mentioned in the introduction of the paper, it is quite challenging to calculate the graph similarities in terms of the computation cost, especially for the exact solvers, which is an NP-hard problem. There is a dilemma that the ground truth GED between the large graphs is often inaccessible due to the intolerant computation cost. Manual labeling is also extremely hard since it is GED between large graphs which cannot be directly observed by human efforts. We have surveyed a lot on the large GED datasets and found most of them are around 10 nodes per graph. This is a promising direction to which our proposed method can be extended. We summarized the statistics of our used four datasets in Table 1 below, and IMDBMulti is significantly larger than others. The comparison on the inference time of four datasets is listed in Table 2 below. As shown in Table 2, our proposed model is at least 42x (i.e., 7.326e-5 / 1.753e-6) faster than SOTA on the IMDBMulti, which is the largest and most challenging dataset of the four. Such results can partially support the superiority of our model on large datasets. Here are the detailed statistics and inference time of the four datasets.

|  | Quantity of Graph Pairs | Avg Nodes per Graph | Avg Edges per Graph |
|:---:|:---:|:---:|:---:|
| AIDS700nef | 78,400 | 8.91 | 17.675 |
| LINUX | 160,000 | 7.59 | 13.92 |
| IMDBMulti |  **360,000** |  **13.66** |  **143.49** |
| ALKANE | 14,400 | 8.88 | 15.77 |

Table 1. Statistics of the testing sets of four benchmarks. We have calculated 1) the number of graph pairs in the test set, 2) the average quantity of nodes per graph, and 3) the average number of edges per graph of four used benchmarks.

|  | Hungarian-A* | GENN-A* | SimGNN | SimGNN-F | E-SimGNN | E-SimGNN-F | Teacher | Student-R | Student-F |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| AIDS700nef | 29.915 | 13.323 | 1.420e-04 | 3.305e-05 | 1.234e-04 | 4.418e-05 | 1.421e-04 | 1.295e-04 | **1.888e-06** |
| LINUX | 2.332 | 2.177 | 2.477e-04 | 2.458e-05 | 2.365e-04 | 3.811e-05 | 2.471e-04 | 2.369e-04 |  **2.144e-06** |
| IMDBMulti | - | - | 5.359e-04 | 7.326e-05 | 6.456e-04 | 1.475e-04 | 6.976e-04 | 7.474e-04 |  **1.753e-06** |
| ALKANE | - | - | 2.663e-04 | 5.653e-05 | 2.469e-04 | 7.160e-05 | 3.067e-04 | 2.657e-04 |  **5.347e-06** |

Table 2. Average inference time (sec) to solve GED problems on **one pair of graphs** of four different benchmarks. The meanings of SimGNN-F and E-SimGNN-F can be referred in the AQ2 to Reviewer 3zgJ. We have misunderstood the Table 2 of the original GENN-A* paper [1]. After I checked its official code [2], which was just released on Aug 4th, and contacted the author for double-checking afterward ,__the reported computation time of Table 2 in GENN-A* paper is on the one pair of graphs, whereas our reported time is on the whole testing set__. Therefore, we updated our Table 3 under the same setting as GENN-A* on a pair of graphs, and reported the time on four datasets. 

As seen in Table 2 above, it is obvious that the CO-based methods are significantly slower than the GNN-based approaches. As for the time comparison between the GNN-based and our distilled methods, the proposed Student-F is faster than the SOTA by at least 10x (i.e., 5.653e-05 / 5.347e-06). Especially on the largest dataset, i.e., IMDBMulti, we have shown 42x faster speed over the SOTA, which sufficiently indicates the superiority of efficiency on the larger datasets.

**Q2**: What is particularly challenging about the AIDS dataset such that the proposed methods do not perform as well as existing SoTA methods despite achieving faster computational efficiency?

**AQ2**: Thanks for your careful observation. The GENN-A* is a Combinatorial Optimization (CO) based method that requires the node-wise search over the tree-structure space. The GENN-A* applies a GNN to filter the irrelevant nodes on the tree whose searching space is greatly reduced to accelerate the searching process. AIDS dataset has less quantity of graphs compared with the LINUX. In GENN-A*, the GNN is applied to predict the node-wise score, requiring more fine-grained learning over the graphs. The GNN is trained by the loss average over a batch of samples, which is not totally reliable since it might be over-smoothing on the LINUX due to the large quantity of non-fine-grained data. Although the GENN-A* outperforms ours on some of the matrices on AIDS, its inference speed is slower than ours with 6 orders which is impossible to use in the fast inference case such as retrieval. In conclusion, the DB-based method has greatly sacrificed its speed for the gain in accuracy.  

**Q3**: At what point does the drop in accuracy become inappropriate? Are there other ways to evaluate performance and demonstrate that the accuracy tradeoff is not too detrimental (i.e. domain-specific metrics)?

**AQ3**: Thanks for this question. The trade-off between accuracy and speed is always one of the core topics in the graph and ML community. To the best of our knowledge, there is no matrix to evaluate both the speed and accuracy in the previous GSC papers. In my personal opinion, the product of average individual computation time and MSE seems a reasonable matrix. I will continuously think about this to propose one in the following version paper.

**Q4**: Why is computational efficiency measured/reported only for the AIDS dataset?

**AQ4**: Thanks for the question. We have extended Table 3 in the paper to the full version, including four benchmarks and two additional baselines for a comprehensive comparison. Please refer to Table 2 above in AQ1 for more details. 

**Q5**: Would it be possible to define "early fusion" earlier in the text (even in the abstract)?

**AQ5**: Thanks for your suggestion. We agree with it and have just updated in the abstract of latest draft.

**Q6**: Why is 6,000 epochs necessary? How long does each epoch take? Is this an improvement from existing SoTA methods?

**AQ6**: The quantity of epochs is a hyper-parameter achieved by the stepwise search. As shown in Fig. 6 of the paper, the loss curve tends to be flattened after 5,000 – 5,500 epochs which makes 6,000 a good choice. To train a teacher network, each epoch takes ~0.0256 secs on the AIDS. This is slightly slower than the SimGNN, which is ~0.0167 secs per epoch. It is unfair to compare with GENN-A* since it requires 50,000 epochs for training.

**Respond to Minor Comments**:
We sincerely appreciate the reviewer for providing such comments on writing and grammar. All these issues are corrected in the latest draft and we will continue polishing the paper until publishing.

[1] Wang, Runzhong, et al. "Combinatorial learning of graph edit distance via dynamic embedding." CVPR-21.

[2] https://github.com/Thinklab-SJTU/GENN-Astar


 Thanks to the reviewer for the recognition of our paper. We sincerely appreciate your valuable comments. Your concern is addressed below.

**Q1**: Why does the student network sometimes outperform the teacher network?

**AQ1**:
To make sure we understand the concern correctly, let us reiterate the observation noted by the reviewer specifically: In Table 1, on the AIDS dataset, the student’s MSE (1.546) is better than the teacher’s (1.601) after KD. The other four matrices are "the larger, the better".

**Sub-Q1**: “if it is often not necessary in terms of accuracy, what is the point?”

**ASQ1**: The result above happens *after* the student is trained with KD (i.e, with the help of the teacher). Without KD, all the students *underperform* the teachers (see Table 2 of the paper). Therefore, the teacher is necessary for the student to achieve good performance.
The reviewer may really ask “why the small student can even surpass the big teacher after KD”. There are two reasons that we think may explain it. (1) Theoretically, there is no theory saying that the student’s performance is upper-bounded by the teacher, so it is possible the student can surpass the teacher. (2) Practically, in fact, “student beats teacher after KD” is a quite common phenomenon noted by many KD papers (e.g., [1, 2]). In [1], the authors even use small students to guide large teachers but still can improve the teachers after KD. Despite being counterintuitive, it indeed happens. This phenomenon remains an open question in the area of KD.

**Sub-Q2**: “How common is this?” 

**ASQ2**: Throughout our paper, there is only one case where students beat teachers after KD, so it is fairly rare.

**Sub-Q3**: "Under what conditions/problems/data sets does this happen?"

**ASQ3**: The observation happens on the AIDS dataset. Beyond this, we do not have a deeper understanding of why it happens for now. As explained above, this remains an open question in the area of KD. We will also keep exploring it.

**Sub-Q4**: “Could one just learn the student model and have the fast inference technique?”

**ASQ4**: Not really. We have conducted the ablations on such a case in Table 2 of the paper. If the student is trained without KD, it does not work as effectively. It will be fast, but its performance is not good.

[1] Yuan, Li, et al. Revisiting knowledge distillation via label smoothing regularization, CVPR 20

[2] Wang, Huan, et al. Knowledge Distillation Thrives on Data Augmentation, arXiv 21.

**Q2**: Test other baselines directly inputting offline features.

**AQ2**: Thanks for your suggestion. Off-line embedding collections is one of the key contributions of this paper. Basically, there are two types of baseline methods: Combinatorial Optimization (CO) based methods (i.e., Hungarian and GENN-A*) and GNN based methods (i.e., SimGNN and E-SimGNN). It is inaccessible to collect the features of CO-based methods under the off-line fashion since such methods require computing the node-wise similarity sequentially. On the contrary, the GNN-based approaches are friendly for off-line embedding collection. Following your suggestion, we have added the two baseline methods (i.e., SimGNN-F and E-SimGNN-F), which can directly input embeddings extracted from a pre-trained GNN. In the latest draft, we revised Table 3 and included the results of such baselines. All the experiment settings and environments exactly follow the original Table 3. Here is the time on the whole test set of AIDS.

|  | SimGNN | SimGNN-F | E-SimGNN | E-SimGNN-F | Teacher | Student-R | Student-F |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Time (sec) | 11.136 | 2.591 | 9.672 | 3.464 | 11.139 | 10.149 | 0.148 |

Tab 1. Inference time (sec) to solve GED problems on the **whole test set** of AIDS.

According to the table above, although the inference time is accelerated by 2 – 3 times, our proposed distilled student model (Student-F) is still much faster (~18x) than the feature-based GNN methods (SimGNN-F and E-SimGNN-F). Such performance could sufficiently demonstrate our superiority over the baselines. For more comparisons over four datasets on the speed, please refer to the Table 2 of AQ1 to Reviewer wszK.

**Q3**: What if bad actors could do an adversarial search in the complex graphs? What could negatively happen if we have very fast graph similarity computation?

**AQ3**: This is a great question. We do care a lot about the transparency and justice of applying ML algorithms to real-world applications. There could be some risk on the social impact of our method. For instance, hackers might use it to retrieve the social networks' files for stealing private information. There are some previous works [1] [2] [3] to address the graph domain attacks with the help of adversarial learning or federated learning. Although defense is not the focus of this paper, we think this is a promising topic, and our method can be combined with such training paradigms to avoid illegal use. We also have explained some of the other potential impacts in the last section of supplementary material.

[3] Wang, Shen, et al. "Adversarial defense framework for graph neural network." arXiv 19.

[4] Zhang, Xiang, and Marinka Zitnik. "GNNGuard: Defending Graph Neural Networks against Adversarial Attacks." NeurIPS 20.

[5] Mei, Guangxu, et al. "Sgnn: A graph neural network based federated learning approach by hiding structure." Big Data 19.


 Thank you very much for your valuable comments. Your concerns and questions are addressed below.


**Q1**: What does the \delta indicate here, and what is the different meaning compared with the ranking score?

**AQ1**: Thanks for this question. In Figure 7 of the paper, the $\Delta$ represents the difference between the predicted score and the ground truth score, which can be denoted as $\Delta = |GED_{pred} – GED_{gt}|$. The ranking score shows the similarity between the source and target graphs, and \delta is to evaluate the predicted GED score.

**Q2**: What is the open-source plan of this paper?

**AQ2**: Thanks for asking this question. Open-source is one of the fundamental parts of the ML community. We plan to release the code as well as the models upon the acceptance of the paper.

**Q3**: Why direct distillation of joint embedding fails to transfer knowledge?

**AQ3**: Thanks for pointing this. Intuitively, the joint embedding involves the information of two graphs which is dynamic and not representative. As shown in the Joint Feat line of Table 2 in the paper, the KD on such joint embedding is significantly inferior to the decomposed one on all the matrices. Such a phenomenon can further verify that the joint embedding KD fails to transfer knowledge from teacher to student.

**Q4**: Why not providing the t-SNE map of the student model?

**AQ4**: Thanks for your careful observation. The t-SNE map of the student model is important to this paper, and we will add such a map in the following version draft.

**Other comments and suggestions**:

**Answer**: Thanks for your suggestions on the writing, formulation, and organization. We will follow these comments to continuously polish the paper to make it better.


The proposed model leverages co-attention mechanisms in graph neural networks and knowledge distillation to compute the similarity between two graphs As a result, the model is able to approximate graph similarity (exact computation is an NP-hard problem) more efficiently without compromising on accuracy  Major Comments
- I recognize the computational challenges of graph similarity computation. Given such great improvements in computational efficiency, how well would the proposed method work on larger graphs? Currently, the benchmark datasets are quite small (i.e. <2000 graphs, each with ~10 nodes)
- What is particularly challenging about the AIDS dataset such that the proposed methods do not perform as well as existing SoTA methods despite achieving faster computational efficiency?
- At what point does the drop in accuracy become inappropriate? Are there other ways to evaluate performance and demonstrate that the accuracy tradeoff is not too detrimental (i.e. domain-specific metrics)?
- Why is computational efficiency measured/reported only for the AIDS dataset?

Minor Comments
- The author checklist is incomplete. There are many "See ." statements with missing pointers to the relevant sections in the main paper. 
- Please be consistent with MSE acronym (i.e. MSE or mse).
- Would it be possible to define "early fusion" earlier in the text (even in the abstract)?
- Why is 6,000 epochs necessary? How long does each epoch take? Is this an improvement from existing SoTA methods?
- "...which, however..." is unnecessarily confusing; easy to stumble.
- "Multi-layer Perception (MLP)" should be "Multi-layer Perceptron (MLP)"
- Many typos, i.e. on line 126 and grammatical errors. No, there is no broader impact statement

This paper presents a novel approach to address Graph Similarity Computation (GSC) both accurately and efficiently. From my point of view, the technique contributions of this paper can be summarized in two perspectives, i.e., feature fusion and knowledge distillation. The proposed slow learning employs multi-scale integration and attention-based feature fusion for cross-graph interaction, making it achieve cutting-edge accuracy on most datasets within a comparable computation cost. The fast inference module is the major contribution and the most attractive point of this paper. The authors apply the embedding decomposition to separate the individual features for knowledge distillation, which successfully improves efficiency by vast scale without much accuracy drop. Overall, this paper has made a solid contribution to this field with comprehensive experimental results to support it.  The detailed comments are provided below.

Strengths:

+ From the technique perspective, the distillation of attention-based models into separate ones is a novel direction with limited previous works [1]. The crucial challenge here is the joint feature is unfriendly for knowledge transfer. This paper presents a novel embedding decomposition method to address it successfully, which is valuable.

+ The improvement in the inference speed is significant. Compared with previous co-attention-based methods, the distilled one shows the overwhelming superiority in speed (65x faster). Such high speed might be helpful for industrial usage.

+ In the teacher model, the proposed embedding fusion network employs a simple stacking of an MLP and an attention layer which is slightly novel and outperforms previous approaches [2] with comparable computation time. The design of the teacher model can be regarded as a minor contribution of this paper.

+ The logic of this paper is clear with detailed formulation, making it easy to understand for readers. And the discussion of related works in Sec2 covers most of the well-known literature.

+ The experimental results are comprehensive. There are multiple matrices and public datasets to evaluate the performance of models. Some detailed ablation analysis and visualization are helpful as well.

[1] A. Miech et al. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. arXiv 21.
[2] Y. Bai et al. Learning-based efficient graph similarity computation via multi-scale convolutional set matching. AAAI 20.

 
Suggestions and Questions:

1. It seems that some of the formulations could be further simplified. I recommend authors carefully check them.

2. I am slightly confused about Fig 7. What does the \delta indicate here, and what is the different meaning compared with the ranking score.

3. Do authors have any plan to release the code? I will be appreciated if doing so. Drawbacks and societal impacts are listed below.

Drawbacks:

-The major concern of this paper is that some of the contents are not clear enough. For instance, since the authors mentioned that the direct distillation of joint embedding fails to transfer knowledge, there should be more explanation on this point. 

-Parts of experiments seem missing. For example, in Fig 5, why not providing the tsne map of the student model. And Fig 6 is not quite clear to compare the difference of baseline methods. I recommend the authors zoom in Fig 6.

-There are some grammar errors or typos. For instance, “the color of dots represent the similarity score” in the caption of Fig 5 should be revised as “the colors of dots represent the similarity score”.

Societal Impact:

I have not seen the negative societal impact of this paper.

Meta-review: This paper addresses the problem of speeding up the computation of accurate similarity between two graphs, which is one of the key problems in the highly active field of graph learning.
In order to mediate between time-consuming-but-detailed comparisons and fast-but-insufficient comparisons, this paper proposes a distillation approach.
The use of distillation in this context of graph comparison is new and makes much sense, and the tailed experiments also supports the effectiveness of the proposed method (although the property for very large graph is still open to investigation.)
Overall, this paper makes a solid technical contribution to this field and will be well received by the NeurIPS community.




Decision: accept