Beyond Classical Diffusion: Ballistic Graph Neural Network

Abstract: This paper presents the ballistic graph neural network. Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline. The ballistic graph neural network does not require to calculate any eigenvalue. The filters propagate exponentially faster(σ2∼T2) comparing to traditional graph neural network(σ2∼T). We use a perturbed coin operator to perturb and optimize the diffusion rate. Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters. We also show the perturbed filters act as better representations comparing to pure ballistic ones. We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network's performance can be improved.

Reviews and author responses:

The paper "Beyond Classical Diffusion: Ballistic Graph Neural Network" tackles the problem of graph vertices representation. While most existing works rely on classical random walks on the graph, the paper proposes to cope with the "speed of diffusion" problem by introducing ballistic walk.

I noticed the comment of the authors that gives a correction for the introduction. But even with it the paper remains very cryptic, with very few pointers to help the reader in understanding the contribution. The introduction (even corrected) is very abrupt and it is very difficult to understand the problem that the authors propose to attack. The problem is that authors start with mathematical discussions before presenting the manipulated concepts and formalizing the adressed problem. I only understood the adressed problem after seing which are the baselines the proposal is compared with in section 4.2. Also, the introduction does not introduce the proposal at all. 

A symptomatic example of the lack of paper positioning is the Related Works section which does not even give a single reference !  A related work section with no related works in it appears to have a limited interest to me...  This section should at least introduce other works in the field of graph embedding, such as those reported as baselines. It would also greatly help to understand the contribution of the paper. Also, the ballistic concept is not introduced at all in section 4. Where does this term comes from ? The proposed approach is completely cryptic, with clearly not enough definition of the notations the algorithm deals with. A global view of the approach, from the input graph to the final representation, would also be required to help the reader to understand the proposal. If the contribution is only a new kind of random walk on a graph, is ICLR the good targeted venue ? If authors think so, they should present their contribution in a representation learning perspective, which would highlight the importance of this new walk for the graph representation learning process. 


From my point of view, without a full re-writting of the paper, this work cannot be published in a conference like ICLR.   

Thanks for the review.
The section after the introduction is called the speed problem, this section discuss the problem in traditional diffusion based method.
Answer: We did not find related work on controlling the diffusion speed of the kernel on GCN, so there is no relevant paper. We now delete the Related Works section. 

For understanding the algorithm:
We are deeply sorry for introducing some uncommon mathematic notations, these notations are usually used in physics community, we showed a demonstration in the new manuscript, the one-dimensional case to help you understand. also, this link can be very helpful
https://en.wikipedia.org/wiki/Quantum_walk
please see part Discrete Time Quantum Walk, it uses a similar notation as we used in this paper.

And this  (we have no interests with this blog author)
https://www.youtube.com/watch?v=EVw90GBU_Rg
https://towardsdatascience.com/creating-a-quantum-walk-with-a-quantum-coin-692dcfa30d90

For "...a limited interest to me..." 
Answer: We are using a a complete new method. The reviewer should read some graph convolutional papers, like some we mentioned in the baseline experiments. They all use classical diffusion methods.

If the contribution is only a new kind of random walk on a graph, is ICLR the good targeted venue ?
Answer: Yes, random walk on graph is a very important problem. In GCN, random walk is the way you collect the information.

From my point of view, without a full re-writting of the paper, this work cannot be published in a conference like ICLR.   
We re-write the paper.

Thanks 

The objective of the Discussion part of this paper is to show the reader what on earth the ballistic walk is doing.
As we discussed in the new manuscript's introduction part: we already know that the laplacian filter the most GCN use is a low pass filter. 
Our paper proposes the ballistic filter and uses the ballistic walk to gather node representation.

As discussed, the ballistic walk is faster because it put more weight on more distant nodes within the same number of walking steps. And also, the ballistic distribution is more oscillated, since it oscillates, it gathers more frequency information.

However, the direct Fourier analysis of ballistic walk is very hard to define because the ballistic algorithm itself can not be easily translated to some matrix operation.

Thus, the purpose of the Discussion part is to show that:

To some extent(1D), the ballistic walk can be seen as a gaussian high pass filter and contain more information.

Thanks for reviewing our paper,
We apologize and will fix the typos soon. 
Some quick answers:
- Sec 3, Eq 1 seems to have been taken from Eq 1 in Defferard et. al, however there’s no reference to it and the terms g, U, etc. are not defined. 
The reference is listed, and the details equations are list in the reference. will add the details of that definition.

 - Sec 4, Algo 1 contains the main core of the proposed algorithm, but it’s only defined for the 2D grid case. The notation therein is extremely unclear. What is H_space, H_c? How does one sample \hat{O}_coin.? The net result is that algorithm is undefined. Without a clear definition of the algorithm, it’s completely unclear what the proposed method does.
H_space, H_c is defined in our paper, \hat{O}_coin is also defined, and its a MATRIX, we do not sample matrix. \hat{O}_coin is defined in the Algorithm, see line 4.

We are deeply sorry for introducing some uncommon mathematic notations, these notations are usually used in physics community, we showed a demonstration in the new manuscript, the one-dimensional case to help you understand. also, this link can be very helpful
https://en.wikipedia.org/wiki/Quantum_walk
please see part Discrete Time Quantum Walk, it uses a similar notation as we used in this paper.

And this  (we have no interests with this blog author)
https://towardsdatascience.com/creating-a-quantum-walk-with-a-quantum-coin-692dcfa30d90
https://www.youtube.com/watch?v=EVw90GBU_Rg

 - Sec 4.2 is completely unparseable. What is problem setting? What is the metric? How have the baselines been implemented? How has data been split for training/testing?
We add more details in the new manuscript.
What is problem setting? it MNIST classification test.
What is the metric?How have the baselines been implemented? It is mentioned in our paper.
 How has data been split for training/testing?
WE are using MNIST data, MNIST is generated by LeCun, please see http://yann.lecun.com/exdb/mnist/ The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
We do NOT split MNIST training/testing because it was splitted.

 - Section 5 mentions that one-third params are used to get 97% but no details are provided as to how less params are consumed.
we provide 1/3 to show the our method is better, it's a estimation, we do not use the same pipeline, the exact comparison of params is meaningless in this case.

 - How is figure 7 generated?
It generated use two different walkers with different initial conditions, see the caption.
The figure means: for classical diffusion-based method, it put small weight on distant nodes. so when you take a 25 steps, since the speed is $\frac{1}{\sqrt{k}}$, the average distance is $\sim \sqrt{25}=5$. But as you use ballistic ones, the average distance is much larger and will exceed the image size, for MNIST the image size is 28, if you start from the center, then when you talk around 14 steps ballistic walk, the average weight distance(following the linear transportation speed) will reach $\sim 14$, thus has the exceeding boundary problem.  


1. The motivation of this method is to accelerate the diffusion speed in a graph. However, as we know, a very severe issue of graph neural network is the over-smoothness issue. The reason is that, in the high layer, the node feature is diffused to far neighbours.  When using the proposed ballistic filter, node features diffuse much faster than the regular GNN. Thus, the over-smoothness will appear in the shallow layer very fast. As a result, we cannot use many layers so that the non-linearity of deep neural networks cannot be fully utilized. Thus, is it necessary to accelerate the diffusion speed for graph neural network?

Answer:  Yes, It is necessary.
When we are saying ballistic filters are faster, it does not mean the filters are smooth, actually the distribution is very oscillated, please see our updated manuscript, we showed the one-dimensional condition. As you use a oscillated filter, unlike the traditional random walk filters, the ballistic distribution make the nodes distinguishable.
The over smooth problem: means after layers and layers, the $L^n$ will become smooth on the spatial region, which becomes a low pass filter. In GCN, as long as you use laplacian based methods(in this paper we call classical diffusion based methods), the distribution will always have a similar shape: The central part of the distribution always larger (or at least equal than the boundary part)
When we exponentially accelerate to diffusion, please see our Figure 15, we show the cumulative distribution of 24th and 25step of ballistic diffusion,  NOTE that now the central part is smaller than than boundary part, in that case over smooth will not happen, and classical diffusion will never reach that shape.
Although the feature diffuses faster, but the shape of distribution has changed, for same number of steps $k$, both ballistic one and classical one hit the $k$-distance boundary, the only difference is the weight they put on the node. 

The figure in this link is the comparison between the one dimensional distribution(classical vs ballistic). 
As you can see, the classical diffusion is not  oscillated enough, thus resulting in the indistinguishable of node representation.
https://upload.wikimedia.org/wikipedia/commons/4/4b/One_dimensional_quantum_random_walk.png

My guess is that the only some exponentially accelerated kernels will solve the over-smooth problem.

2. There is only one dataset for  the comparison of the performance of different graph neural networks. More datasets are needed to thoroughly verify the performance of the proposed ballistic graph neural network.

Please see the updated Manuscript. we run experiments on more datasets.

3. Is it possible to slow down the diffusion speed with the proposed ballistic filter?
Yes, we already slow down to a ballistic to classical region, please see Figure 10, slow down the diffusion to classical case means use Laplacian matrix to extract the feature. If you want to slow down to the localization condition, a simple strategy is control how 'lazy' the random walk is:  for example, take $L=(I+\alpha D^{-1}A)$, $\alpha$ controls the speed, smaller $\alpha$ results in slower speed.



This paper proposed a new diffusion operation for the graph neural network. Specifically, the ballistic graph neural network does not require to calculate any eigenvalue and can propagate exponentially faster comparing to traditional graph neural network. Extensive experiments have been conducted to verify the performance of the proposed method.

1. The motivation of this method is to accelerate the diffusion speed in a graph. However, as we know, a very severe issue of graph neural network is the over-smoothness issue. The reason is that, in the high layer, the node feature is diffused to far neighbours.  When using the proposed ballistic filter, node features diffuse much faster than the regular GNN. Thus, the over-smoothness will appear in the shallow layer very fast. As a result, we cannot use many layers so that the non-linearity of deep neural networks cannot be fully utilized. Thus, is it necessary to accelerate the diffusion speed for graph neural network?

2. There is only one dataset for  the comparison of the performance of different graph neural networks. More datasets are needed to thoroughly verify the performance of the proposed ballistic graph neural network.

3. Is it possible to slow down the diffusion speed with the proposed ballistic filter?




This paper was extremely hard to read or comprehend. It’s riddled with typos, inaccurate notations and undefined variables (see below for a sampling). The authors will need to significantly polish and improve the presentation of the paper. 

After a few forward and backward passes through the paper, I was able to gather the following high level ideas about the paper:
(1) This paper is somewhat related to the Defferard et. al, 2016 in that the authors want to define a propagation filter for graph neural networks.
2) This proposed filter known as “ballistic filter” should have the property of allowing fast diffusion through the network. 
(3) The authors claim that the ballistic kernel diffuses @ O(k) as compared to O(\sqrt k) when compared to traditional GCNs, where k is the number of propagation steps.
(4) The authors additionally claim that their approach needs one-third the number of parameters.
(5) The authors provide some plots to visualize the linear diffusion rate of their proposed filter.

--- Issues and clarifications ---
- Sec 3, Eq 1 seems to have been taken from Eq 1 in Defferard et. al, however there’s no reference to it and the terms g, U, etc. are not defined.
 - Sec 4, Algo 1 contains the main core of the proposed algorithm, but it’s only defined for the 2D grid case. The notation therein is extremely unclear. What is H_space, H_c? How does one sample \hat{O}_coin.? The net result is that algorithm is undefined. Without a clear definition of the algorithm, it’s completely unclear what the proposed method does.
 - Sec 4.2 is completely unparseable. What is problem setting? What is the metric? How have the baselines been implemented? How has data been split for training/testing?
 - Section 5 mentions that one-third params are used to get 97% but no details are provided as to how less params are consumed.
 - How is figure 7 generated?
 - Sec 8, feel totally unrelated to the paper. There are a whole bunch of random, unmotivated diffusion equations  Eq 6, mentions “.. \hat{g}(f) decreases as f increases and thus can be seen as a low pass filter…” . This is not true from the formula.
 

 --- A sampling of typos ---
Sec 4.1, .. consisits …
Sec 5 “REVISIT” -> “REVISITING”
Figure 6, text, “cassical”
Sec 6.2 title, “SUMMAY”
Sec 8  “aggreated”
Sec 8  t=\-tau to -\tau
Several typos with Hardmard, Hadmard instead of Hadamard.

Overall, the major criticisms of this paper:
 - The proposed algorithm is not clear. 
 - The authors need much more experimentation to bolster their claims in the paper. It’s completely unclear if fast diffusion even if it were possible will help GNNs perform better on a diverse set of tasks.
 - The paper needs a lot more polish and proof reading to make this paper presentable.


Meta-review: This submission has been assessed by three reviewers who scored it 3/1/3, and they have remained unconvinced after the rebuttal. The main issues voiced are the difficult readability of the paper, cryptic at times due to a mix of physical and DL notations, and a lack of sufficient experimentation to support all claims. The reviewers acknowledge the authors' efforts to resolve the main issues but find these efforts insufficient. Thus, this paper cannot be accepted to ICLR2020.

Decision: reject