Repository: dmlc/dgl

Issue:
u_mul_e is not deterministic
## 🐛 Bug

If I run the same graph/data through `u_mul_e` many times, I can get different results even though none of the underlying tensors are changing.  I have only been able to observe this on a particular graph and it unfortunately does not have the same behavior on any of the built-in graphs I have tried.

## To Reproduce
This is the code I am using, which is an implementation of Label Propagation using edge weights:

```python
import torch as th 
from torch import nn
import dgl.function as fn
import dgl

def test_graph_equality(g1, g2):
    assert th.equal(g1.edges()[0], g2.edges()[0])
    assert th.equal(g1.edges()[1], g2.edges()[1])
    assert th.equal(g1.nodes(), g2.nodes())
    
    # Adjacency matrix
    assert th.equal(g1.adj().coalesce().indices(), g2.adj().coalesce().indices())
    assert th.equal(g1.adj().coalesce().values(), g2.adj().coalesce().values())
    assert g1.adj().coalesce().size() == g2.adj().coalesce().size()

    
class LPAMinimal(nn.Module):

    def __init__(self):
        super(LPAMinimal, self).__init__()
        
    def forward(self, graph, y, k_iters):
        g_copy = dgl.graph(graph.edges()) # Just for testing that nothing is being modified
        
        y_out = y.clone().detach().requires_grad_(True)
        y_out_copy = y.clone().detach().requires_grad_(True) # Again, for testing that nothing is changing
        
        ys = list()
        with graph.local_scope():
            for k in range(k_iters):
                # Each iteration should generate the *same* output
                graph.ndata['h'] = y_out
                graph.update_all(
                    fn.u_mul_e('h', 'edge_weight', 'm'), # This causes the problem, even when "edge_weights" are all 1.
                    #fn.copy_u('h', 'm'),  # This makes the problem go away.
                    fn.mean('m', 'h'),
                )
                h = graph.ndata['h']
                ys.append(h)
                
                # test_graph_equality(g_copy, graph) # This passes 
                # assert th.equal(y_out, y_out_copy) # This passes 

                # y_out = h  # Un-commenting this would implement Label Propagation; Ignore for this bug.
                           
        return ys

def lpa_test_small(g, labels):
    lpa = LPAMinimal()
                      
    g.edata['edge_weight'] = th.ones((g.num_edges(), 1))  # Set to 1.0 so should match "fn.copy_u('h', 'm')
    ys = list()
    for epoch in range(10):
        ys += lpa(g.clone(), labels.unsqueeze(1).clone(), k_iters=3) # runs inner loop 3 times
        
    equal = [th.equal(ys[0], ys[i]) for i in range(1, len(ys))] # Tests if all iterations are equal
    return all(equal)
```

## Expected behavior

Since the graph, edge weight tensor, and initial embeddings "y" are all constant, I expect any number of iterations to give identical results.  This is the case for all the built-in dgl.data graphs I have tried, but fails for my graph.  All of the graph equality tests pass and there's no indication that any underlying data structure involved in the computation is changing.

## Environment

 - DGL Version (e.g., 1.0): 0.5.2
 - Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):  PyTorch 1.5.0
 - OS (e.g., Linux):  Amazon Linux (SageMaker notebook, p3 instance)
 - How you installed DGL (`conda`, `pip`, source): pip
 - Python version: 3.6.10
 - CUDA/cuDNN version (if applicable): 10.0
 - GPU models and configuration (e.g. V100): p3 instance
 
## Additional context

It's fairly difficult to get the inconsistency to happen.  It will seem to be consistent for many iterations before generating a new result and can sometimes go back to generating the original, consistent result.  It's been tricky to track down because it's relatively infrequent.  Unfortunately, I can't share my graph publicly.  But reach out to jostza@ in AWS if you need to discuss more details of the data.



Discussion:
Update:
If I replace `fn.u_mul_e('h', 'edge_weight', 'm')` by `lambda e: {'m' : e.src['h'] * e.data['edge_weight']}`, it seems to solve the problem and give consistent results.
@zjost could you please also provide us with the graph you use? I cannot reproduce your bug with a random graph:
```python
g = dgl.rand_graph(100, 300)
y = th.rand(100, 40)
print(lpa_test_small(g, y))
```
and the result is:
```python
[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
```
which is the desired behavior IMO.
The `u_mul_e` should be deterministic since 0.5.0, can you confirm that you were using dgl 0.5.2 by:
```python
print(dgl.__version__)
```
I uninstalled and re-installed now using version 0.5.3 and confirm the problem still exists.  As I mentioned above, I can't share this graph publicly, but I work at AWS, so please reach out to me at "jostza@" on Slack and we can figure something out.
@yzh119
Hey! I had a similar problem with `fn.u_mul_e`. I want to train a large graph by sampling nodes. But there's chance that the built-in `fn.u_mul_e` produces `NaN` values for some edges in the block. However, the `lambda e: {'m' : e.src['h'] * e.data['norm']}` works fine consistently. 

I used `pickle` to dump this block and successfully reproduced it in my machine. Hope this file will help.

[u_mul_e_case.zip](https://github.com/dmlc/dgl/files/5894815/u_mul_e_case.zip)

It is possible that the built-in `fn.u_mul_e`  produces different results when I reload this block. And the udf `u_mul_e` works correctly all the time.

I ran the codes below twice.
```python
with open('./u_mul_e_case.pkl', 'rb') as f:
    b = pickle.load(f)
b.apply_edges(fn.u_mul_e('h', 'norm', 'm'))
msg = b.edata['m']
print(f'Built-in u_mul_e produces {msg.isnan().sum().item()} NaN.')

with open('./u_mul_e_case.pkl', 'rb') as f:
    b = pickle.load(f)
b.apply_edges(lambda e: {'m' : e.src['h'] * e.data['norm']})
msg = b.edata['m']
print(f'The udf u_mul_e produces {msg.isnan().sum().item()} NaN.')
```

The outputs are:
```
Built-in u_mul_e produces 1280 NaN.
The udf u_mul_e produces 0 NaN.
```
and
```
Built-in u_mul_e produces 1792 NaN.
The udf u_mul_e produces 0 NaN.
```
,respectively.


**My Environment**
- DGL Version (e.g., 1.0): 0.5.1
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.6.0
- OS (e.g., Linux): Ubuntu 18.04 LTS
- How you installed DGL (conda, pip, source): pip
- Python version: 3.6.10
- CUDA/cuDNN version (if applicable): 10.2 / cudnn7.6.5
- GPU models and configuration (e.g. V100): RTX 2080 Ti
Thanks @changym3 , I'll have a try,
@changym3 the reason you encounter this issue is that the node feature has dtype float64 while the edge feature has dtype float32. DGL did not auto cast their data type to the same, and the cuda kernel mistakenly regarding float32 tensor as a float64 one, resulting in NaN values. You can either use float32 node features (that's the recommended way because it's faster and saves GPU memory) or convert edge feature to float64.

@zjost please check if it's also your case.

I'll create a patch to check if input's data types are matched. Thanks very much for your feedback.

Resolving patch:
diff --git a/python/dgl/sparse.py b/python/dgl/sparse.py
--- a/python/dgl/sparse.py
+++ b/python/dgl/sparse.py
@@ -120,6 +120,11 @@ def _gspmm(gidx, op, reduce_op, u, e):
         raise DGLError("We only support gspmm on graph with one edge type")
     use_u = op != 'copy_rhs'
     use_e = op != 'copy_lhs'
+    if use_u and use_e:
+        if F.dtype(u) != F.dtype(e):
+            raise DGLError("The node features' data type {} doesn't match edge"
+                           " features' data type {}, please convert them to the"
+                           " same type.".format(F.dtype(u), F.dtype(e)))
     # deal with scalar features.
     expand_u, expand_e = False, False
     if use_u:
@@ -219,6 +224,10 @@ def _gsddmm(gidx, op, lhs, rhs, lhs_target='u', rhs_target='v'):
         raise DGLError("We only support gsddmm on graph with one edge type")
     use_lhs = op != 'copy_rhs'
     use_rhs = op != 'copy_lhs'
+    if use_lhs and use_rhs:
+        if F.dtype(lhs) != F.dtype(rhs):
+            raise DGLError("The operands data type don't match: {} and {}, please convert them"
+                           " to the same type.".format(F.dtype(lhs), F.dtype(rhs)))
     # deal with scalar features.
     expand_lhs, expand_rhs = False, False
     if use_lhs:


Test patch: