Have no cue about the batchnormalization1d impacts the reproducibility or may be a bug

ZhaoBo_Hu: <p>There are snippets of code about batchnorm1d.<br>
Even I set random seed and made the model as <code>eval()</code> or <code>with torch.no_grad():</code>, the output result was varied every time. But while I removed the batchnorm1d, or activation funtion <code>F.relu()</code>, the output maintained the same.  It’s a very stranger behavior, does anyone meet this or have already solved it ?</p>
<pre><code class="lang-auto">def set_seed(random_seed):
    np.random.seed(random_seed)
    random.seed(random_seed)
    
    
    torch.manual_seed(random_seed)
    torch.cuda.manual_seed(random_seed)
    torch.cuda.manual_seed_all(random_seed)

    
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<pre><code class="lang-auto">dataset = Planetoid(root='data/Planetoid', name='Cora', transform=NormalizeFeatures())
device = 'cuda' if torch.cuda.is_available() else 'cpu'
data = dataset[0]
</code></pre>
<pre><code class="lang-auto">class GCN(torch.nn.Module):
    def __init__(self, hidden_channels, num_layer = 10):
        super().__init__()
        self.num_layer = num_layer
        
        self.conv_list = ModuleList([])
        self.bn_list = ModuleList([])

        self.cached = True
        
        
        self.conv_list.append(GCNConv(dataset.num_features, hidden_channels, cached=self.cached, bias=False))
        for _ in range(self.num_layer - 2):
            self.conv_list.append(GCNConv(hidden_channels, hidden_channels, cached=self.cached, bias=False))
        self.conv_list.append(GCNConv(hidden_channels, len(data.y.unique()), cached=self.cached, bias=False))
        
        for i in range(self.num_layer):
            self.bn_list.append(nn.BatchNorm1d(self.conv_list[i].out_channels, momentum=0.3))
            
            
    def forward(self, x, edge_index):
        
        for i in range(self.num_layer):
            if i == 0 or i == self.num_layer-1:
                x = F.dropout(x, 0.6, training=self.training)
            x = self.conv_list[i](x, edge_index)
            x = self.bn_list[i](x)
            x = F.relu(x)
            
            
        return x
</code></pre>
<pre><code class="lang-auto">set_seed(10)
model = GCN(hidden_channels=16).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
criterion = torch.nn.CrossEntropyLoss()

def train():
      model.train()
      optimizer.zero_grad()  # Clear gradients.
      out = model(data.x.to(device), data.edge_index.to(device))  # Perform a single forward pass.
      loss = criterion(out[data.train_mask].to(device), data.y[data.train_mask].to(device))  # Compute the loss solely based on the training nodes.
      loss.backward()  # Derive gradients.
      optimizer.step()  # Update parameters based on gradients.
      return loss

def test():
    
    
    with torch.no_grad():
        model.eval()
        out = model(data.x.to(device), data.edge_index.to(device))
    pred = out.argmax(dim=1)  # Use the class with highest probability.
    test_correct = pred[data.test_mask].to(device) == data.y[data.test_mask].to(device)  # Check against ground-truth labels.
    test_acc = int(test_correct.sum()) / int(data.test_mask.sum())  # Derive ratio of correct predictions.
    return test_acc


for epoch in range(1, 201):
    loss = train()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')
    
test_acc = test()
print(f'Test Accuracy: {test_acc:.4f}')
</code></pre>

J_Johnson: <p>Welcome to the forums!</p>
<p>How are you defining <code>GCNConv</code>? It is not clear from your code and this class does not exist natively in PyTorch.</p>
<p>At any rate, I’m not finding this issue with the BatchNorm1d layers with ReLU. The output is unchanged. Here is a simple example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model = nn.Sequential(nn.Conv1d(3, 64, kernel_size=3, padding=1),
                      nn.BatchNorm1d(64), nn.ReLU(),
                      nn.Conv1d(64, 1, kernel_size=3),
                      nn.BatchNorm1d(1), nn.ReLU(),
                      nn.AdaptiveAvgPool1d(1))

temp_data=torch.rand((1, 3, 128))

model.eval()
with torch.no_grad():
    while True:
        print(model(temp_data))
</code></pre>
<p>If you mean you are getting a difference between having <code>.eval()</code> on and off, that is to be expected. See <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.eval" rel="noopener nofollow ugc">here</a> and <a href="https://discuss.pytorch.org/t/what-does-model-eval-do-for-batchnorm-layer/7146">here</a>.</p>

ZhaoBo_Hu: <p>Hi johnson, GCNConv is from Library PyTorch Geometrics, a powerful library for graph neural network, I believe that the issue not coming from there, this is the reference website : <a href="https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GCNConv.html#torch_geometric.nn.conv.GCNConv" rel="noopener nofollow ugc">GCNConv</a></p>
<p>And this is the complete code, would you mind testing on your side for multiple times to check if the output varied ?</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from torch.nn import ModuleList, ReLU, Dropout
from torch import nn
from torch_geometric.datasets import Planetoid, CitationFull, Coauthor, Amazon
from torch_geometric.transforms import NormalizeFeatures

from torch_geometric.nn import GCNConv, GATConv


import random
import pandas as pd
import numpy as np

def set_seed(random_seed):
    np.random.seed(random_seed)
    random.seed(random_seed)
    
    
    torch.manual_seed(random_seed)
    torch.cuda.manual_seed(random_seed)
    torch.cuda.manual_seed_all(random_seed)

    
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False


class GCN(torch.nn.Module):
    def __init__(self, hidden_channels, num_layer = 10):
        super().__init__()
        self.num_layer = num_layer
        
        self.conv_list = ModuleList([])
        self.bn_list = ModuleList([])

        self.cached = True
        
        
        self.conv_list.append(GCNConv(dataset.num_features, hidden_channels, cached=self.cached, bias=False))
        for _ in range(self.num_layer - 2):
            self.conv_list.append(GCNConv(hidden_channels, hidden_channels, cached=self.cached, bias=False))
        self.conv_list.append(GCNConv(hidden_channels, len(data.y.unique()), cached=self.cached, bias=False))
        
        for i in range(self.num_layer):
            self.bn_list.append(nn.BatchNorm1d(self.conv_list[i].out_channels, momentum=0.3))
            
            
    def forward(self, x, edge_index):
        
        for i in range(self.num_layer):
            if i == 0 or i == self.num_layer-1:
                x = F.dropout(x, 0.6, training=self.training)
            x = self.conv_list[i](x, edge_index)
            x = self.bn_list[i](x)
            x = F.relu(x)
            
            
        return x

dataset = Planetoid(root='data/Planetoid', name='Cora', transform=NormalizeFeatures())
device = 'cuda' if torch.cuda.is_available() else 'cpu'
data = dataset[0]


set_seed(10)
model = GCN(hidden_channels=16).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
criterion = torch.nn.CrossEntropyLoss()

def train():
      model.train()
      optimizer.zero_grad()  # Clear gradients.
      out = model(data.x.to(device), data.edge_index.to(device))  # Perform a single forward pass.
      loss = criterion(out[data.train_mask].to(device), data.y[data.train_mask].to(device))  # Compute the loss solely based on the training nodes.
      loss.backward()  # Derive gradients.
      optimizer.step()  # Update parameters based on gradients.
      return loss

def test():
    
    
    with torch.no_grad():
        model.eval()
        out = model(data.x.to(device), data.edge_index.to(device))
    pred = out.argmax(dim=1)  # Use the class with highest probability.
    test_correct = pred[data.test_mask].to(device) == data.y[data.test_mask].to(device)  # Check against ground-truth labels.
    test_acc = int(test_correct.sum()) / int(data.test_mask.sum())  # Derive ratio of correct predictions.
    return test_acc


for epoch in range(1, 201):
    loss = train()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')
    
test_acc = test()
print(f'Test Accuracy: {test_acc:.4f}')
</code></pre>

J_Johnson: <p>It is not necessary for me to install another library to demonstrate that <code>.eval()</code> and <code>.train()</code> change the outputs of batchnorm and dropout layers.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model = nn.Sequential(nn.BatchNorm1d(1), nn.AdaptiveAvgPool1d(1))

data = torch.tensor([[[0.1, 0.3, 0.2]]])

print(model(data))
model.eval()
print(model(data))
</code></pre>
<pre><code class="lang-auto">tensor([[[3.9736e-08]]], grad_fn=&lt;SqueezeBackward1&gt;)
tensor([[[0.1896]]], grad_fn=&lt;SqueezeBackward1&gt;)
</code></pre>
<p>That is exactly as expected.</p>
<aside class="quote no-group" data-username="J_Johnson" data-post="2" data-topic="177309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>See <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.eval" rel="noopener nofollow ugc">here</a> and <a href="https://discuss.pytorch.org/t/what-does-model-eval-do-for-batchnorm-layer/7146">here</a>.</p>
</blockquote>
</aside>

ZhaoBo_Hu: <p>Hi, the issue is not about the on or off <code>eval()</code></p>
<p>I check my code on CPU, output result is unchange every time but GPU performs differently.</p>
<p>You can check this channel, but I am sure if this bug still exists : <a href="https://github.com/pytorch/pytorch/issues/53691" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/53691</a></p>