Zero gradient problem..please help

Sana_Sabah: <p>this the first time I post here,<br>
I am new in Pytorch, I am trying to train my first image recognition with MNIST.<br>
I noted when I don’t use transforms.Normalize((0.5,), (0.5,)) , The NN wights don’t change, and the gradient remains zero, but when I use the normalization the wights are updated and the gradient change, why is that?</p>
<p>transform = transforms.Compose([transforms.ToTensor(),<br>
transforms.Normalize((0.5,), (0.5,)),<br>
])<br>
mnist_train = datasets.MNIST("./", train=True, download=True, transform=transform)<br>
train_loader = DataLoader(mnist_train, batch_size=batch_size, shuffle=False)</p>
<p>input_size=784<br>
hidden_size=100<br>
output_size=10<br>
network = nn.Sequential(<br>
nn.Linear(input_size, hidden_size), # First layer (hidden)of the network takes the entire image and reduces it to 100 dimensions<br>
nn.ReLU(),<br>
nn.Linear(hidden_size, output_size), # The second layer(output) takes those 100 dimensions and reduces them into estimeated values for each digit<br>
nn.Softmax(dim=1)<br>
)</p>
<p>images, labels = next(iter(train_loader))<br>
images = images.view(-1,784)<br>
labels=torch.eye(10)[labels]<br>
output = network(images)<br>
los_fun=nn.MSELoss()<br>
loss = los_fun(output, labels) <span class="hashtag">#calculate</span> the loss</p>
<pre><code>print('Before backward pass: \n', network[0].weight.grad)

loss.backward() # to calculate gradients of parameter 
print('After backward pass: \n',network[0].weight.grad)
</code></pre>
<p>optimizer = optim.SGD(network.parameters(), lr=0.01, momentum=0.9)</p>
<p>print('Initial weights - ', network[0].weight)</p>
<p>images, labels = next(iter(train_loader))<br>
images = images.view(-1,784)</p>
<h1>Clear the gradients, do this because gradients are accumulated</h1>
<p>optimizer.zero_grad()</p>
<h1>Forward pass</h1>
<p>output = network(images)<br>
labels=torch.eye(10)[labels]</p>
<p>loss = los_fun(output, labels)</p>
<h1>the backward pass and update weights</h1>
<p>loss.backward()<br>
print(‘Gradient -’, network[0].weight.grad)</p>

Henry_Chibueze: <p>What <code>transforms.Normalize((0.5,), (0.5,))</code> does ;</p>
<pre><code class="lang-auto">(X - mean(X) )/ std(X)
</code></pre>
<p>Where X is image data<br>
Mean (X) mean pixel value of X<br>
std(X) standard deviation of X</p>
<p>The normalize function normalizes your data to be with the range of (-1, 1)<br>
So using values 0.5 as mean and standard deviation means that your dataset has been scaled to range (0, 1).<br>
If your dataset is not scaled to range (0, 1) before normalize, the u either scale them by dividing the raw image by 255 or setting ur mean and std in normalize function to 255/2.</p>
<p>After the <code>loss.backward()</code> you school also call the <code>optimizer.step()</code> function those are what the backward propagation and weight update consists of.</p>
<p>If u did that then:<img src="https://discuss.pytorch.org/images/emoji/apple/point_down.png?v=9" title=":point_down:" class="emoji" alt=":point_down:"></p>
<p>As for why your gradients don’t change without the normalize, this is the 1st time I’m hearing such<br>
Does it change but the change is very minute and insignificant or it doesn’t change at all?</p>