Why do we need Gradient Checking

TheOraware: <p>When pytorch builtin optimizer loss.backward() could calculate gradient and it is proven correct in its calculation , even though why we need to check derivative computation? Whats wrong in my following code to check derivative computation in back propagation? Or we check gradient computation when we implement gradient computation using our python code not using pytorch loss.backward()?</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    # forward pass (prediction) and loss
    y_predicted = model(X)
    loss = criterion(y_predicted,y)
    # backward loss
    # calculate the gradients 
    loss.backward()
    # update weights
    optimizer.step()
    
    optimizer.zero_grad()
    
    running_loss +=loss.item()
</code></pre>

ptrblck: <p><code>gradcheck</code> and <code>gradgradcheck</code> can be used for custom operations as a test and is also used internally in unit tests.<br>
While I’m sure the gradient calculations are correctly implemented, I’m also sure they would break without proper testing due to (various) changes in the code base. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for giving insight , but in above code as i mentioned there is no any custom code all is code belongs to builtin pytorch , in which scenario it may break?</p>

ptrblck: <p>I’m not sure, if I understand the question correctly, but it could break, if a developer pushes a change to the code base, which could contain a mistake in any calculation, which is why tests are needed.<br>
If you don’t write custom <code>autograd.Functions</code> or work on the backend directly, these methods might be uninteresting for you.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks , could you please share a simple code for writing custom autograd functions?</p>
<p>Still i am not clear if i write following code framework to train neural network then how could it be break? Calculating gradients loss.backward and update weights optimizer.step are builtin in pytorch how come it will break? Are you talking about changes made by pytroch team within new package/release of pytorch?</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    # forward pass (prediction) and loss
    y_predicted = model(X)
    loss = criterion(y_predicted,y)
    # backward loss
    # calculate the gradients 
    loss.backward()
    # update weights
    optimizer.step()
    
    optimizer.zero_grad()
    
    running_loss +=loss.item()
</code></pre>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">This tutorial</a> explains how to write custom <code>autograd.Fucntion</code>s.</p>
<aside class="quote no-group" data-username="TheOraware" data-post="5" data-topic="123336">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theoraware/48/30315_2.png" class="avatar"> TheOraware:</div>
<blockquote>
<p>Still i am not clear if i write following code framework to train neural network then how could it be break? Calculating gradients loss.backward and update weights optimizer.step are builtin in pytorch how come it will break? Are you talking about changes made by pytroch team within new package/release of pytorch?</p>
</blockquote>
</aside>
<p>If you are “just using” built-in PyTorch methods, you are most likely not interested in testing utilities.<br>
Yes, since PyTorch is an open source framework with a lot of active developers, things can accidentally break, which is why testing is needed.</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks much appreciated - can you please just share a small piece of example where gradient is being checked , further what need to measure to decide graident is correct or not?</p>