Gradient Tape in TF vs Autograd in PyTorch

username: alessandro_mondin
cooked: <p>I know that this topic has been discussed many times but among all the past posts I couldn’t find an exhaustive explanation.</p>
<p>I am trying to translate this Tensorflow code to PyTorch where model is a logistic regression.</p>
<pre><code class="lang-auto">    for X, Y in train_data:
    with tf.GradientTape() as tape:
        X = X / 255.0
        y_hat = self.logistic_regression(X, W, b)
        one_hot = tf.one_hot(Y, 43)
        loss = self.func.cross_entropy(y_hat, one_hot)
        losses.append(tf.math.reduce_mean(loss))
        grads = tape.gradient(loss, [W, b])
        self.sgd([W, b], grads, lr, X.shape[0])
</code></pre>
<p>I know that the easiest way would be to create a class inheriting from nn.Module and use loss.backward() and optim.step(), nevertheless since I wanted to use torch.autograd.grad, I ended up with this torch script:</p>
<pre><code class="lang-auto">            for X, Y in train_data:
                X = X / 255
                y_hat = self.logistic_regression(X, W, b)
                one_hot = torch.nn.functional.one_hot(Y, 43).bool()
                loss = self.func.cross_entropy(y_hat, one_hot)
                losses.append(torch.mean(loss).item())
                grads = torch.autograd.grad(loss, [W,b], grad_outputs=torch.ones_like(loss))
                self.sgd([W, b], grads, lr, X.shape[0])
</code></pre>
<p>I noticed that tape.gradient() in TF expects the target (loss) to be multidimensional, while  torch.autograd.grad by default expects a scalar. This difference as far as I understood can be overcame by adding the parameter grad_outputs=torch.ones_like(loss) to torch.autograd.grad.<br>
The problem however, is that even though the two scripts that I have pasted above in TF and PT show be equivalent, the results are very different: the TF one converges rapidly while the one in PyTorch doesn’t and the while in TF the with 10 epochs the training loss goes from 3.97 to 1.32, in PyTorch starts from 3.76 and diminishes to 3.74.</p>
<p>Do you know why?<br>
Thanks a lot</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="alessandro_mondin" data-post="1" data-topic="151104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessandro_mondin/48/46669_2.png" class="avatar"> alessandro_mondin:</div>
<blockquote>
<p><code>self.func.cross_entropy</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="alessandro_mondin" data-post="1" data-topic="151104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessandro_mondin/48/46669_2.png" class="avatar"> alessandro_mondin:</div>
<blockquote>
<p><code>y_hat = self.logistic_regression</code></p>
</blockquote>
</aside>
<p>Would you be able to share the script that you are working with?<br>
The parts that you have shown looks fine to me.</p>
<p>Moreover,  instead of looking at the loss reduction empirically,  you could try to initialize the params, inputs to same values in both tensorflow and pytorch,  &amp; verify whether the loss,  gradients are same.</p>

username: alessandro_mondin
cooked: <p>Hello, <a href="https://github.com/AlessandroMondin/TF_vs_PT/blob/master/training.py" class="inline-onebox" rel="noopener nofollow ugc">TF_vs_PT/training.py at master · AlessandroMondin/TF_vs_PT · GitHub</a> here is the script</p>

username: InnovArul
cooked: <aside class="onebox githubpullrequest" data-onebox-src="https://github.com/AlessandroMondin/TF_vs_PT/pull/1">
  <header class="source">

      <a href="https://github.com/AlessandroMondin/TF_vs_PT/pull/1" target="_blank" rel="noopener">github.com/AlessandroMondin/TF_vs_PT</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/AlessandroMondin/TF_vs_PT/pull/1" target="_blank" rel="noopener">data transform fix for pytorch</a>
    </h4>

    <div class="branches">
      <code>AlessandroMondin:master</code> ← <code>InnovArul:master</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-05-08" data-time="18:48:25" data-timezone="UTC">06:48PM - 08 May 22 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/InnovArul" target="_blank" rel="noopener">
          <img alt="InnovArul" src="https://avatars.githubusercontent.com/u/4779147?v=4" class="onebox-avatar-inline" width="20" height="20">
          InnovArul
        </a>
      </div>

      <div class="lines" title="1 commits changed 3 files with 7 additions and 9 deletions">
        <a href="https://github.com/AlessandroMondin/TF_vs_PT/pull/1/files" target="_blank" rel="noopener">
          <span class="added">+7</span>
          <span class="removed">-9</span>
        </a>
      </div>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">ToTensor() already normalizes the data to [0,1]

https://discuss.pytorch.org/t<span class="show-more-container"><a href="https://github.com/AlessandroMondin/TF_vs_PT/pull/1" target="_blank" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">/gradient-tape-in-tf-vs-autograd-in-pytorch/151104</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>In the project that you have shared, the data transformation of PyTorch was not equivalent compared to TensorFlow. Specifically, the image data range in TensorFlow was between [0,1] &amp; in PyTorch, it was not in that range.</p>

username: alessandro_mondin
cooked: <p>Thanks a lot, now works there’s consistency in the results between the libraries</p>