Pytorch to Skorch: zero_grad()

username: cjohns
cooked: <p>I’m looking to migrate an embedding model from pure torch to skorch so I can gridsearch to find the best parameters. In PyTorch, I set the embedding size, learning rate, epochs, and batch size.  I set the skorch model up to do the same. Both models run and produce embeddings, but the produced embeddings are pretty different.</p>
<p>Going back through the loop vs. the skorch model, the only thing I can come up with is that the pure torch method uses a loop that sets the gradients back to zero.  I don’t think that skorch is doing this, and I’m wondering if there is a way to implement it.</p>
<pre><code class="lang-auto">def fit(iterator, model, optimizer, criterion): 
  for x, y in iterator: 
    optimizer.zero_grad()
    y_hat = model(x.to(device))
    loss = criterion(y_hat, y.to(device))
    train_loss += loss.item() * x.shape[0]
    loss.backward()
    optimizer.step()
return train_loss / len(iterator.dataset)) 


</code></pre>

username: InnovArul
cooked: <p>It seems skorch does <code>zero_grad()</code> as well.</p>
<p>Fit loop:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L1103-L1112">
  <header class="source">

      <a href="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L1103-L1112" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L1103-L1112" target="_blank" rel="noopener">skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L1103-L1112</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1103" style="counter-reset: li-counter 1102 ;">
          <li>for _ in range(epochs):</li>
          <li>    self.notify('on_epoch_begin', **on_epoch_kwargs)</li>
          <li>
          </li>
<li>    self.run_single_epoch(iterator_train, training=True, prefix="train",</li>
          <li>                          step_fn=self.train_step, **fit_params)</li>
          <li>
          </li>
<li>    self.run_single_epoch(iterator_valid, training=False, prefix="valid",</li>
          <li>                          step_fn=self.validation_step, **fit_params)</li>
          <li>
          </li>
<li>    self.notify("on_epoch_end", **on_epoch_kwargs)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>train_step:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L979-L1022">
  <header class="source">

      <a href="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L979-L1022" target="_blank" rel="noopener">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L979-L1022" target="_blank" rel="noopener">skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L979-L1022</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="979" style="counter-reset: li-counter 978 ;">
          <li>def train_step(self, batch, **fit_params):</li>
          <li>    """Prepares a loss function callable and pass it to the optimizer,</li>
          <li>    hence performing one optimization step.</li>
          <li>
          </li>
<li>    Loss function callable as required by some optimizers (and accepted by</li>
          <li>    all of them):</li>
          <li>    https://pytorch.org/docs/master/optim.html#optimizer-step-closure</li>
          <li>
          </li>
<li>    The module is set to be in train mode (e.g. dropout is</li>
          <li>    applied).</li>
          <li>
          </li>
<li>    Parameters</li>
          <li>    ----------</li>
          <li>    batch</li>
          <li>      A single batch returned by the data loader.</li>
          <li>
          </li>
<li>    **fit_params : dict</li>
          <li>      Additional parameters passed to the ``forward`` method of</li>
          <li>      the module and to the train_split call.</li>
          <li>
      </li>
</ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/skorch-dev/skorch/blob/1491d5a611db5577e81cbeb03219efb2dec0df44/skorch/net.py#L979-L1022" target="_blank" rel="noopener">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>