Is the optimizer on CUDA?

username: FlorentMeyer
cooked: <p>Hey!</p>
<p>I just have a quick question.<br>
I got a bit confused after having read this thread:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/2830#issuecomment-336194949" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/2830#issuecomment-336194949" target="_blank" rel="nofollow noopener">optimizer load_state_dict() problem?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-09-22" data-time="13:33:28" data-timezone="UTC">01:33PM - 22 Sep 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-11-28" data-time="14:56:40" data-timezone="UTC">02:56PM - 28 Nov 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/JianyuZhan" target="_blank" rel="nofollow noopener">
          <img alt="JianyuZhan" src="https://avatars1.githubusercontent.com/u/649042?v=4" class="onebox-avatar-inline" width="20" height="20">
          JianyuZhan
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Hi, I encountered this bug:
 optimizer.step()
 exp_avg.mul_(beta1).add_(1 - beta1, grad)
TypeError: add_ received an invalid combination of arguments - got (float, torch.cuda.FloatTensor),...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">awaiting response (this tag is deprecated)</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">needs reproduction</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
It’s quite old now and I saw things change quickly here!<br>
My question is (I didn’t really find much about it on the forum or the doc): does the optimizer have to be set onto CUDA?<br>
I know you must set the model onto CUDA before calling the optimizer (therefore its <code>state_dict</code> is put onto CUDA I guess), and set the inputs onto CUDA too. But what about the optimizer? It does have a <code>state_dict</code> but I found nothing except the linked example about it. Does it mean optimization computations are always performed on the GPU (or maybe CPU)? Are the <code>optimizer.state.values</code> the same as its <code>state_dict</code>?
<p>I’m quite curious so if someone knows what determines the device used for computations… I know that model.cuda() sets the optimizable parameters onto CUDA and it just works, but what if I tried to put half the layers’ parameters on CUDA and half on CPU? Where would the computations then happen?</p>
<p>Bonus question: according to <a href="https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165/2">this other thread</a> it looks like constructing the optimizer before <code>model.cuda()</code> works just fine except for optimizers that use a “buffer” (dunno what this buffer is) which makes me think the optimizer “lives his own life” no matter the place the model is stored into…</p>
<p>Thanks in advance,<br>
Regards,<br>
Florent</p>

username: FlorentMeyer
cooked: <p>Please allow me to re-up this question <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>