akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error in move model to cuda

username: aviadp
cooked: <p>Hello, I have assembled a model from several small models.<br>
I transferred the model to cuda and debugged to make sure it works with cuda,<br>
on the other hand when I run the code I get the following error:<br>
“Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same”</p>
<p>I would be happy if someone has an ideaת<br>
Thanks,<br>
Aviad</p>

username: ptrblck
cooked: <p>This error is raised, if (some) model parameters are still on the CPU while the input is on the GPU.<br>
If you’ve registered all parameters and buffers as <code>nn.Parameter</code> or <code>self.register_buffer</code>, the <code>model.to('cuda')</code> operation should move them to the GPU.</p>

username: aviadp
cooked: <p>Thank you,<br>
I’ll try to add that.</p>
How not to consider gradient for specific operation without copying tensors?

username: ZdsAlpha
cooked: <p>I have AutoEncoder consisting of encoder and decoder. I want to reconstruct image. So I am optimizing parameters of both encoder and decoder but at the same time I need to use encoder layers (Usually you use VGG) for computing reconstruction loss. But for second encoder pass, I dont want to consider gradient. torch.no_grad() completely discards graph. Here is what code should have looked like:</p>
<pre><code class="lang-auto">latent = encoder(x)
x2 = decoder(x)
with torch.no_grad():
    latent2 = encoder(x2)
loss = F.mse_loss(x2, x) + F.mse_loss(latent2, latent)
loss.backward()
</code></pre>
<p>Problem is not that its not possible. It can be done by just copying encoder but that’s memory and compute inefficient as its being done for each iteration.</p>
<pre><code class="lang-auto">latent = encoder(x)
x2 = decoder(x)
latent2 = clone(encoder)(x)
loss = F.mse_loss(x2, x) + F.mse_loss(latent2, latent)
loss.backward()
</code></pre>

username: Ouasfi
cooked: <p>Hi, you can detach a tensor from the computational graph with <code>latent2.detach()</code>. You can find more details <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2">here</a>.</p>
<p><code>loss = F.mse_loss(x2, x) + F.mse_loss(latent2.detach(), latent)</code></p>

username: ZdsAlpha
cooked: <p>This way decoder will not be optimized for latent loss. Only MSE loss will be used to optimize the decoder.<br>
Lets suppose I make a copy of encoder in each iteration. Here are computation graphs:<br>
x → encoder → latent<br>
x → encoder → decoder → x2<br>
x → encoder → decoder → encoder_copy → latent2<br>
Optimizer has only encoder and decoder parameters and not encoder_copy parameters. So the gradient will be calculated for parts that are inside square brackets:<br>
x → [encoder] → latent<br>
x → [encoder → decoder] → x2<br>
x → [encoder → decoder] → encoder_copy → latent2<br>
encoder_copy is not being optimized. I dont want to copy encoder in each iteration. But I will end up calculated gradient for encoder in second forward pass which I dont want (it will cause model to diverge). Here is what I end-up with<br>
x → [encoder] → latent<br>
x → [encoder → decoder] → x2<br>
x → [encoder → decoder → encoder] → latent2</p>
<p>But what I want:<br>
x → [encoder → decoder] → encoder → latent2</p>