Tensors of the same index must be on the same device and the same dtype except `step` tensors that can be CPU and float32 notwithstanding

username: gengxingze
cooked: <p>for example.</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn as nn

@torch.jit.interface
class ModuleInterface(torch.nn.Module):
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        pass
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.net = nn.Sequential(
            nn.Linear(in_features=200, out_features=1, bias=True)
        )
        self.net1 = nn.Parameter(torch.Tensor(1, 200))

    def forward(self, feat):
        feat.requires_grad_()
        m = torch.rand(1, 200).to("cuda")
        b = feat * m
        return b

class SANNet(nn.Module):
    def __init__(self):
        super(SANNet, self).__init__()

        self.net = nn.Sequential(
            nn.Linear(in_features=200, out_features=1, bias=True)
        )
        self.net1 = nn.Parameter(torch.Tensor(1, 200))
        self.net2 = nn.ModuleList()
        self.net2.append(Net())
    def forward(self, feat):
        feat.requires_grad_()
        m = torch.rand(1, 200).to("cuda")
        net:ModuleInterface = self.net[0]
        c = net(m)
        b = feat * c
        return b
torch.set_default_dtype(torch.float64)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = SANNet()
model = model.to(device)
model.train()
opt = torch.optim.Adam(model.parameters(),lr=0.001)
f = nn.MSELoss()
aa = torch.rand(1,200)
data = torch.utils.data.DataLoader(aa, batch_size=1, shuffle=True, num_workers=1)
for i,x in enumerate(data) :
    x = torch.rand(1, 200).to(device)
    y = model(x)
    loss = f(x, y)/10.0
    loss.backward()
    opt.step()
    opt.zero_grad()

</code></pre>
<p>when I set torch.set_default_dtype(torch.float32),it can work.<br>
but I set torch.set_default_dtype(torch.float64),it will report an error</p>
<pre><code class="lang-auto">Tensors of the same index must be on the same device and the same dtype except `step` tensors that can be CPU and float32 notwithstanding
</code></pre>
<p>I can identify the location of the problem as</p>
<pre><code class="lang-auto">net:ModuleInterface = self.net[0]
c = net(m)
</code></pre>

username: soulitzer
cooked: <p>Thanks for the report, this seems like a bug. I’ve simplified your repro and filed an issue here <a href="https://github.com/pytorch/pytorch/issues/111671" class="inline-onebox" rel="noopener nofollow ugc">Foreach optimizers don't work with torch.set_default_dtype(torch.float64) · Issue #111671 · pytorch/pytorch · GitHub</a>.</p>
<p>Is there any particular reason you need to use <code>torch.float64</code> for your case?</p>

username: DavidVelazco26
cooked: <p>Hello fiend:<br>
I’m having the same problem, do you have idea what is the problem in the bug?<br>
RuntimeError: Tensors of the same index must be on the same device and the same dtype except <code>step</code> tensors that can be CPU and float32 notwithstanding</p>
<p>Thanks!</p>

username: soulitzer
cooked: <p>Hi,<br>
Looks like someone has signed up to work on it, so there should be a fix in a nightly sometime soon, but some workarounds today are:</p>
<ul>
<li>use foreach=False with optimizer (you’d be missing out on performance here)</li>
<li>avoid torch.set_default_dtype(torch.float64)</li>
</ul>

username: gengxingze
cooked: <p>Thank you very much for your attention to this issue</p>

username: gengxingze
cooked: <p>One possible solution is not to use torch.set_default_dtype(torch.float64),But we can explicitly specify the type in the network</p>
<pre><code class="lang-auto">model.double
m = torch.rand(1,200).to(device).to(torch.float64)
</code></pre>
<p>It gets verbose, but at least you can train using double</p>

username: Judah_Goldfeder
cooked: <p>Is this fixed yet in the new build? I am still getting an error.</p>

username: gengxingze
cooked: <p>A more verbose solution would be to manually display specify all types and devices</p>

username: edwardmilsom
cooked: <p>Any news on this issue? I’m still getting this error in 2.1.1.</p>

username: mmitjans
cooked: <p>Same issue here, but I do have <code>torch.set_default_dtype(torch.float32)</code> and still getting the same exact issue.</p>

username: amisu
cooked: <p>Same issue here for me. I also do have <code>torch.set_default_dtype(torch.float32)</code> set but still getting <code>RuntimeError: Tensors of the same index must be on the same device and the same dtype except 'step' tensors that can be CPU and float32 notwithstanding </code><br>
The torch version output when doing <code>python -c "import torch; print(torch.__version__)"</code> shows <code>2.1.2+cu118</code>. Any idea how to fix this? Currently only setting <code>foreach=False</code> inside the optimizer like this <code>self.train_op = torch.optim.Adam(model.parameters(), lr=self.learning_rate, foreach=False)</code> stops the error. However, above it is being suggested not to use <code>foreach=False</code>.</p>