Bias type Double

Salman_Abbasi: <p>Hi</p>
<p>I am running a model. The inputs are float32 but it shows the following error:<br>
“Input type (float) and bias type (double) should be the same”</p>
<p>The model is below:</p>
<p>class DnCNN(nn.Module):<br>
def <strong>init</strong>(self, depth=17, n_channels=64, image_channels=1, use_bnorm=True, kernel_size=3):<br>
super(DnCNN, self).<strong>init</strong>()<br>
kernel_size = 3<br>
padding = 1<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<pre><code>    layers.append(nn.Conv2d(in_channels=image_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=True))
    layers.append(nn.ReLU(inplace=True))
    for _ in range(depth-2):
        layers.append(nn.Conv2d(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=False))
        layers.append(nn.BatchNorm2d(n_channels, eps=0.0001, momentum = 0.95))
        layers.append(nn.ReLU(inplace=True))
    layers.append(nn.Conv2d(in_channels=n_channels, out_channels=image_channels, kernel_size=kernel_size, padding=padding, bias=False))
    self.dncnn = nn.Sequential(*layers)
    self._initialize_weights()
</code></pre>
<p>How to resolve this error</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

Salman_Abbasi: <p>thanks. I am trying to put in the code without giiving inputs. My inputs are float32, which I beleive is consistent to pytorch standards. I suspect the problem is in the intialize_weights function . But I could be wrong.</p>
<p>class Normalization:<br>
def <strong>init</strong>(self, mean_val=None,std_val=None):<br>
self.mean_val = mean_val<br>
self.std_val = std_val</p>
<pre><code>def normalize(self, x):
    return (x-self.mean_val)/self.std_val

def unnormalize(self, x):
    return x*self.std_val + self.mean_val
</code></pre>
<h1><a name="params-1" class="anchor" href="#params-1"></a>Params</h1>
<p>parser = argparse.ArgumentParser(description=‘PyTorch DnCNN’)<br>
parser.add_argument(‘–model’, default=‘DnCNN’, type=str, help=‘choose a type of model’)<br>
parser.add_argument(‘–data_dir’, default=‘data/train’, type=str, help=‘path of train data’)<br>
parser.add_argument(‘–sigma’, default=25, type=int, help=‘noise level’)<br>
parser.add_argument(‘–sigma2’, default=35, type=int, help=‘noise level2’)<br>
parser.add_argument(‘–epoch’, default=30, type=int, help=‘number of train epoches’)<br>
parser.add_argument(‘–lr’, default=1e-3, type=float, help=‘initial learning rate for Adam’)<br>
parser.add_argument(‘–batch_size’, default=5, type=int, help=‘batch size’)</p>
<h1><a name="parseradd_argument-patch_size-default110150-typeint-helppatch-size-2" class="anchor" href="#parseradd_argument-patch_size-default110150-typeint-helppatch-size-2"></a>parser.add_argument(‘–patch_size’, default=(1101,50), type=int, help=‘patch size’)</h1>
<p>parser.add_argument(‘–patch_size’, default=(3000,240), type=int, help=‘patch size’)<br>
parser.add_argument(‘–stride’, default=(32,32), type=int, help=‘the step size to slide on the data’)<br>
parser.add_argument(‘–jump’, default=3, type=int, help=‘the space between shot’)<br>
parser.add_argument(‘–download’, default=False, type=bool, help=‘if you will download the dataset from the internet’)<br>
parser.add_argument(‘–datasets’, default = 0, type = int, help=‘the num of datasets you want be download,if download = True’)</p>
<h1><a name="parseradd_argument-train_data_num-default201-typeint-helpthe-num-of-the-train_data-3" class="anchor" href="#parseradd_argument-train_data_num-default201-typeint-helpthe-num-of-the-train_data-3"></a>parser.add_argument(‘–train_data_num’, default=201, type=int, help=‘the num of the train_data’)</h1>
<p>parser.add_argument(‘–train_data_num’, default=4000, type=int, help=‘the num of the train_data’)<br>
parser.add_argument(‘–aug_times’, default=0, type=int, help=‘Number of aug operations’)<br>
parser.add_argument(‘–scales’, default=[1], type=list, help=‘data scaling’)<br>
parser.add_argument(‘–agc’, default=True, type=int, help=‘Normalize each trace by amplitude’)<br>
parser.add_argument(‘–verbose’, default=True, type=int, help=‘Whether to output the progress of data generation’)<br>
parser.add_argument(‘–display’, default=5, type=int, help=‘interval for displaying loss’)<br>
args = parser.parse_args()</p>
<p>batch_size = args.batch_size<br>
cuda = torch.cuda.is_available()<br>
torch.set_default_dtype(torch.float64)</p>
<p>n_epoch = args.epoch<br>
sigma = args.sigma<br>
sigma2 = args.sigma2</p>
<p>class DnCNN(nn.Module):<br>
def <strong>init</strong>(self, depth=17, n_channels=64, image_channels=1, use_bnorm=True, kernel_size=3):<br>
super(DnCNN, self).<strong>init</strong>()<br>
kernel_size = 3<br>
padding = 1<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<pre><code>    layers.append(nn.Conv2d(in_channels=image_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=True))
    layers.append(nn.ReLU(inplace=True))
    for _ in range(depth-2):
        layers.append(nn.Conv2d(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=False))
        layers.append(nn.BatchNorm2d(n_channels, eps=0.0001, momentum = 0.95))
        layers.append(nn.ReLU(inplace=True))
    layers.append(nn.Conv2d(in_channels=n_channels, out_channels=image_channels, kernel_size=kernel_size, padding=padding, bias=False))
    self.dncnn = nn.Sequential(*layers)
    self._initialize_weights()

def forward(self, x):
    y = x
    out = self.dncnn(x)
    return y-out

def _initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            init.orthogonal_(m.weight)
            
            if m.bias is not None:
                init.constant_(m.bias, 0)
        elif isinstance(m, nn.BatchNorm2d):
            init.constant_(m.weight, 1)
            init.constant_(m.bias, 0)
    print('init weight')
</code></pre>
<p>model = model.to(device)</p>
<p>optimizer = optim.Adam(model.parameters(), lr=args.lr)<br>
scheduler = MultiStepLR(optimizer, milestones=[30, 60, 90], gamma=0.2)  # learning rates</p>
<p>for epoch in range(initial_epoch, n_epoch):</p>
<pre><code>    scheduler.step(epoch)  # step to the learning rate in this epcoh
    DDataset = DenoisingDataset(xs, xs2)
    # DLoader = DataLoader(dataset=DDataset, num_workers=4, drop_last=True, batch_size=batch_size, shuffle=True)
    DLoader = DataLoader(dataset=DDataset, drop_last=True, batch_size=batch_size, shuffle=True)

    epoch_loss = 0
    start_time = time.time()

    for n_count, batch_yx in enumerate(DLoader):
            optimizer.zero_grad()
            if cuda:
                batch_y, batch_x = batch_yx[0].cuda(), batch_yx[1].cuda()
            else:
                batch_x, batch_y = batch_yx[0], batch_yx[1] # #batchx is odd batcg y even
            loss = criterion(model(batch_x), batch_y)
            epoch_loss += loss.item()
            loss.backward()
            optimizer.step()
</code></pre>

ptrblck: <p>I cannot reproduce the issue using:</p>
<pre data-code-wrap="python"><code class="lang-python">class DnCNN(nn.Module):
    def __init__(self, depth=17, n_channels=64, image_channels=1, use_bnorm=True, kernel_size=3):
        super(DnCNN, self).__init__()
        kernel_size = 3
        padding = 1
        layers = []
        
        layers.append(nn.Conv2d(in_channels=image_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=True))
        layers.append(nn.ReLU(inplace=True))
        for _ in range(depth-2):
            layers.append(nn.Conv2d(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=False))
            layers.append(nn.BatchNorm2d(n_channels, eps=0.0001, momentum = 0.95))
            layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Conv2d(in_channels=n_channels, out_channels=image_channels, kernel_size=kernel_size, padding=padding, bias=False))
        self.dncnn = nn.Sequential(*layers)
        self._initialize_weights()

    def forward(self, x):
        y = x
        out = self.dncnn(x)
        return y-out
    
    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                init.orthogonal_(m.weight)
                
                if m.bias is not None:
                    init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                init.constant_(m.weight, 1)
                
model = DnCNN()
for name, param in model.named_parameters():
    print("param: {},dtype: {}".format(name, param.dtype))

# param: dncnn.0.weight,dtype: torch.float32
# param: dncnn.0.bias,dtype: torch.float32
# param: dncnn.2.weight,dtype: torch.float32
# param: dncnn.3.weight,dtype: torch.float32
# param: dncnn.3.bias,dtype: torch.float32
# param: dncnn.5.weight,dtype: torch.float32
# param: dncnn.6.weight,dtype: torch.float32
# param: dncnn.6.bias,dtype: torch.float32
# param: dncnn.8.weight,dtype: torch.float32
# param: dncnn.9.weight,dtype: torch.float32
# param: dncnn.9.bias,dtype: torch.float32
# param: dncnn.11.weight,dtype: torch.float32
# param: dncnn.12.weight,dtype: torch.float32
# param: dncnn.12.bias,dtype: torch.float32
# param: dncnn.14.weight,dtype: torch.float32
# param: dncnn.15.weight,dtype: torch.float32
# param: dncnn.15.bias,dtype: torch.float32
# param: dncnn.17.weight,dtype: torch.float32
# param: dncnn.18.weight,dtype: torch.float32
# param: dncnn.18.bias,dtype: torch.float32
# param: dncnn.20.weight,dtype: torch.float32
# param: dncnn.21.weight,dtype: torch.float32
# param: dncnn.21.bias,dtype: torch.float32
# param: dncnn.23.weight,dtype: torch.float32
# param: dncnn.24.weight,dtype: torch.float32
# param: dncnn.24.bias,dtype: torch.float32
# param: dncnn.26.weight,dtype: torch.float32
# param: dncnn.27.weight,dtype: torch.float32
# param: dncnn.27.bias,dtype: torch.float32
# param: dncnn.29.weight,dtype: torch.float32
# param: dncnn.30.weight,dtype: torch.float32
# param: dncnn.30.bias,dtype: torch.float32
# param: dncnn.32.weight,dtype: torch.float32
# param: dncnn.33.weight,dtype: torch.float32
# param: dncnn.33.bias,dtype: torch.float32
# param: dncnn.35.weight,dtype: torch.float32
# param: dncnn.36.weight,dtype: torch.float32
# param: dncnn.36.bias,dtype: torch.float32
# param: dncnn.38.weight,dtype: torch.float32
# param: dncnn.39.weight,dtype: torch.float32
# param: dncnn.39.bias,dtype: torch.float32
# param: dncnn.41.weight,dtype: torch.float32
# param: dncnn.42.weight,dtype: torch.float32
# param: dncnn.42.bias,dtype: torch.float32
# param: dncnn.44.weight,dtype: torch.float32
# param: dncnn.45.weight,dtype: torch.float32
# param: dncnn.45.bias,dtype: torch.float32
# param: dncnn.47.weight,dtype: torch.float32
</code></pre>
<p>Your current code is not properly formatted so post a minimal and executable code snippet reproducing the issue in case you are stuck.</p>

Salman_Abbasi: <p>Thanks. Please see below. I hope this is reproduceable:</p>
<p>import argparse<br>
import re<br>
import os, glob, datetime, time<br>
os.environ[‘KMP_DUPLICATE_LIB_OK’]=‘True’<br>
os.environ[‘KMP_DUPLICATE_LIB_OK’]=‘True’<br>
os.environ[“CUDA_VISIBLE_DEVICES”] = “0”<br>
import numpy as np<br>
import torch<br>
import torch.nn as nn<br>
from torch.nn.modules.loss import _Loss<br>
import torch.nn.init as init<br>
from torch.utils.data import DataLoader<br>
import torch.optim as optim<br>
from torch.optim.lr_scheduler import MultiStepLR<br>
import multiprocessing<br>
import <a href="http://scipy.io" rel="noopener nofollow ugc">scipy.io</a><br>
from torch.utils.data import Dataset</p>
<p>class Normalization:<br>
def <strong>init</strong>(self, mean_val=None,std_val=None):<br>
self.mean_val = mean_val<br>
self.std_val = std_val</p>
<pre><code>def normalize(self, x):
    return (x-self.mean_val)/self.std_val

def unnormalize(self, x):
    return x*self.std_val + self.mean_val
</code></pre>
<p>parser = argparse.ArgumentParser(description=‘PyTorch DnCNN’)<br>
parser.add_argument(‘–model’, default=‘DnCNN’, type=str, help=‘choose a type of model’)<br>
parser.add_argument(‘–data_dir’, default=‘data/train’, type=str, help=‘path of train data’)<br>
parser.add_argument(‘–sigma’, default=25, type=int, help=‘noise level’)<br>
parser.add_argument(‘–sigma2’, default=35, type=int, help=‘noise level2’)<br>
parser.add_argument(‘–epoch’, default=30, type=int, help=‘number of train epoches’)<br>
parser.add_argument(‘–lr’, default=1e-3, type=float, help=‘initial learning rate for Adam’)<br>
parser.add_argument(‘–batch_size’, default=5, type=int, help=‘batch size’)<br>
parser.add_argument(‘–patch_size’, default=(3000,240), type=int, help=‘patch size’)<br>
parser.add_argument(‘–stride’, default=(32,32), type=int, help=‘the step size to slide on the data’)<br>
parser.add_argument(‘–jump’, default=3, type=int, help=‘the space between shot’)<br>
parser.add_argument(‘–download’, default=False, type=bool, help=‘if you will download the dataset from the internet’)<br>
parser.add_argument(‘–datasets’, default = 0, type = int, help=‘the num of datasets you want be download,if download = True’)<br>
parser.add_argument(‘–train_data_num’, default=4000, type=int, help=‘the num of the train_data’)<br>
parser.add_argument(‘–aug_times’, default=0, type=int, help=‘Number of aug operations’)<br>
parser.add_argument(‘–scales’, default=[1], type=list, help=‘data scaling’)<br>
parser.add_argument(‘–agc’, default=True, type=int, help=‘Normalize each trace by amplitude’)<br>
parser.add_argument(‘–verbose’, default=True, type=int, help=‘Whether to output the progress of data generation’)<br>
parser.add_argument(‘–display’, default=5, type=int, help=‘interval for displaying loss’)<br>
args = parser.parse_args()</p>
<p>batch_size = args.batch_size<br>
cuda = torch.cuda.is_available()<br>
torch.set_default_dtype(torch.float64)</p>
<p>n_epoch = args.epoch<br>
sigma = args.sigma<br>
sigma2 = args.sigma2</p>
<p>if not os.path.exists(‘models_denoise’):<br>
os.mkdir(‘models_denoise’)</p>
<p>save_dir = os.path.join(‘models_denoise’, args.model+‘_’ + ‘noNoise’ )</p>
<p>if not os.path.exists(save_dir):<br>
os.mkdir(save_dir)</p>
<p>class DenoisingDataset(Dataset):<br>
“”“Dataset wrapping tensors.<br>
Arguments:<br>
xs (Tensor): clean data patches<br>
sigma: noise level, e.g., 25<br>
“””<br>
def <strong>init</strong>(self, xs, xs2):<br>
super(DenoisingDataset, self).<strong>init</strong>()<br>
self.xs = xs<br>
self.xs2 = xs2</p>
<pre><code>def __getitem__(self, index):
    batch_x = self.xs[index]  
    batch_y=self.xs2[index]


    return batch_x, batch_y 

def __len__(self):
    return self.xs.size(0)
</code></pre>
<p>class DnCNN(nn.Module):<br>
def <strong>init</strong>(self, depth=17, n_channels=64, image_channels=1, use_bnorm=True, kernel_size=3):<br>
super(DnCNN, self).<strong>init</strong>()<br>
kernel_size = 3<br>
padding = 1<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<pre><code>    layers.append(nn.Conv2d(in_channels=image_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=True))
    layers.append(nn.ReLU(inplace=True))
    for _ in range(depth-2):
        layers.append(nn.Conv2d(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size, padding=padding, bias=False))
        layers.append(nn.BatchNorm2d(n_channels, eps=0.0001, momentum = 0.95))
        layers.append(nn.ReLU(inplace=True))
    layers.append(nn.Conv2d(in_channels=n_channels, out_channels=image_channels, kernel_size=kernel_size, padding=padding, bias=False))
    self.dncnn = nn.Sequential(*layers)
    self._initialize_weights()

def forward(self, x):
    y = x
    out = self.dncnn(x)
    return y-out

def _initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            init.orthogonal_(m.weight)
            
            if m.bias is not None:
                init.constant_(m.bias, 0)
        elif isinstance(m, nn.BatchNorm2d):
            init.constant_(m.weight, 1)
            init.constant_(m.bias, 0)
    print('init weight')
</code></pre>
<p>def findLastCheckpoint(save_dir):<br>
file_list = glob.glob(os.path.join(save_dir, 'model_<em>.pth’))<br>
if file_list:<br>
epochs_exist = []<br>
for file_ in file_list:<br>
result = re.findall(".<em>model_(.</em>).pth.</em>", file_)<br>
epochs_exist.append(int(result[0]))<br>
initial_epoch = max(epochs_exist)<br>
else:<br>
initial_epoch = 0<br>
return initial_epoch</p>
<p>def log(*args, **kwargs):<br>
print(datetime.datetime.now().strftime(“%Y-%m-%d %H:%M:%S:”), *args, **kwargs)</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
print(‘===&gt; Building model’)<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
model = DnCNN()</p>
<pre><code>initial_epoch = findLastCheckpoint(save_dir=save_dir)  # load the last model in matconvnet style
if initial_epoch &gt; 1:
    print('resuming by loading epoch %03d\n' % (initial_epoch-1))
    if initial_epoch &gt;= n_epoch:
        print("training have finished")

    else:
        model = torch.load(os.path.join(save_dir, 'model_%03d.pth' % (initial_epoch-1)))
model.train()
criterion = nn.MSELoss(reduce = True,size_average = False)


if cuda:
    model = model.to(device)

optimizer = optim.Adam(model.parameters(), lr=args.lr)
scheduler = MultiStepLR(optimizer, milestones=[30, 60, 90], gamma=0.2)  # learning rates



xss=np.random.random((400,3000,240)).astype(np.float32)
xss2=np.random.random((400,3000,240)).astype(np.float32)

xs=xss.transpose(0,2,1)    
xs_mean= torch.tensor(np.mean(xs,axis=(0,-1),keepdims=True)).to(device)
xs_std=torch.tensor(np.std(xs,axis=(0,-1),keepdims=True)).to(device)
xs_normalization = Normalization(mean_val=xs_mean,std_val=xs_std)
xs=torch.tensor(xs).to(device)
xs=xs_normalization.normalize(xs)
xs=torch.unsqueeze(xs,1)


xs2=xss2.transpose(0,2,1)    
xs2_mean= torch.tensor(np.mean(xs2,axis=(0,-1),keepdims=True)).to(device)
xs2_std=torch.tensor(np.std(xs2,axis=(0,-1),keepdims=True)).to(device)
xs2_normalization = Normalization(mean_val=xs2_mean,std_val=xs2_std)
xs2=torch.tensor(xs2).to(device)
xs2=xs2_normalization.normalize(xs2)
xs2=torch.unsqueeze(xs2,1)




for epoch in range(initial_epoch, n_epoch):

    scheduler.step(epoch)  # step to the learning rate in this epcoh
    DDataset = DenoisingDataset(xs, xs2)
    DLoader = DataLoader(dataset=DDataset, drop_last=True, batch_size=batch_size, shuffle=True)

    epoch_loss = 0
    start_time = time.time()

    for n_count, batch_yx in enumerate(DLoader):
            optimizer.zero_grad()
            if cuda:
                batch_y, batch_x = batch_yx[0].cuda(), batch_yx[1].cuda()
            else:
                batch_x, batch_y = batch_yx[0], batch_yx[1] # #batchx is odd batcg y even
            loss = criterion(model(batch_x), batch_y)
            epoch_loss += loss.item()
            loss.backward()
            optimizer.step()
</code></pre>

Salman_Abbasi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hi</p>
<p>is there a chance to look at the code.<br>
thanks</p>

Salman_Abbasi: <p>For someone facing error message. I just had to put my model on float.</p>
<p>Like model.float()</p>
<p><span class="mention">@ptrblock</span>  following response helped me. thanks</p>
<aside class="quote" data-post="3" data-topic="126411">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nathb26/48/33816_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-input-type-torch-cuda-floattensor-and-weight-type-torch-cuda-doubletensor-should-be-the-same/126411/3">RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.DoubleTensor) should be the same</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Thank you ! I converted every input into float and it worked
  </blockquote>
</aside>