Runtime error in torch.cuda.FloatTensor and weight type torch.Float.Tensor should be same

username: RANJITH
cooked: <p>hey, guys</p>
<p>Actually I’m facing the this problem, i go through the previous solution but is could not resolve, I implementing the <strong>device</strong> but is still showing the error</p>
<p>here my full error report<br>
{<br>
“name”: “RuntimeError”,<br>
“message”: “Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same”,<br>
“stack”: "---------------------------------------------------------------------------<br>
RuntimeError                              Traceback (most recent call last)<br>
Cell In[35], line 25<br>
22 parameter = Parameter(budget=budget)<br>
24 # Run ZOOpt<br>
—&gt; 25 solution = Opt.min(objective, parameter)<br>
26 best_params = solution.get_x()  # Extract the best parameters<br>
27 best_learning_rate = best_params[0]</p>
<p>File ~/.local/lib/python3.11/site-packages/zoopt/opt.py:50, in Opt.min(objective, parameter)<br>
48     result = sre.opt()<br>
49 else:<br>
—&gt; 50     result = optimizer.opt(objective, parameter)<br>
51 result.print_solution()<br>
52 return result</p>
<p>File ~/.local/lib/python3.11/site-packages/zoopt/algos/opt_algorithms/racos/racos_optimization.py:57, in RacosOptimization.opt(self, objective, parameter, strategy)<br>
55     else:<br>
56         self.__algorithm = SRacos()<br>
—&gt; 57     self.__best_solution = self.__algorithm.opt(<br>
58             objective, parameter, strategy, ub)<br>
59 else:<br>
60     self.__algorithm = Racos()</p>
<p>File ~/.local/lib/python3.11/site-packages/zoopt/algos/opt_algorithms/racos/sracos.py:47, in SRacos.opt(self, objective, parameter, strategy, ub)<br>
45 self.set_objective(objective)<br>
46 self.set_parameters(parameter)<br>
—&gt; 47 self.init_attribute()<br>
48 stopping_criterion = self._parameter.get_stopping_criterion()<br>
49 i = 0</p>
<p>File ~/.local/lib/python3.11/site-packages/zoopt/algos/opt_algorithms/racos/racos_common.py:92, in RacosCommon.init_attribute(self)<br>
90     break<br>
91 if distinct_flag:<br>
—&gt; 92     self._objective.eval(x)<br>
93     self._data.append(x)<br>
94     i += 1</p>
<p>File ~/.local/lib/python3.11/site-packages/zoopt/objective.py:83, in Objective.eval(self, solution)<br>
81 for i in range(self.__resample_times):<br>
82     if self.__reducedim is False:<br>
—&gt; 83         val = self.__func(solution)<br>
84     else:<br>
85         x = solution.get_x()</p>
<p>Cell In[35], line 9, in objective_function(solution)<br>
6 num_layers = int(params[1])<br>
7 dropout_rate = params[2]<br>
----&gt; 9 train_losses, val_losses, val_accuracies, _ = train_model(num_layers, dropout_rate, learning_rate)<br>
10 return -val_accuracies[-1]</p>
<p>Cell In[33], line 3, in train_model(num_layers, dropout_rate, learning_rate)<br>
2 def train_model(num_layers, dropout_rate, learning_rate):<br>
----&gt; 3     model = CNN(num_layers, dropout_rate).to(device)<br>
5     # Print the model summary<br>
6     print("Model Summary:")</p>
<p>Cell In[30], line 18, in CNN.<strong>init</strong>(self, num_layers, dropout_rate)<br>
15     out_channels *= 2<br>
17 # Initialize flatten_size dynamically<br>
—&gt; 18 self.flatten_size = self._calculate_flatten_size()<br>
19 self.fc1 = nn.Linear(self.flatten_size, 256)<br>
20 self.fc2 = nn.Linear(256, len(dataset.classes))  # Ensure dataset.classes is defined</p>
<p>Cell In[30], line 28, in CNN._calculate_flatten_size(self)<br>
26 x = dummy_input<br>
27 for layer in self.layers:<br>
—&gt; 28     x = layer(x)<br>
29 return int(torch.prod(torch.tensor(x.size())))</p>
<p>File ~/.local/lib/python3.11/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)<br>
1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]<br>
1531 else:<br>
 → 1532     return self._call_impl(*args, **kwargs)</p>
<p>File ~/.local/lib/python3.11/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)<br>
1536 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1537 # this function, and just call forward.<br>
1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1539         or _global_backward_pre_hooks or _global_backward_hooks<br>
1540         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1541     return forward_call(*args, **kwargs)<br>
1543 try:<br>
1544     result = None</p>
<p>File ~/.local/lib/python3.11/site-packages/torch/nn/modules/conv.py:460, in Conv2d.forward(self, input)<br>
459 def forward(self, input: Tensor) → Tensor:<br>
 → 460     return self._conv_forward(input, self.weight, self.bias)</p>
<p>File ~/.local/lib/python3.11/site-packages/torch/nn/modules/conv.py:456, in Conv2d._conv_forward(self, input, weight, bias)<br>
452 if self.padding_mode != ‘zeros’:<br>
453     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),<br>
454                     weight, bias, self.stride,<br>
455                     _pair(0), self.dilation, self.groups)<br>
 → 456 return F.conv2d(input, weight, bias, self.stride,<br>
457                 self.padding, self.dilation, self.groups)</p>
<p>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same"<br>
}</p>

username: Soumya_Kundu
cooked: <p>You would need to share some code as to understand where this is happening.</p>

username: RANJITH
cooked: <p>class CNN(nn.Module):<br>
def <strong>init</strong>(self, num_layers, dropout_rate):<br>
super(CNN, self).<strong>init</strong>()<br>
self.layers = nn.ModuleList()<br>
in_channels = 3<br>
out_channels = 32<br>
kernel_size = 3<br>
padding = 1</p>
<pre><code>    for _ in range(num_layers):
        self.layers.append(nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=padding))
        self.layers.append(nn.ReLU())
        self.layers.append(nn.MaxPool2d(2, 2))
        in_channels = out_channels
        out_channels *= 2
    
    # Initialize flatten_size dynamically
    self.flatten_size = self._calculate_flatten_size()
    self.fc1 = nn.Linear(self.flatten_size, 256)
    self.fc2 = nn.Linear(256, len(dataset.classes))  # Ensure dataset.classes is defined
    self.dropout = nn.Dropout(p=dropout_rate)

def _calculate_flatten_size(self):
    # Dummy input to calculate flatten_size
    dummy_input = torch.zeros(1, 3, 32, 32).to(device)
    x = dummy_input
    for layer in self.layers:
        x = layer(x)
    return int(torch.prod(torch.tensor(x.size())))

def forward(self, x):
    for layer in self.layers:
        x = layer(x)
    x = x.view(-1, self.flatten_size)
    x = F.relu(self.fc1(x))
    x = self.dropout(x)
    x = self.fc2(x)
    return x
</code></pre>
<h1><a name="p-453697-training-function-1" class="anchor" href="#p-453697-training-function-1"></a>Training function</h1>
<p>def train_model(num_layers, dropout_rate, learning_rate):<br>
model = CNN(num_layers, dropout_rate).to(device)</p>
<pre><code># Print the model summary
print("Model Summary:")
summary(model, input_size=(3, 32, 32))

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
num_epochs = 20

train_losses = []
val_losses = []
val_accuracies = []

for epoch in range(num_epochs):
    model.train()
    running_loss = 0.0
    correct = 0
    total = 0
    
    for images, labels in train_loaders:
        # Move images and labels to the correct device
        images, labels = images.to(device), labels.to(device)
        
        optimizer.zero_grad()
        outputs = model(images)  # Forward pass

        # Ensure the shapes are correct
        if outputs.size(0) != labels.size(0):
            raise ValueError(f'Batch size mismatch: outputs.size(0)={outputs.size(0)}, labels.size(0)={labels.size(0)}')

        loss = criterion(outputs, labels)  # Calculate loss
        loss.backward()  # Backpropagation
        optimizer.step()  # Update weights
        
        running_loss += loss.item()
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    train_losses.append(running_loss / len(train_loaders))

    # Validation phase
    model.eval()
    val_loss = 0.0
    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in val_loaders:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)

            # Ensure the shapes are correct
            if outputs.size(0) != labels.size(0):
                raise ValueError(f'Batch size mismatch: outputs.size(0)={outputs.size(0)}, labels.size(0)={labels.size(0)}')

            loss = criterion(outputs, labels)
            val_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    val_losses.append(val_loss / len(val_loaders))
    val_accuracies.append(correct / total)

return train_losses, val_losses, val_accuracies, model
</code></pre>
<h1><a name="p-453697-zoopt-objective-function-2" class="anchor" href="#p-453697-zoopt-objective-function-2"></a>ZOOpt objective function</h1>
<p>def objective_function(solution):<br>
# Convert ZOOpt Solution object to list of parameters<br>
params = solution.get_x()<br>
learning_rate = params[0]<br>
num_layers = int(params[1])<br>
dropout_rate = params[2]</p>
<pre><code>train_losses, val_losses, val_accuracies, _ = train_model(num_layers, dropout_rate, learning_rate)
return -val_accuracies[-1]  # Return negative accuracy for minimization
</code></pre>
<h1><a name="p-453697-zoopt-configuration-3" class="anchor" href="#p-453697-zoopt-configuration-3"></a>ZOOpt configuration</h1>
<p>dim = Dimension(<br>
3,<br>
[[1e-5, 1e-2],  # learning_rate<br>
[1, 5],        # num_layers<br>
[0.1, 0.5]],   # dropout_rate<br>
[True, True, True]<br>
)<br>
objective = Objective(objective_function, dim)<br>
budget = 20  # Number of function evaluations<br>
parameter = Parameter(budget=budget)</p>
<h1><a name="p-453697-run-zoopt-4" class="anchor" href="#p-453697-run-zoopt-4"></a>Run ZOOpt</h1>
<p>solution = Opt.min(objective, parameter)<br>
best_params = solution.get_x()  # Extract the best parameters<br>
best_learning_rate = best_params[0]<br>
best_num_layers = int(best_params[1])<br>
best_dropout_rate = best_params[2]</p>
<p>print(“Best parameters found by ZOOpt:”)<br>
print(“Learning rate:”, best_learning_rate)<br>
print(“Number of layers:”, best_num_layers)<br>
print(“Dropout rate:”, best_dropout_rate)</p>
<p><span class="hashtag-raw">#Final</span> training with the best parameters found by ZOOpt<br>
train_losses, val_losses, val_accuracies, final_model = train_model(best_num_layers, best_dropout_rate, best_learning_rate)</p>
<p>“here i try to implementing derivative free optimization called zoopt  why run the script is showing the error”  device = ‘cuda’</p>

username: Soumya_Kundu
cooked: <p>Haven’t gone through your code in depth, but have you tried going through each line where it needs to be sent to a device?</p>