AttributeError: 'tuple' object has no attribute 'dim' error Transfer learning inception_v3

Ayshine: <p>I’m trying to classify my images using transfer learning with inception_v3 and having an error<br>
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM<br>
What would be the reason of this error ?<br>
My transform</p>
<pre><code class="lang-auto">
transform = transforms.Compose([ transforms.CenterCrop(1000), transforms.Resize((299,299)),
                                 transforms.ToTensor(),
                                 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

</code></pre>
<p>My model implementation</p>
<pre><code class="lang-auto"># Use GPU if it's available
from collections import OrderedDict
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = models.inception_v3(pretrained=True)

# Freeze parameters so we don't backprop through them
for param in model.parameters():
    param.requires_grad = False
    
classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(2048, 500)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(500, 2)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))
    
model.classifier = classifier

criterion = nn.NLLLoss()

# Only train the classifier parameters, feature parameters are frozen
optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

model.to(device);
</code></pre>
<p>and the full error I am having</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-11-912dafb06827&gt; in &lt;module&gt;
     12 
     13         logps = model.forward(inputs)
---&gt; 14         loss = criterion(logps, labels)
     15         loss.backward()
     16         optimizer.step()

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    208     @weak_script_method
    209     def forward(self, input, target):
--&gt; 210         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
    211 
    212 

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1780     if size_average is not None or reduce is not None:
   1781         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 1782     dim = input.dim()
   1783     if dim &lt; 2:
   1784         raise ValueError('Expected 2 or more dimensions (got {})'.format(dim))

AttributeError: 'tuple' object has no attribute 'dim'
</code></pre>

ptrblck: <p>Could you check your <code>input</code> in the training loop?<br>
Based on the error message, it looks like you are trying to pass a <code>tuple</code> instead of a data <code>tensor</code>.<br>
This might be the case, if you used just one return value for the data and target:</p>
<pre><code class="lang-python">for batch in loader:
    data = batch[0]
    target = batch[1]
    ...
</code></pre>
<p>In this example this error would be thrown, if you try to pass <code>batch</code> directly to the model instead of <code>data</code>.</p>

Ayshine: <p>Isn’t this line doing the same thing  <code>for inputs, labels in train_loader:</code> inputs as data and labels as target ?</p>

ptrblck: <p>Yes, should do the same.<br>
Could you post your <code>Dataset</code> definition and training loop so that we could have a look?</p>

Ayshine: <p>Data</p>
<pre><code class="lang-auto">num_workers = 0
# how many samples per batch to load
batch_size = 20
# percentage of data set to use as test
test_size = 0.3

transform = transforms.Compose([ transforms.CenterCrop(1000), transforms.Resize((299,299)),
                                 transforms.ToTensor(),
                                 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

data_set = dset.ImageFolder(root="data",transform=transform)
dataloader = torch.utils.data.DataLoader(data_set, batch_size=4,shuffle=True,num_workers=2)

# obtain training indices that will be used for test
num_data = len(data_set)
indices = list(range(num_data))
np.random.shuffle(indices)
split = int(np.floor(test_size * num_data))
train_idx, test_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
test_sampler  = SubsetRandomSampler(test_idx)

# prepare data loaders
train_loader = torch.utils.data.DataLoader(data_set, batch_size=batch_size,
                                           sampler = train_sampler, num_workers=num_workers)
test_loader  = torch.utils.data.DataLoader(data_set, batch_size=batch_size, 
                                           sampler = test_sampler, num_workers=num_workers)

classes = ('ebrus','suminagashis')
</code></pre>
<p>Training loop</p>
<pre><code class="lang-auto">epochs = 5
steps = 0
running_loss = 0
print_every = 5
for epoch in range(epochs):
    for inputs, labels in train_loader:
        steps += 1
        # Move input and label tensors to the default device
        inputs, labels = inputs.to(device), labels.to(device)
        
        optimizer.zero_grad()
        
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
#         loss.backward()
        optimizer.step()

        running_loss += loss.item()
        

    test_loss = 0
    accuracy = 0
    model.eval()
    with torch.no_grad():
         for inputs, labels in test_loader:
            inputs, labels = inputs.to(device), labels.to(device)
            logps = model.forward(inputs)
            batch_loss = criterion(logps, labels)
                    
            test_loss += batch_loss.item()
                    
                    # Calculate accuracy
            ps = torch.exp(logps)
            top_p, top_class = ps.topk(1, dim=1)
            equals = top_class == labels.view(*top_class.shape)
            accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
                    
    print(f"Epoch {epoch+1}/{epochs}.. "
           f"Train loss: {running_loss:.3f}.. "
           f"Test loss: {test_loss/len(test_loader):.3f}.. "
           f"Test accuracy: {accuracy/len(test_loader):.3f}")
    running_loss = 0
    model.train()
</code></pre>

ptrblck: <p>Thanks for the code. It looks alright, so could you add these print statements into the training loop:</p>
<pre><code class="lang-python">inputs, labels = inputs.to(device), labels.to(device)
print(type(inputs))
print(type(labels))
print(inputs.size())
print(inputs.dim())
</code></pre>

Ayshine: <p>Getting this output<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5fd0a9cf365fdf66e4896c9b78556b8afeec699b.png" alt="image" data-base62-sha1="dFCpXPkE93Tce8hry5obwhfchL5" width="523" height="251"></p>

ptrblck: <p>Thanks for the debugging.<br>
It seems the error is related to the output of your model.<br>
By default the inception model returns two outputs, the class logits and the <code>aux_loss</code>.<br>
This is explained in the original inception paper.<br>
In case you don’t need the auxiliary loss, you can create the model using <code>aux_logits=False</code>:</p>
<pre><code class="lang-python">model = models.inception_v3(pretrained=True, aux_logits=False)
</code></pre>

Ayshine: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="36334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>aux_logits=False)</p>
</blockquote>
</aside>
<p>this is what I get when I add aux_logits = False</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-16-ffd4a44975ff&gt; in &lt;module&gt;
      3 device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
      4 
----&gt; 5 model = models.inception_v3(pretrained=True,aux_logits=False)
      6 
      7 # Freeze parameters so we don't backprop through them

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torchvision/models/inception.py in inception_v3(pretrained, **kwargs)
     25             kwargs['transform_input'] = True
     26         model = Inception3(**kwargs)
---&gt; 27         model.load_state_dict(model_zoo.load_url(model_urls['inception_v3_google']))
     28         return model
     29 

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    767         if len(error_msgs) &gt; 0:
    768             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--&gt; 769                                self.__class__.__name__, "\n\t".join(error_msgs)))
    770 
    771     def _named_members(self, get_members_fn, prefix='', recurse=True):

RuntimeError: Error(s) in loading state_dict for Inception3:
	Unexpected key(s) in state_dict: "AuxLogits.conv0.conv.weight", "AuxLogits.conv0.bn.weight", "AuxLogits.conv0.bn.bias", "AuxLogits.conv0.bn.running_mean", "AuxLogits.conv0.bn.running_var", "AuxLogits.conv1.conv.weight", "AuxLogits.conv1.bn.weight", "AuxLogits.conv1.bn.bias", "AuxLogits.conv1.bn.running_mean", "AuxLogits.conv1.bn.running_var", "AuxLogits.fc.weight", "AuxLogits.fc.bias".
</code></pre>

Ayshine: <p>the overall goal I have is to choose latest pretrained models and use them on my dataset and compare all of them. Looks like this tutorial covers what I am trying to do usehttps://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html<br>
except I want to use  the best performing version of the model for the models having more than one version. Looks like we can also cite this webpage on our paper. I’ll reimplement all transfer learning section then come back here and update this thread. Hopefully find a solution.</p>

ptrblck: <p>In that case could you try to pass <code>logps[0]</code> to your loss function? The first entry should be the class logits while the second one should be the <code>aux_logits</code>.</p>

Ayshine: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="36334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>logps[0]</p>
</blockquote>
</aside>
<p>Now having this <code> RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code> this means inception_v3 does not have a backprop function right?</p>

Ayshine: <p>getting this error now</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
&lt;ipython-input-18-bac6a09fefde&gt; in &lt;module&gt;
     26             inputs, labels = inputs.to(device), labels.to(device)
     27             logps = model.forward(inputs)
---&gt; 28             batch_loss = criterion(logps[0], labels)
     29 
     30             test_loss += batch_loss.item()

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    208     @weak_script_method
    209     def forward(self, input, target):
--&gt; 210         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
    211 
    212 

~/anaconda3/envs/pytorch10/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1782     dim = input.dim()
   1783     if dim &lt; 2:
-&gt; 1784         raise ValueError('Expected 2 or more dimensions (got {})'.format(dim))
   1785 
   1786     if input.size(0) != target.size(0):

ValueError: Expected 2 or more dimensions (got 1)
</code></pre>

ptrblck: <p>Could you print the shape out <code>logps[0]</code>? It should be <code>[batch_size, nb_classes]</code>.</p>
<p>I also just realized, that you are assigning your <code>Sequential</code> classifier module to <code>model.classifier</code>.<br>
If you are using <code>inception_v3</code>, you should use <code>model.fc</code> instead.</p>
<p>Here is a minimal code snippet which should work:</p>
<pre><code class="lang-python">model = models.inception_v3(pretrained=True, aux_logits=True)
criterion = nn.CrossEntropyLoss()

x = torch.randn(2, 3, 299, 299)
target = torch.empty(2, dtype=torch.long).random_(1000)

output = model(x)
loss = criterion(output[0], target)
loss.backward()

print(model.fc.weight.grad)
</code></pre>
<p>Could you try to compare your code to this snippet?</p>

Ayshine: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Thanks a lot for the help! Now my model is training. Here are the changes I made after your last reply:</p>
<ul>
<li>changed <code>model.classifier</code> to <code>model.fc</code> (Is this information included in pytorch documentation?)</li>
</ul>
<pre><code class="lang-auto">fc = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(2048, 500)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(500, 2)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))
    
model.fc = fc
</code></pre>
<ul>
<li>
<p>Changed my criterion from <code>nn.NLLLoss()</code> to <code>nn.CrossEntropyLoss()</code> (I don’t think this is relevant but I’ll check loss documentation)</p>
</li>
<li>
<p>Changed <code>loss = criterion(logps, labels)</code> to <code>loss = criterion(logps[0], labels)</code></p>
</li>
</ul>
<p>First and the third changes according to your suggestions helped to start training the network.</p>

ptrblck: <p>Hi Aysin,<br>
I’m glad it’s working now!</p>
<ul>
<li>You would have to print the model or look at the source to see all attributes.</li>
<li>If you are using <code>nn.LogSoftmax</code>, you should stick to <code>nn.NLLLoss</code>, as it’ll be applied twice otherwise.</li>
</ul>

rajasekhar: <p>From <a href="http://arxiv.org/abs/1409.4842" rel="noopener nofollow ugc">the paper</a> (page 6):</p>
<blockquote>
<p>During training, their loss gets added to the total loss of the network with a discount weight (the losses of the auxiliary classifiers were weighted by 0.3).<br>
~ <a href="https://github.com/BVLC/caffe/pull/1367" class="inline-onebox" rel="noopener nofollow ugc">GoogLeNet training in Caffe by amiralush · Pull Request #1367 · BVLC/caffe · GitHub</a></p>
</blockquote>
<p>Can we do similar to that in PyTorch?</p>

ptrblck: <p>Sure! You would have to set <code>aux_logits=True</code> when initializing the Inception model. In your training loop you will get two outputs, the output of the last linear layer and the auxiliary logits. After calculating the losses, you can weight and sum them to get the final loss.</p>

Ahmad_Altameemi: <p>hello<br>
I got same error<br>
AttributeError: ‘tuple’ object has no attribute ‘dim’<br>
print(‘Starting’)<br>
print(‘Running for’,epochs,‘Epochs’)<br>
<span class="hashtag">#print</span>(‘Batch Size:’, mnist.trainloader.batch_size)<br>
<span class="hashtag">#n_of_batches</span> = int(len(mnist.train_data.data)/mnist.trainloader.batch_size)<br>
<span class="hashtag">#print</span>(‘No of batches per epoch:’, n_of_batches)<br>
losses = []<br>
test_acc = []<br>
for epoch in range(epochs):<br>
loss2 = []<br>
for i,data in enumerate(zip(train_loader, train_labels),0):<br>
pcae.train()<br>
inputs, labels = data<br>
inputs = torch.tensor(inputs, dtype = torch.float32)<br>
labels = torch.tensor(labels, dtype = torch.float32)<br>
inputs, labels = inputs.to(‘cuda’), labels.to(‘cuda’)<br>
optimizer.zero_grad()<br>
out = pcae(inputs)<br>
loss = loss_func(out)<br>
loss.mean().backward()<br>
optimizer.step()<br>
loss2.append(loss.mean().item())<br>
# print("\rEpoch <span class="hashtag">#:</span>",epoch+1, "  Batch <span class="hashtag">#:</span>", i+1, “\tImage Likelihood:”,"{0:.2f}".format(-img_lik.item()),<br>
#       “\tPart Likelihood:”,"{0:.2f}".format(-part_lik.item()), “\tPrior Sparsity:”,"{0:.2f}".format(prior_sparsity.item()),<br>
#       “\tPosterior Sparsity:”,"{0:.2f}".format(posterior_sparsity.item()) end="")<br>
print("\rEpoch <span class="hashtag">#:</span>",epoch+1, "  Batch <span class="hashtag">#:</span>", i+1, “\tImage Likelihood:”,"{0:.2f}".format(loss.mean().item()), end="")<br>
l = np.sum(loss2)/(len(loss2))<br>
correct = 0<br>
total = 0<br>
with torch.no_grad():<br>
for data in zip(train_loader, train_labels) :<br>
pcae.eval()<br>
inputs, labels = data<br>
inputs = torch.tensor(inputs, dtype = torch.float32)<br>
labels = torch.tensor(labels, dtype = torch.float32)<br>
inputs, labels = inputs.to(‘cuda’), labels.to(‘cuda’)<br>
outs= pcae(inputs)<br>
norm = torch.norm(outs, dim=2)<br>
predicted = np.argmax(norm.cpu().numpy(),axis=1)<br>
total += labels.size(0)<br>
correct += (predicted == labels.cuda().numpy()).sum().item()<br>
acc = (100 * correct / total)<br>
print(’\rEpoch <span class="hashtag">#:</span>’, epoch+1, ’  Training Loss: ',"{0:.2f}".format(l))<br>
print(‘CapsNet Testing Accuracy on MNIST: ‘,acc, ‘%’)<br>
test_acc.append(acc)<br>
if max(test_acc)==acc:<br>
<span class="hashtag">#torch</span>.save(capsnet.state_dict(), ‘gdrive/My Drive/capsnet_40_epochs.pth’)<br>
losses.append(l)<br>
print(’\rEpoch <span class="hashtag">#:</span>’, epoch+1, ’  Training Loss: ',"{0:.2f}".format(l))<br>
losses.append(l)<br>
<span class="hashtag">#torch</span>.save(pcae.state_dict(), “./gdrive/My Drive/trained_pcae_2.pth”)<br>
print(‘Finished’)</p>

ptrblck: <p>Are the previously mentioned solutions not working for you?<br>
PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier.</p>