Can not able to load inputs and labels to GPU

IamSparky: <p>firstly i added data generators for train and val<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dfe20d409e2f3e1d2aa57c39a989ce035d603d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/9dfe20d409e2f3e1d2aa57c39a989ce035d603d0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dfe20d409e2f3e1d2aa57c39a989ce035d603d0.png" alt="image" data-base62-sha1="mxFqFhq0UsKZHNGZqZgdOFkiqMU" width="457" height="500" data-dominant-color="EEF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">464×507 26.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>then added data loaders<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ada73dff71402177b557e12be069287a0d9c6123.png" data-download-href="https://discuss.pytorch.org/uploads/default/ada73dff71402177b557e12be069287a0d9c6123" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ada73dff71402177b557e12be069287a0d9c6123.png" alt="image" data-base62-sha1="oMcURltC6D5Mqhe3lF5JmbszSgz" width="690" height="126" data-dominant-color="EEF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×132 9.05 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>now when i am iterating over the dataloader from the dictionary created on gpu it is giving me the following error :–<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e71e5528a62b4288f63b698b78cd91fb09a7888.png" alt="image" data-base62-sha1="kk7OVuZIiBZrtIRioFeQoz9MFG8" width="545" height="335"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60776248f8afeb758f34955b0d54b170bf07c763.png" data-download-href="https://discuss.pytorch.org/uploads/default/60776248f8afeb758f34955b0d54b170bf07c763" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60776248f8afeb758f34955b0d54b170bf07c763_2_690x260.png" alt="image" data-base62-sha1="dLnCc4JorzrUFTPzDITZvPZUe55" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60776248f8afeb758f34955b0d54b170bf07c763_2_690x260.png, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60776248f8afeb758f34955b0d54b170bf07c763.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60776248f8afeb758f34955b0d54b170bf07c763.png 2x" data-dominant-color="282C32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1012×382 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention-group" href="/groups/admins">@admins</a> <a class="mention" href="/u/smth">@smth</a> Please help</p>

ptrblck: <p>Please don’t tag certain persons, as this might discourage others to answer.<br>
Also, it’s not a good idea to tag all admins when creating the topic.</p>
<p>That being said, it seems that your <code>input</code> is a <code>str</code> object while a tensor is expected.<br>
I would recommend to check the implementation of your <code>Dataset</code> and make sure that tensors are returned.</p>

IamSparky: <p>This was the Dataset which I implemented<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ef50847125b439c35183004011d44d91d66f4e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ef50847125b439c35183004011d44d91d66f4e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ef50847125b439c35183004011d44d91d66f4e0_2_644x500.png" alt="image" data-base62-sha1="koEMmVMfjvQlOINU9nlL8oIyC0o" width="644" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ef50847125b439c35183004011d44d91d66f4e0_2_644x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ef50847125b439c35183004011d44d91d66f4e0_2_966x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ef50847125b439c35183004011d44d91d66f4e0.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">982×762 86.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>then i did the sanity check<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15a059f5ffeff114297afa4ac0f3db2e7a4401fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/15a059f5ffeff114297afa4ac0f3db2e7a4401fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a059f5ffeff114297afa4ac0f3db2e7a4401fb_2_690x338.png" alt="image" data-base62-sha1="35jzQz2j8BG8Xbuq3LYh3PrEgCT" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15a059f5ffeff114297afa4ac0f3db2e7a4401fb_2_690x338.png, https://discuss.pytorch.org/uploads/default/original/3X/1/5/15a059f5ffeff114297afa4ac0f3db2e7a4401fb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/5/15a059f5ffeff114297afa4ac0f3db2e7a4401fb.png 2x" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">953×467 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>which returned<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b1e21067e584d2f28c78e22e2ec4437b9a27008.png" alt="image" data-base62-sha1="aIwmp8WOJUzS2H6mLFwAgidH4Eg" width="451" height="446"></p>
<p>and then I did<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/337266818ea6200592b9aa2196e46df2bb8f6faa.png" data-download-href="https://discuss.pytorch.org/uploads/default/337266818ea6200592b9aa2196e46df2bb8f6faa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/337266818ea6200592b9aa2196e46df2bb8f6faa.png" alt="image" data-base62-sha1="7l7s4n4Xtka7qvrJJlF3W408kgW" width="439" height="500" data-dominant-color="EEF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">453×515 24.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>among which the train dataloder is working fine while the validation dataloader not.<br>
I did the same exact steps after the above code mentioned in the following topic</p>

Valerio_Biscione: <p>I would strongly suggest you not to put images but real code and possible a small reprudicible code snippet so we can try it out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> You will get much more people to answer</p>

IamSparky: <p>so this was the dataset class which i created</p>
<h1>defining dataset</h1>
<pre><code class="lang-auto">from PIL import Image
import cv2
import albumentations
import torch
import numpy as np
import io
from torch.utils.data import Dataset

class FlowerDataset(Dataset):
    def __init__(self, id , classes , image , img_height , img_width, mean , std , is_valid):
        self.id = id
        self.classes = classes
        self.image = image
        if is_valid == 1:
            self.aug = albumentations.Compose([
               albumentations.Resize(img_height , img_width, always_apply = True) ,
               albumentations.Normalize(mean , std , always_apply = True) 
            ])
        else:
            self.aug = albumentations.Compose([
                albumentations.Resize(img_height , img_width, always_apply = True) ,
                albumentations.Normalize(mean , std , always_apply = True),
                albumentations.ShiftScaleRotate(shift_limit = 0.0625,
                                                scale_limit = 0.1 ,
                                                rotate_limit = 5,
                                                p = 0.9)
            ])
        
    def __len__(self):
        return len(self.id)
    
    def __getitem__(self, index):
        id = self.id[index]
        img = np.array(Image.open(io.BytesIO(self.image[index]))) 
        img = cv2.resize(img, dsize=(128, 128), interpolation=cv2.INTER_CUBIC)
        img = self.aug(image = img)['image']
        img = np.transpose(img , (2,0,1)).astype(np.float32)
       
        
        return {
            'image' : torch.tensor(img, dtype = torch.float),
            'class' : torch.tensor(self.classes[index], dtype = torch.long) 
        }
</code></pre>
<p>and after that I did the sanity check which also worked fine</p>
<h1>sanity check for FlowerDataset class created</h1>
<pre><code class="lang-auto">train_dataset = FlowerDataset(id = train_ids, classes = train_class, image = train_images, 
                        img_height = 128 , img_width = 128, 
                        mean = (0.485, 0.456, 0.406),
                        std = (0.229, 0.224, 0.225) , is_valid = 0)

val_dataset = FlowerDataset(id = val_ids, classes = val_class, image = val_images, 
                        img_height = 128 , img_width = 128, 
                        mean = (0.485, 0.456, 0.406),
                        std = (0.229, 0.224, 0.225) , is_valid = 1)


import matplotlib.pyplot as plt
%matplotlib inline

idx = 119
img = val_dataset[idx]['image']

print(val_dataset[idx]['class'])

npimg = img.numpy()
plt.imshow(np.transpose(npimg, (1,2,0)))
</code></pre>

ptrblck: <p>The code looks generally alright.<br>
I would suggest to loop once over the complete dataset and check, if the returned batches are indeed tensors or if at some index a <code>str</code> object might be returned.<br>
If this works, then you should check, if you are reassigning <code>input</code> somewhere in your code to a <code>str</code>.</p>

IamSparky: <p>When I loop over the train datalader  as --&gt;</p>
<h1>setting up the train data loader</h1>
<pre><code class="lang-auto">!pip install git+https://github.com/ildoonet/cutmix
from cutmix.cutmix import CutMix

train_dataloader = CutMix(train_dataset, 
                          num_class=104, 
                          beta=1.0, 
                          prob=0.5, 
                          num_mix=2)

for inputs, labels in train_dataloader:
    if torch.cuda.is_available():
        inputs = inputs.cuda()
        labels = labels.cuda()
</code></pre>
<p><strong>it did not produce any error</strong></p>
<p>Whereas;<br>
When I loop over the validation datalader  as --&gt;</p>
<h1>setting up the val data loader</h1>
<pre><code class="lang-auto">from torch.utils.data import DataLoader

val_dataloader = DataLoader(val_dataset,
                        shuffle=False,
                        num_workers=4,
                        batch_size=128
                       )

for inputs, labels in val_dataloader:
    if torch.cuda.is_available():
        inputs = inputs.cuda()
        labels = labels.cuda()
</code></pre>
<p>it is producing the error . Can’t able to figure out where am I going wrong .</p>

ptrblck: <p>That’s not bad at all and good debugging! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Could you use:</p>
<pre><code class="lang-python">val_dataloader = DataLoader(val_dataset,
                        shuffle=False,
                        num_workers=4,
                        batch_size=1
                       )

for idx, (inputs, labels) in enumerate(val_dataloader):
    print(idx)
    inputs = inputs.cuda()
    labels = labels.cuda()
</code></pre>
<p>to get the index where the “bad” sample is created?<br>
Once you get this index, try to use it in the <code>Dataset</code> via:</p>
<pre><code class="lang-python">data, labels = val_dataset[idx]
data, labels = data.cuda(), labels.cuda()
</code></pre>
<p>which should also create the same error and look into the data loading in <code>Dataset.__init__</code>, why this sample is creating this <code>str</code> object.</p>
<p>Let me know, if you get stuck.</p>

IamSparky: <p>When I used your cde</p>
<pre><code class="lang-auto">val_dataloader = DataLoader(val_dataset,
                        shuffle=False,
                        num_workers=4,
                        batch_size=1
                       )

for idx, (inputs, labels) in enumerate(val_dataloader):
    print(idx)
    inputs = inputs.cuda()
    labels = labels.cuda()
</code></pre>
<p>it is getting error at every index for the string thing<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8d83b591bee9523149f6beaf4d02e1446244413.png" data-download-href="https://discuss.pytorch.org/uploads/default/b8d83b591bee9523149f6beaf4d02e1446244413" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8d83b591bee9523149f6beaf4d02e1446244413.png" alt="image" data-base62-sha1="qnd80kxXxy7i6QdvOQScn7rxsJR" width="690" height="272" data-dominant-color="3E4246"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">749×296 21.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and when</p>
<pre><code class="lang-auto">data, labels = val_dataset[0]
data, labels = data.cuda(), labels.cuda()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1df5da166d7409bda771a266b3936ca15371af21.png" data-download-href="https://discuss.pytorch.org/uploads/default/1df5da166d7409bda771a266b3936ca15371af21" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1df5da166d7409bda771a266b3936ca15371af21.png" alt="image" data-base62-sha1="4h2A4TJzeOo4NZKSXKEaLqvS2dP" width="690" height="163" data-dominant-color="2A2E33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">750×178 16.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/877c79100f174798c34d8ab76af586ec4c4d46d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/877c79100f174798c34d8ab76af586ec4c4d46d4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/877c79100f174798c34d8ab76af586ec4c4d46d4.png" alt="image" data-base62-sha1="jkz8HYldvXzinjDZKuFweH5KfU8" width="616" height="500" data-dominant-color="43464B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">793×643 63.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are returning a <code>dict</code> with two keys (<code>"image"</code> and <code>"class"</code>).<br>
If you are unwrapping it, <code>data</code> and <code>label</code> will contain the keys, which are strings:</p>
<pre><code class="lang-python">batch = {'image': torch.randn(1),
         'class': torch.randn(1)}

data, labels = batch
print(data)
&gt; image
print(labels)
&gt; class
</code></pre>
<p>Create the <code>data</code> and <code>label</code> tensors via:</p>
<pre><code class="lang-python">for batch in val_dataloader:
    data = batch['image']
    labels = batch['class']
</code></pre>

IamSparky: <p>thanks , the model is training now after I changed the following thing in training loop–&gt;</p>
<pre><code class="lang-auto">for batch in dataloaders[phase]:
     if torch.cuda.is_available():
        inputs = batch['image'].cuda()
        labels = batch['class'].cuda()
</code></pre>
<p>but it is returning me same loss and accuracy at every iternation of the epoch , please help me as I am really finding it difficult to solve it by my own.</p>
<h1><a name="setting-up-the-optimizer-loss-func-scheduler-for-training-1" class="anchor" href="#setting-up-the-optimizer-loss-func-scheduler-for-training-1"></a>setting up the optimizer , loss func. &amp; scheduler for training</h1>
<pre><code class="lang-auto">from cutmix.utils import CutMixCrossEntropyLoss 
from torchcontrib.optim import SWA #for Stochastic Weight Averaging in PyTorch


base_optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

optimizer = SWA(base_optimizer, swa_start=5, swa_freq=5, swa_lr=0.05)

loss_fn = CutMixCrossEntropyLoss(True)

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
</code></pre>
<h1><a name="setting-a-function-to-freeze-the-parameters-learned-in-a-pretrained-model-2" class="anchor" href="#setting-a-function-to-freeze-the-parameters-learned-in-a-pretrained-model-2"></a>setting a function to freeze the parameters learned in a pretrained model</h1>
<pre><code class="lang-auto">def set_parameters_requires_grad(model , extracting):
    if extracting:
        for param in model.parameters():
            param.requires_grad = False
</code></pre>
<h1><a name="setting-up-the-training-function-3" class="anchor" href="#setting-up-the-training-function-3"></a>setting up the training function</h1>
<pre><code class="lang-auto">if __name__ == "__main__":
    
    set_parameters_requires_grad(model , True)
    
    epochs = 25
    
    for epoch in range(epochs):
        print('Epoch ', epoch,'/',epochs-1)
        print('-'*15)

        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0.0

        # Iterate over data.
        for batch in dataloaders[phase]:
            if torch.cuda.is_available():
                inputs = batch['image'].cuda()
                labels = batch['class'].cuda()
    
            # zero the parameter gradients
            optimizer.zero_grad()

            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = loss_fn(outputs, labels)

                # we backpropagate to set our learning parameters only in training mode
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)

        # scheduler for weight decay
        if phase == 'train':
            scheduler.step()

        epoch_loss = running_loss / dataset_sizes[phase]
        epoch_acc = running_corrects / dataset_sizes[phase]

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))
    optimizer.swap_swa_sgd()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/921af05bd84ba20b2404997cb79f272eb7023fcd.png" alt="image" data-base62-sha1="kQvqtrNHyAkS5NMp2hoFir0GDIV" width="335" height="208"></p>

ptrblck: <p>It seems you are freezing the complete model and also <code>phase</code> is never set to <code>'train'</code> after these lines of code:</p>
<pre><code class="lang-python">        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode
</code></pre>
<p>This will only execute the loop and continue with the rest of the code in <code>phase=='val'</code>, which will never train your model. Note that freezing the complete model will also raise an error, if you are trying to call <code>loss.backward()</code>, so this is another sign that <code>phase</code> is never <code>'train'</code> in this part of the code.</p>

IamSparky: <p>Can you please help me in making changes for the training loop. I will be really thankful to you.<br>
Actually I followed this link for setting up the trainng loop for my model .<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>

ptrblck: <p>The indentation is wrong and you would have to add an indent to the code after the posted code snippet, so that it will be executed in the <code>for phase</code> loop.<br>
Also, once this is fixed, you’ll run into the other error, since you are freezing all parameters of the model, so you might want to keep some parameters trainable.</p>

IamSparky: <p>And one thing more is the placement for Stochastic Weight averaging is correct in my current code snippet…i.e. at the last , after all the training loops get complete ?</p>

ptrblck: <p>I’m not deeply familiar with <code>SWA</code>, but based on <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/">this document</a> it seems you are using it right. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

IamSparky: <p>I am still stucked here …the model is not training and returing 0 accuracy for training dataset ,</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    
    set_parameters_requires_grad(model , True)

    epochs = 25

    for epoch in range(epochs):
        print('Epoch ', epoch,'/',epochs-1)
        print('-'*15)

        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0.0
            
            # Iterate over data.
            for batch in dataloaders[phase]:
                if torch.cuda.is_available():
                    inputs = batch['image'].cuda()
                    labels = batch['class'].cuda()

                # zero the parameter gradients
                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = loss_fn(outputs, labels)

                    # we backpropagate to set our learning parameters only in training mode
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            # scheduler for weight decay
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))
    optimizer.swap_swa_sgd()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f38e69ed856276408c22b9da5f04e2db8255a76d.png" alt="image" data-base62-sha1="yKB913DKmTIw6XiDnhpP6bH2bgx" width="329" height="230"></p>
<p>I have also changed the optimizer to train only the final layer of efficeint net model .</p>
<h1><a name="setting-up-the-optimizer-loss-func-scheduler-for-training-1" class="anchor" href="#setting-up-the-optimizer-loss-func-scheduler-for-training-1"></a>setting up the optimizer , loss func. &amp; scheduler for training</h1>
<pre><code class="lang-auto">from cutmix.utils import CutMixCrossEntropyLoss #for Stochastic Weight Averaging in PyTorch

from torchcontrib.optim import SWA

base_optimizer = torch.optim.Adam(model._fc.parameters(), lr=1e-4)

optimizer = SWA(base_optimizer, swa_start=5, swa_freq=5, swa_lr=0.05)

loss_fn = CutMixCrossEntropyLoss(True)

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
</code></pre>

ptrblck: <p>Since the loss as well as the accuracy are both zero, I would recommend to debug the code step by step and check the data types as well as values of <code>loss</code>, <code>running_loss</code>, <code>running_correct</code>, <code>epoch_loss</code>, and <code>epoch_acc</code>. I guess that you might be using an integer division at some point, which would create the zeros.</p>
<p>PS: the usage of <code>.data</code> is not recommended, as it might have unwanted side effect.</p>

IamSparky: <aside class="quote no-group" data-username="ptrblck" data-post="19" data-topic="88727">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>epoch_acc</p>
</blockquote>
</aside>
<p>on your update<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4d1a778b03c8571925db2119ed2ed5fca898b1b.png" alt="image" data-base62-sha1="wEe2O49VfoRj3AeihjnBHVNpQZt" width="362" height="189"></p>
<p>I also typecast the values of dataset_sizes[phase] ,  float(dataset_sizes[phase]  to float(dataset_sizes[phase]) , float(dataset_sizes[phase]) respectively as they were integer varibles before;</p>
<p>But the strange  thing I noticed  was here</p>
<h1><a name="iterate-over-data-1" class="anchor" href="#iterate-over-data-1"></a>Iterate over data.</h1>
<pre><code class="lang-auto">  for batch in dataloaders[phase]:
     print(phase)
     if torch.cuda.is_available():
         inputs = batch['image'].cuda()
         labels = batch['class'].cuda()
</code></pre>
<p>it gave me this output<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6411dc508bf036288fe0d3df258d96a3dc89d914.png" alt="image" data-base62-sha1="ehg0OMVDKB0dmgnilEEAiQFMseM" width="328" height="280"></p>
<p>means that phase is not going to train mode…am I right?</p>

ptrblck: <p>This might be still the case and as explained before, the indentation was the reason for it, which should have been fixed by now.</p>