Training and Validation Loss Too High and Not Converging

username: swilliams2000
cooked: <p><strong>I am trying to calculate training and validation loss however I am getting an extremely high amount that is not converging</strong></p>
<p>‘’'transform = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5,), (0.5,))<br>
])</p>
<p>batch_size = 32</p>
<p>cifar10 = torchvision.datasets.CIFAR10(root=‘./data’, download=True, transform=torchvision.transforms.ToTensor())<br>
pivot = 40000<br>
cifar10 = sorted(cifar10, key=lambda x: x[1])<br>
train_set = torch.utils.data.Subset(cifar10, range(pivot))<br>
val_set = torch.utils.data.Subset(cifar10, range(pivot, len(cifar10)))<br>
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)<br>
val_loader = DataLoader(val_set, batch_size=batch_size, shuffle=True)‘’’</p>
<p>class Network(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
<span class="hashtag">#Using</span> padding convolution 2d if downsampling is performed by average pooling<br>
self.conv1 = nn.Conv2d(3, 6, kernel_size = 5, padding = 2)<br>
<span class="hashtag">#MaxPooling2D</span> has no attribute with torch.nn so changed it to MaxPool2d<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, kernel_size = 5, padding = 2)<br>
self.fc1 = nn.Linear(8<em>8</em>16, 120)<br>
self.fc2 = nn.Linear(120, 2)<br>
self.fc3 = nn.Linear(2, 10)</p>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    #Flatten has no attribute with torch so changed it to flatten
    x = torch.flatten(x, 1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
<p>model = Network()</p>
<p>criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=1e3, momentum=0.9)</p>
<p>with wandb.init(project = ‘Tier-1-Test’, save_code=True) as run:<br>
for epoch in range(5):<br>
current_loss = 0</p>
<pre><code>    model.train()

    for i, data in enumerate(train_loader):
        images, labels = data

        outputs = model(images)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()

        current_loss += loss
        
    run.log({'train_loss': current_loss / (i + 1)}) 
    
    model.eval()

    current_loss = 0

    for i, data in enumerate(val_loader):
        images, labels = data
        outputs = model(images)

        loss = criterion(outputs, labels)

        current_loss += loss

    run.log({'val_loss': current_loss / (i + 1)})
</code></pre>

username: ptrblck
cooked: <p>You are sorting the labels in:</p>
<pre><code class="lang-python">cifar10 = sorted(cifar10, key=lambda x: x[1])
</code></pre>
<p>and split this sorted dataset into a training and validation <code>Subset</code> which would move classes 0-7 into the <code>train_set</code> and <code>8-9</code> into the <code>val_loader</code> based on the <code>pivot</code>.<br>
Your model thus has never seen any classes present in the validation split and I would expect it to fail.</p>

username: swilliams2000
cooked: <p>Oh alright yea that makes sense! Probably a dumb question because I am missing something. But I changed the range of the val_set by combining it with the first row but I do not thing it got the classes still. Is there a better way for this?</p>
<pre><code class="lang-auto">cifar10 = sorted(cifar10, key=lambda x: x[1])
train_set = torch.utils.data.Subset(cifar10, range(pivot))
val_set_1 = torch.utils.data.Subset(cifar10, range(0))
val_set_2 = torch.utils.data.Subset(cifar10, range(pivot, len(cifar10)))
val_set = torch.utils.data.ConcatDataset([val_set_1, val_set_2])
</code></pre>

username: ptrblck
cooked: <p><code>range(0)</code> is an empty range and this <code>val_set_1</code> is also empty, so your code snippet doesn’t change anything.<br>
You could either remove the sorting, split the train and val indices manually by making sure both sets see all classes, or you could also use e.g. <code>sklearn.model_selection.train_test_split</code> with the <code>stratify</code> argument.</p>