Resnet50 gets only 93% accuracy on MNIST

username: dangmanhtruong1995
cooked: <p>Hi, I recently tried some code using Resnet50 on MNIST, however I only got 93% while other results on the web said that they got to around 99%, can anyone check my code to see if there is anything wrong ? Thank you very much.</p>
<p>Here is my code:</p>
<blockquote>
<p>import numpy as np # linear algebra<br>
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)</p>
<h1>Input data files are available in the read-only “…/input/” directory</h1>
<h1>For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory</h1>
<p>import os<br>
for dirname, _, filenames in os.walk(‘/kaggle/input’):<br>
for filename in filenames:<br>
print(os.path.join(dirname, filename))</p>
<h1>You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using “Save &amp; Run All”</h1>
<h1>You can also write temporary files to /kaggle/temp/, but they won’t be saved outside of the current session</h1>
<h1>In[22]:</h1>
<p>import numpy as np<br>
import pandas as pd<br>
import matplotlib.pyplot as plt</p>
<p>import torch<br>
import torch.nn as nn<br>
from torch.autograd import Variable</p>
<p>import torchvision<br>
import torchvision.transforms as transforms<br>
from torch.utils.data import Dataset, DataLoader<br>
from sklearn.metrics import confusion_matrix</p>
<p>from pdb import set_trace</p>
<h1>In[23]:</h1>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
device</p>
<h1>In[24]:</h1>
<p>train_augment_transforms = transforms.Compose([<br>
transforms.RandomHorizontalFlip(0.5),</p>
<h1>transforms.RandomAffine(degrees=10, translate=(0.1, 0.1)),</h1>
<pre><code>transforms.ToTensor(),
</code></pre>
<h1>transforms.RandomErasing(),</h1>
<p>])</p>
<p><span class="hashtag-raw">#train_set</span> = torchvision.datasets.FashionMNIST(“./data”, download=True, transform=</p>
<h1>train_augment_transforms)</h1>
<p><span class="hashtag-raw">#test_set</span> = torchvision.datasets.FashionMNIST(“./data”, download=True, train=False, transform=</p>
<h1>transforms.Compose([transforms.ToTensor()]))</h1>
<p>train_set = torchvision.datasets.MNIST(“./data”, download=True, train=True, transform=<br>
train_augment_transforms)<br>
test_set = torchvision.datasets.MNIST(“./data”, download=True, train=False, transform=<br>
transforms.Compose([transforms.ToTensor()]))</p>
<p>train_loader = torch.utils.data.DataLoader(train_set,<br>
batch_size=8192)<br>
test_loader = torch.utils.data.DataLoader(test_set,<br>
batch_size=8192)</p>
<h1>In[30]:</h1>
<p>def output_label(label):<br>
output_mapping = {<br>
0: “T-shirt/Top”,<br>
1: “Trouser”,<br>
2: “Pullover”,<br>
3: “Dress”,<br>
4: “Coat”,<br>
5: “Sandal”,<br>
6: “Shirt”,<br>
7: “Sneaker”,<br>
8: “Bag”,<br>
9: “Ankle Boot”<br>
}<br>
input = (label.item() if type(label) == torch.Tensor else label)<br>
return output_mapping[input]</p>
<p><span class="hashtag-raw">#demo_loader</span> = torch.utils.data.DataLoader(train_set, batch_size=10)<br>
<span class="hashtag-raw">#batch</span> = next(iter(demo_loader))<br>
<span class="hashtag-raw">#images</span>, labels = batch</p>
<h1></h1>
<p><span class="hashtag-raw">#grid</span> = torchvision.utils.make_grid(images, nrow=10)</p>
<h1></h1>
<p><span class="hashtag-raw">#plt</span>.figure(figsize=(15, 20))<br>
<span class="hashtag-raw">#plt</span>.imshow(np.transpose(grid, (1, 2, 0)))<br>
<span class="hashtag-raw">#print</span>("labels: “, end=” ")<br>
<span class="hashtag-raw">#for</span> i, label in enumerate(labels):</p>
<h1>print(output_label(label), end=", ")</h1>
<h1>In[31]:</h1>
<p>def plot_grad_flow(named_parameters):<br>
ave_grads = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for n, p in named_parameters:<br>
if(p.requires_grad) and (“bias” not in n):<br>
layers.append(n)<br>
ave_grads.append(p.grad.abs().mean().detach().cpu().numpy())<br>
plt.plot(ave_grads, alpha=0.3, color=“b”)<br>
plt.hlines(0, 0, len(ave_grads)+1, linewidth=1, color=“k” )<br>
plt.xticks(range(0,len(ave_grads), 1), layers, rotation=“vertical”)<br>
plt.xlim(xmin=0, xmax=len(ave_grads))<br>
plt.xlabel(“Layers”)<br>
plt.ylabel(“average gradient”)<br>
plt.title(“Gradient flow”)<br>
plt.grid(True)</p>
<h1>In[32]:</h1>
<p>class BasicConv2d(nn.Module):</p>
<pre><code>def __init__(self, input_channels, output_channels, **kwargs):
    super().__init__()
    self.conv = nn.Conv2d(input_channels, output_channels, bias=False, **kwargs)
    self.bn = nn.BatchNorm2d(output_channels)
    self.relu = nn.ReLU(inplace=True)
    self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
    # set_trace()
    
def forward(self, x):
    x = self.conv(x)
    x = self.bn(x)
    x = self.relu(x)
    x = self.pool(x)

    return x
</code></pre>
<p>class Flatten(nn.Module):<br>
def forward(self, input):<br>
return input.view(input.size(0), -1)</p>
<p>def multi_acc(y_pred, y_test):<br>
try:<br>
y_pred_softmax = torch.log_softmax(y_pred, dim = 0)<br>
except:<br>
set_trace()<br>
_, y_pred_tags = torch.max(y_pred_softmax, dim = 1)<br>
try:<br>
correct_pred = (y_pred_tags == y_test).float()<br>
except:<br>
set_trace()<br>
acc = correct_pred.sum() / len(correct_pred)<br>
acc = torch.round(acc * 100)<br>
return acc</p>
<p>def make_train_step(model, model_name, loss_fn, optimizer):<br>
“”" Function to make one training step<br>
“”"<br>
def perform_train_step(X_train_batch, y_train_batch):<br>
model.train()<br>
# <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#inception-v3" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 2.1.1+cu121 documentation</a><br>
if model_name == “inception_v3”:<br>
y_train_pred, aux_outputs = model(X_train_batch)<br>
y_train_pred = y_train_pred.squeeze()<br>
aux_outputs = aux_outputs.squeeze()<br>
train_loss_1 = loss_fn(y_train_pred, y_train_batch)<br>
train_loss_2 = loss_fn(aux_outputs, y_train_batch)<br>
train_loss = train_loss_1 + 0.4*train_loss_2<br>
else:<br>
y_train_pred = model(X_train_batch).squeeze()<br>
train_loss = loss_fn(y_train_pred, y_train_batch)</p>
<pre><code>    train_acc = multi_acc(y_train_pred, y_train_batch)
    train_loss.backward()
    plot_grad_flow(model.named_parameters())
    optimizer.step()
    optimizer.zero_grad()
    return (train_acc.item(), train_loss.item())

return perform_train_step
</code></pre>
<p>def make_val_step(model, model_name, loss_fn, optimizer):<br>
“”" Function to make one validation step<br>
“”"<br>
def perform_val_step(X_val_batch, y_val_batch):<br>
model.eval()<br>
y_val_pred = model(X_val_batch).squeeze()<br>
val_acc = multi_acc(y_val_pred, y_val_batch)<br>
val_loss = loss_fn(y_val_pred, y_val_batch)<br>
return (val_acc.item(), val_loss.item())</p>
<pre><code>return perform_val_step
</code></pre>
<p>def mini_batch(device, data_loader, step_fn):<br>
“”" Function to run through a mini-batch (train or validation)<br>
“”"<br>
mini_batch_acc_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
mini_batch_loss_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for (X_batch, y_batch) in data_loader:<br>
X_batch = X_batch.to(device)<br>
y_batch = y_batch.to(device)</p>
<pre><code>    (mini_batch_acc, mini_batch_loss) = step_fn(X_batch, y_batch)

    mini_batch_acc_list.append(mini_batch_acc)
    mini_batch_loss_list.append(mini_batch_loss)

loss = np.mean(mini_batch_loss_list)
acc = np.mean(mini_batch_acc_list)
</code></pre>
<h1>set_trace()</h1>
<pre><code>return (acc, loss)
</code></pre>
<p>import torch<br>
import torch.nn as  nn<br>
import torch.nn.functional as F</p>
<p>class Bottleneck(nn.Module):<br>
expansion = 4<br>
def <strong>init</strong>(self, in_channels, out_channels, i_downsample=None, stride=1):<br>
super(Bottleneck, self).<strong>init</strong>()</p>
<pre><code>    self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=1, padding=0)
    self.batch_norm1 = nn.BatchNorm2d(out_channels)
    
    self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, padding=1)
    self.batch_norm2 = nn.BatchNorm2d(out_channels)
    
    self.conv3 = nn.Conv2d(out_channels, out_channels*self.expansion, kernel_size=1, stride=1, padding=0)
    self.batch_norm3 = nn.BatchNorm2d(out_channels*self.expansion)
    
    self.i_downsample = i_downsample
    self.stride = stride
    self.relu = nn.ReLU()
    
def forward(self, x):
    identity = x.clone()
    x = self.relu(self.batch_norm1(self.conv1(x)))
    
    x = self.relu(self.batch_norm2(self.conv2(x)))
    
    x = self.conv3(x)
    x = self.batch_norm3(x)
    
    #downsample if needed
    if self.i_downsample is not None:
        identity = self.i_downsample(identity)
    #add identity
    x+=identity
    x=self.relu(x)
    
    return x
</code></pre>
<p>class Block(nn.Module):<br>
expansion = 1<br>
def <strong>init</strong>(self, in_channels, out_channels, i_downsample=None, stride=1):<br>
super(Block, self).<strong>init</strong>()</p>
<pre><code>    self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=stride, bias=False)
    self.batch_norm1 = nn.BatchNorm2d(out_channels)
    self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1, stride=stride, bias=False)
    self.batch_norm2 = nn.BatchNorm2d(out_channels)

    self.i_downsample = i_downsample
    self.stride = stride
    self.relu = nn.ReLU()

def forward(self, x):
  identity = x.clone()

  x = self.relu(self.batch_norm2(self.conv1(x)))
  x = self.batch_norm2(self.conv2(x))

  if self.i_downsample is not None:
      identity = self.i_downsample(identity)
  print(x.shape)
  print(identity.shape)
  x += identity
  x = self.relu(x)
  return x
</code></pre>
<p>class ResNet(nn.Module):<br>
def <strong>init</strong>(self, ResBlock, layer_list, num_classes, num_channels=3):<br>
super(ResNet, self).<strong>init</strong>()<br>
self.in_channels = 64</p>
<pre><code>    self.conv1 = nn.Conv2d(num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
    self.batch_norm1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU()
    self.max_pool = nn.MaxPool2d(kernel_size = 3, stride=2, padding=1)
    
    self.layer1 = self._make_layer(ResBlock, layer_list[0], planes=64)
    self.layer2 = self._make_layer(ResBlock, layer_list[1], planes=128, stride=2)
    self.layer3 = self._make_layer(ResBlock, layer_list[2], planes=256, stride=2)
    self.layer4 = self._make_layer(ResBlock, layer_list[3], planes=512, stride=2)
    
    self.avgpool = nn.AdaptiveAvgPool2d((1,1))
    self.fc = nn.Linear(512*ResBlock.expansion, num_classes)
    
def forward(self, x):
    x = self.relu(self.batch_norm1(self.conv1(x)))
    x = self.max_pool(x)

    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)
    
    x = self.avgpool(x)
    x = x.reshape(x.shape[0], -1)
    x = self.fc(x)
    
    return x
    
def _make_layer(self, ResBlock, blocks, planes, stride=1):
    ii_downsample = None
    layers = []
    
    if stride != 1 or self.in_channels != planes*ResBlock.expansion:
        ii_downsample = nn.Sequential(
            nn.Conv2d(self.in_channels, planes*ResBlock.expansion, kernel_size=1, stride=stride),
            nn.BatchNorm2d(planes*ResBlock.expansion)
        )
        
    layers.append(ResBlock(self.in_channels, planes, i_downsample=ii_downsample, stride=stride))
    self.in_channels = planes*ResBlock.expansion
    
    for i in range(blocks-1):
        layers.append(ResBlock(self.in_channels, planes))
        
    return nn.Sequential(*layers)
</code></pre>
<p>def ResNet50(num_classes, channels=3):<br>
return ResNet(Bottleneck, [3,4,6,3], num_classes, channels)</p>
<p>model = ResNet50(10, channels=1)</p>
<p>model.to(device)</p>
<p>error = nn.CrossEntropyLoss()</p>
<p>learning_rate = 0.001<br>
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)</p>
<h1>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)</h1>
<h1>print(model)</h1>
<p>num_epochs = 300<br>
train_step_fn = make_train_step(model, “FashionCNN”, error, optimizer)<br>
val_step_fn = make_val_step(model, “FashionCNN”, error, optimizer)</p>
<p>def perform_training(model, train_loader, val_loader, num_epochs = 20):<br>
count = 0<br>
accuracy_stats = {<br>
‘train’: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“val”: <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
}<br>
loss_stats = {<br>
‘train’: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“val”: <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
}</p>
<pre><code>for epoch in range(num_epochs):
    (train_epoch_acc, train_epoch_loss) = mini_batch(device, train_loader, train_step_fn)
    
    with torch.no_grad():
        (val_epoch_acc, val_epoch_loss) = mini_batch(device, val_loader, val_step_fn)
        loss_stats['val'].append(val_epoch_loss)
        accuracy_stats['val'].append(val_epoch_acc)
    
    print("Epoch: %d, train loss: %f, val loss: %f, train acc: %f, val acc: %f"  % (epoch, train_epoch_loss, val_epoch_loss, train_epoch_acc, val_epoch_acc))
</code></pre>
<p>perform_training(model, train_loader, test_loader, num_epochs=num_epochs)</p>
</blockquote>

username: J_Johnson
cooked: <aside class="quote no-group" data-username="dangmanhtruong1995" data-post="1" data-topic="181052">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dangmanhtruong1995/48/59870_2.png" class="avatar"> dangmanhtruong1995:</div>
<blockquote>
<p>model = ResNet50(10, channels=1)</p>
</blockquote>
</aside>
<p>If I recall correctly, models that perform in the 99% percentile are from pre-trained weights and just replacing the model head for fine-tuning. See here for fine-tuning methods:</p>
<p><a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>
<p>TLDR: A CNN classifier that has ‘seen’ a lot more than just the current dataset is better equipped to deal with outliers.</p>

username: dangmanhtruong1995
cooked: <p>Hi I was wondering if my code has any bug in it, previously I tried FashionMNIST and for some reason the train accuracy is higher than test accuracy. I just tried MNIST using pretrained weights and the test accuracy reaches around 98%, I hope it’s okay.</p>