Image Classification completely wrong with PyTorch Mobile iOS example

dormouse: <p>Dear all,</p>
<p>My trained/traced model got a good performance on PC. However, when i ship the model.pt to PyTorch Mobile and tested on iOS. the classification of same Image is completely wrong.<br>
i have no idea where is the problem and how to solve it.<br>
my model.pt is generated using Transfer learning with resnet18…</p>
<p>Please help! thanks</p>

ptrblck: <p>Without seeing the code it’s hard to debug, but you could check:</p>
<ul>
<li>your image loading and make sure all images are loaded in RGB format (which is the default in PyTorch, as it’s using <code>PIL</code>)</li>
<li>make sure the same preprocessing is applied (same resizing, normalization etc.)</li>
<li>call <code>model.eval()</code> after creating the instance and loading the <code>state_dict</code>.</li>
</ul>

dormouse: <p>here is the code of a Standard Transfer Learning with PyTorch:</p>
<pre><code class="lang-auto">
from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy

plt.ion()   # interactive mode


# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.ColorJitter(brightness=1, contrast=1, saturation=1, hue=0.5),
        transforms.RandomRotation(degrees=15),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = '20200328_data/EA2020Test'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    #torch.save(model.state_dict(), 'newtest.pth')
    return model




def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
	param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, len(class_names))

model_conv = model_conv.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that only parameters of final layer are being optimized as
# opposed to before.
optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

model_conv = train_model(model_conv, criterion, optimizer_conv,exp_lr_scheduler, num_epochs=25)

example = torch.rand(1, 3, 224, 224)

traced_script_module = torch.jit.trace(model_conv.cpu(), example)

traced_script_module.save("mymodel-jit-cpu.pt")


#model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

#visualize_model(model_conv)

</code></pre>
<p>And here is the code of validation on MacOS:</p>
<pre><code class="lang-auto"># sample execution (requires torchvision)

import torch
from PIL import Image
from torchvision import datasets, models, transforms
import os

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.ColorJitter(brightness=1, contrast=1, saturation=1, hue=0.5),
        transforms.RandomRotation(degrees=15),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

input_image = Image.open('./20200328_data/EA2020Test/val/Class3/frame100.jpg')
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

data_dir = '20200328_data/EA2020Test'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes



model_conv = torch.jit.load('mymodel-jit-cpu.pt')
model_conv.eval()

# move the input and model to GPU for speed if available
if torch.cuda.is_available():
    input_batch = input_batch.to('cuda')
    model.to('cuda')

with torch.no_grad():
    output = model_conv(input_batch)
    _, preds = torch.max(output, 1)
print(class_names[preds[0]])

</code></pre>
<p>With these codes, the single image is correctly classified. (e.g. Class3)</p>
<p>And here is the github of where the HelloWorld PyTorch mobile used:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld" target="_blank" rel="nofollow noopener">pytorch/ios-demo-app</a></h3>

<p>PyTorch iOS examples. Contribute to pytorch/ios-demo-app development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I just replaced the model-jit-cpu.pt and words.txt in the aboved example code.<br>
but the same image (e.g. frame28.jpg) is classified as Class23…</p>
<p>Thanks for helping me to figure out where is the problem.</p>

ptrblck: <p>The workflow seems to be correct. Are you able to check the raw outputs of your model and compare it to your Python validation code?</p>

dormouse: <p>thanks a lot for the hints, i will let you know the raw outputs dataset (40 tensor) later. Thanks</p>

dormouse: <p>Here is the raw output of my mode on iOS xcode environment:<br>
where the classification is totally wrong, the 23th tensor 7.87…</p>
<pre><code class="lang-auto">override func viewDidLoad() {
        super.viewDidLoad()
        let image = UIImage(named: "frame28.jpg")!
        imageView.image = image
        let resizedImage = image.resized(to: CGSize(width: 224, height: 224))
        guard var pixelBuffer = resizedImage.normalized() else {
            return
        }
        guard let outputs = module.predict(image: UnsafeMutableRawPointer(&amp;pixelBuffer)) else {
            return
        }
        print(outputs)
        let zippedResults = zip(labels.indices, outputs)
        print(labels.indices)
        let sortedResults = zippedResults.sorted { $0.1.floatValue &gt; $1.1.floatValue }.prefix(3)
        
        var text = ""
        for result in sortedResults {
            text += "\u{2022} \(labels[result.0]) \n\n"
        }
        resultView.text = text
</code></pre>
<p>result:<br>
[0.7592794, 1.170417, 0.4990637, 4.677132, -0.9478517, -2.447758, 1.918674, 0.5767481, 1.738559, -0.2370548, 0.4759183, 2.106035, -0.5786518, -1.511588, -2.636765, -2.717191, -0.9784743, -0.7371585, -2.024575, -2.237826, -2.281566, -0.03008107, <strong>7.870783</strong>, 1.824642, 0.3037256, -0.4274126, 0.4281527, -0.4137115, -2.859639, -2.277968, -1.029291, -0.2246231, 1.358673, 2.990178, 2.051248, -0.05761524, 2.349458, 0.004223851, -1.494618, -2.429391]</p>
<p>Here is the code of Python validation:</p>
<pre><code class="lang-auto"># sample execution (requires torchvision)

import torch
from PIL import Image
from torchvision import datasets, models, transforms
import os

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.ColorJitter(brightness=1, contrast=1, saturation=1, hue=0.5),
        transforms.RandomRotation(degrees=15),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

input_image = Image.open('./20200328_data/EA2020Test/val/Class3/frame28.jpg')
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

data_dir = '20200328_data/EA2020Test'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes



model_conv = torch.jit.load('mymodel-jit-cpu.pt')
model_conv.eval()

# move the input and model to GPU for speed if available
if torch.cuda.is_available():
    input_batch = input_batch.to('cuda')
    model.to('cuda')

with torch.no_grad():
    output = model_conv(input_batch)
    _, preds = torch.max(output, 1)
print(class_names[preds[0]])
print(output)
# The output has unnormalized scores. To get probabilities, you can run a softmax on it.
# print(torch.nn.functional.softmax(output[0], dim=0))
</code></pre>
<p>And here is the output tensor:<br>
tensor([[ 3.0214e+00,  3.8240e-01, <strong>-9.9551e-01</strong>,  3.9651e+00,  2.6704e+00,<br>
-1.5943e+00,  9.1830e-01, -6.0747e-03, -1.3737e+00, -6.4241e-01,<br>
5.8611e-01,  2.2428e+00, -1.2174e+00, -1.3305e-01, -2.4817e+00,<br>
-1.5725e+00, -1.0427e+00, -2.0795e+00, -2.4415e+00, -2.3984e-01,<br>
-2.3185e+00, -9.6899e-02,  <em><strong>9.8113e+00</strong></em>,  1.5801e-01, -8.7105e-01,<br>
-1.9445e+00, -5.7967e-01, -2.9598e-01, -1.9398e+00, -1.6080e+00,<br>
-7.6736e-01, -9.2793e-01,  1.9376e+00,  1.9328e+00,  1.3377e+00,<br>
-1.8019e-01,  2.2197e+00, -7.0646e-02, -2.4584e-01, -1.6802e+00]])<br>
And Python classificaiton is correct, where the 3rd class (Class3) is labeled.<br>
I got some feeling it might be some difference between torch.max (Python validation) and torch.exp<br>
(Torch Mobile)?</p>
<p>here is the code of TorchModule.mm</p>
<pre><code class="lang-auto">#import "TorchModule.h"
#import &lt;LibTorch/LibTorch.h&gt;

@implementation TorchModule {
 @protected
  torch::jit::script::Module _impl;
}

- (nullable instancetype)initWithFileAtPath:(NSString*)filePath {
  self = [super init];
  if (self) {
    try {
      auto qengines = at::globalContext().supportedQEngines();
      if (std::find(qengines.begin(), qengines.end(), at::QEngine::QNNPACK) != qengines.end()) {
        at::globalContext().setQEngine(at::QEngine::QNNPACK);
      }
      _impl = torch::jit::load(filePath.UTF8String);
      _impl.eval();
    } catch (const std::exception&amp; exception) {
      NSLog(@"%s", exception.what());
      return nil;
    }
  }
  return self;
}

- (NSArray&lt;NSNumber*&gt;*)predictImage:(void*)imageBuffer {
  try {
    at::Tensor tensor = torch::from_blob(imageBuffer, {1, 3, 224, 224}, at::kFloat);
    torch::autograd::AutoGradMode guard(false);
    at::AutoNonVariableTypeMode non_var_type_mode(true);
    auto outputTensor = _impl.forward({tensor}).toTensor();
    float* floatBuffer = outputTensor.data_ptr&lt;float&gt;();
    if (!floatBuffer) {
      return nil;
    }
    NSMutableArray* results = [[NSMutableArray alloc] init];
    for (int i = 0; i &lt; 40; i++) {
      [results addObject:@(floatBuffer[i])];
    }
    return [results copy];
  } catch (const std::exception&amp; exception) {
    NSLog(@"%s", exception.what());
  }
  return nil;
}

@end

</code></pre>
<p>thansk a lot for your help!</p>

xta0: <p><a class="mention" href="/u/dormouse">@dormouse</a>, could you try using png instead of jpg? - <a href="https://github.com/pytorch/pytorch/issues/27813" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27813</a></p>

dormouse: <p>Yes, i have just tried with frame28.png, but the classification is exactly the same (far from ground truth) as jpg. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

dormouse: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/xta0">@xta0</a><br>
I have figured out where is the problem.<br>
i double checked the predict output of Desktop and iOS, the torch.max gives the same tensor value(more or less same)<br>
But in iOS, the label.txt should be generated along with PyTorch learning process.<br>
I manually created a label.txt, like:<br>
Class1<br>
Class2<br>
Class3<br>
…<br>
which is not the case during PyTorch learning.<br>
During the PyTorch learning, with the dataloader, the class_name should be shuffled. So, just replace the shuffled label.txt with manually generated label.txt in xcode project should be fine!</p>
<p>Thanks for your attention.</p>

ducnm: <p>Great topic, I can learn more from here. Thanks <a class="mention" href="/u/dormouse">@dormouse</a>, <a class="mention" href="/u/xta0">@xta0</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
But I have a problem. When deployed a direct app from Xcode to devices, the result is good. But when create .ipa (ipa ad hoc distribution) and deploy through <a href="http://installonair.com" rel="noopener nofollow ugc">http://installonair.com</a>. My app returns a <strong>bad result</strong>. I used the same device and picture. How to fix it?</p>