Performance(accuracy) difference between desktop and android

username: sklationd
cooked: <h1>Environment</h1>
<ul>
<li>Desktop (Colab)
<ul>
<li>pytorch version: 1.7.0</li>
</ul>
</li>
<li>Android
<ul>
<li>torch version: 1.7.0</li>
<li>torchvision version: 1.7.0</li>
</ul>
</li>
</ul>
<p>I am working on a image classification task with mobilenet v2 on android machine.<br>
Firstly, I train &amp; save model with colab by below code snippet. (I have five labels)</p>
<pre><code class="lang-auto"># model definition with five labels

model = torchvision.models.mobilenet_v2(pretrained=True)
model.classifier[1] = torch.nn.Linear(in_features=model.classifier[1].in_features, out_features=5)
model.eval()
model.to(device)
</code></pre>
<pre><code class="lang-auto"># save model with quantization and optimization

from torch.utils.mobile_optimizer import optimize_for_mobile
def model_save(model, name):
    q_model = torch.quantization.convert(model)
    traced_script_module = torch.jit.trace(q_model, torch.rand(1,3,224,224).to(device))
    opt_model = optimize_for_mobile(traced_script_module)
    torch.jit.save(opt_model, name)
</code></pre>
<pre><code class="lang-auto"># training step

for epoch in range(num_epochs):
  epoch_loss = 0
  test_loss = 0
  best_test_loss = 1

  for i, samples in enumerate(train_loader):
      model.train()
      imgs, annotations = samples
      imgs, annotations = imgs.to(device), annotations.to(device)
      
      output = model(imgs)
      loss = loss_func(output,annotations)
      
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      print(f'Iteration: {i+1}/{len(train_loader)}, Loss: {loss.item()}')
      epoch_loss += loss.item()

  avg_train_loss = epoch_loss/len(train_loader)

  # Save result for plotting
  train_loss_list.append(avg_train_loss)

  # Print epoch's test loss
  print(f'Epoch {epoch} train loss: {avg_train_loss}')

  # validation
  for i,test_samples in enumerate(test_loader):
    with torch.no_grad():
      model.eval()
      test_imgs, test_annotations = test_samples
      test_imgs, test_annotations = test_imgs.to(device), test_annotations.to(device)

      test_output = model(test_imgs)
      loss = loss_func(test_output, test_annotations)
      test_loss += loss.item()

  avg_test_loss = test_loss/len(test_loader)

  # save best
  if best_test_loss &gt; avg_test_loss:
    best_test_loss = avg_test_loss
    model_save(model, 'quantized_best.pt')
</code></pre>
<p>And I transform this model manually in my local machine(mac).<br>
Because If I don’t transform model, I get some errors in android machine.</p>
<pre><code class="lang-auto">import torch
quantized_torchmodel = torch.jit.load("quantized_best.pt", map_location='cpu')
torch.jit.save(quantized_torchmodel, "best_cpu_quantized.pt")
</code></pre>
<pre><code class="lang-auto"># ERROR that appears when I don't transform model
2020-11-26 13:33:34.877 416-750/org.pytorch.demo E/PyTorchDemo: Error during image analysis
    com.facebook.jni.CppException: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. 'aten::empty_strided' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].

</code></pre>
<p>and load this model to module in android code and inference some images.</p>
<pre><code class="lang-auto">mModule = Module.load(moduleFileAbsoluteFilePath);

...

final Tensor outputTensor = mModule.forward(IValue.from(mInputTensor)).toTensor();
</code></pre>
<p>However, the output(score) of inference with same image is quite different between mobile and desktop(colab).</p>
<p>I used same image and same transformation(resize &amp; normalization …), but output is different.<br>
And strangely, emulator’s output is same with colab’s output.<br>
How can it possible?</p>

username: ptrblck
cooked: <p>Could you use a static input (e.g. all ones) and compare the output on both platforms? If these results are equal, I would guess that the image loading (decoding) and processing might differ between the approaches.</p>

username: sklationd
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your reply.<br>
However, unfortunately, outputs are different. I tried with below three environments.</p>
<ul>
<li>Colab</li>
<li>Android Emulator (Pixel XL API 30)</li>
<li>Actual Android Device (Samsung Galaxy S8)</li>
</ul>
<hr>
<ol>
<li>Colab</li>
</ol>
<pre><code class="lang-auto">// colab code snippet for testing
model = torch.jit.load(MYPATH, map_location='cpu')
model.eval()
one = torch.ones([1,3,224,224])
model(one)
</code></pre>
<pre><code class="lang-auto">// output of colab
tensor([[ 2.7785,  1.4376, -0.8041, -1.5155, -0.8308]])
</code></pre>
<hr>
<ol start="2">
<li>Android Emulator &amp; Actual Device</li>
</ol>
<pre><code class="lang-auto">// android code snippet for testing
mModule = Module.load(MYPATH);

...

// Initialize input to one
for(int i=0;i&lt;3*224*224;i++){
    mInputTensorBuffer.put(i,1);
}

// Get output
final Tensor outputTensor = mModule.forward(IValue.from(mInputTensor)).toTensor();
final float[] scores = outputTensor.getDataAsFloatArray();
Log.d("DEBUG", Arrays.toString(scores));

</code></pre>
<pre><code class="lang-auto">// Output of Emulator
2020-11-29 12:56:51.328 3030-3030/org.pytorch.demo D/DEBUG: [2.778521, 1.4375764, -0.80405545, -1.5155371, -0.83080786]
</code></pre>
<pre><code class="lang-auto">// Output of Actual Device
2020-11-29 12:58:08.625 725-725/org.pytorch.demo D/DEBUG: [1.224158, -0.65208256, -0.3742584, -0.123134516, -0.085230656]
</code></pre>
<hr>
<p>Emulator’s output is same with colab’s one, but android’s output is not.<br>
I could not figure out why this happened…</p>
<p>I tried with very latest release of pytorch &amp; pytorch-android.</p>

username: ptrblck
cooked: <p>I assume the models are all in <code>eval</code> mode? If so, this might be a new bug so could you create a GitHub issue so that we could track and reproduce it, please?</p>

username: sklationd
cooked: <p>Yes, every models are all in eval mode.<br>
I will create an issue about this in pytorch repo.<br>
Thank you.</p>

username: sklationd
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/48551" rel="noopener nofollow ugc">Link to Github Issue</a></p>