How to print out model weights in iOS Libtorch?

username: Bryan_Wang
cooked: <p>Seems like the stuff that people are doing in C++ doesn’t quite work for iOS. For example, I tried:</p>
<pre><code class="lang-auto">for (const auto&amp; params : module.parameters()) {
          std::cout &lt;&lt; params.values() &lt;&lt; std::endl;
}
</code></pre>
<p>which yields:</p>
<p><code>values not implemented for TensorTypeSet(VariableTensorId, CPUTensorId) (values at /Users/distiller/project/build_ios/aten/src/ATen/core/TensorMethods.h:3534) (no backtrace available)</code></p>

username: Bryan_Wang
cooked: <p>Turns out params is a tensor so the following works:</p>
<pre><code class="lang-auto">  for (const auto&amp; params : _impl.parameters()) {
          params.print();
  }
</code></pre>
<p>However this only outputs the tensor shapes.</p>

username: xta0
cooked: <p>can you try <code>params[i][j].item&lt;float&gt;()</code> or use the buffer pointer <code>float* buffer = params.data_ptr&lt;float&gt;()</code></p>

username: Bryan_Wang
cooked: <p>Thanks <a class="mention" href="/u/xta0">@xta0</a>, that works great. I am using your iOS sample project as a template, and have run into a big problem that I was wondering if you could help with (I can make another thread if needed).</p>
<p>The traced model outputs very different results in Swift than it does in Python, and I don’t know why. I followed your steps in the sample project:</p>
<p>First, trace the model and see the output on a tensor of ones:</p>
<pre><code class="lang-auto">model = ExplicitSpeechResModel(36, 19)
model.load('../output/model_1h.pt')
model.eval()
ones = torch.ones([1, 101, 40])
traced_script_module = torch.jit.trace(model, ones)
traced_script_module.save("../output/traced_model.pt")
traced_script_module(ones)

//outputs:
//tensor([[-18.7624, -30.3478, -31.0299,  -1.1888,   8.6857,  33.7217, -36.2783,
// 51.8277,  55.9391,   9.0642,   8.6428,  10.3509,   0.2688,  43.9576,
// -7.1114, -55.3318, -16.7983, -13.5788,  -3.9336,  -1.1792,  14.3855,
//-31.8519, 101.3712, -43.9597,  40.5726, -16.2946, -15.8538,  21.1088,
//-31.5852, -14.2146, -14.5817,  19.9373, -21.5292,   9.4006, -45.0686,
// 21.4724]], grad_fn=&lt;DifferentiableGraphBackward&gt;)
</code></pre>
<p>On the Swift side, here is my TorchModule.mm. As you will see, the output on the same tensor is completely different:</p>
<pre><code class="lang-auto">@implementation TorchModule {
 @protected
  torch::jit::script::Module _impl;
}

- (nullable instancetype)initWithFileAtPath:(NSString*)filePath {
  self = [super init];
  if (self) {
    try {
      auto qengines = at::globalContext().supportedQEngines();
      if (std::find(qengines.begin(), qengines.end(), at::QEngine::QNNPACK) != qengines.end()) {
        at::globalContext().setQEngine(at::QEngine::QNNPACK);
      }
      _impl = torch::jit::load(filePath.UTF8String);
      _impl.eval();
    } catch (const std::exception&amp; exception) {
      NSLog(@"%s", exception.what());
      return nil;
    }
  }
  return self;
}


- (NSArray&lt;NSNumber*&gt;*)predictImage:(void*)imageBuffer {
  try {

    // try out dummy input of all ones
    at::Tensor tensor = torch::ones({1, 101, 40});
    torch::autograd::AutoGradMode guard(false);
    at::AutoNonVariableTypeMode non_var_type_mode(true);
    auto outputTensor = _impl.forward({tensor}).toTensor();
    float* floatBuffer = outputTensor.data_ptr&lt;float&gt;();
    if (!floatBuffer) {
      return nil;
    }
    NSMutableArray* results = [[NSMutableArray alloc] init];
    for (int i = 0; i &lt; 36; i++) {
      [results addObject:@(floatBuffer[i])];
    }
    return [results copy];
  } catch (const std::exception&amp; exception) {
    NSLog(@"%s", exception.what());
  }
  return nil;
}
@end
</code></pre>
<p>Calling predictImage results in the following vector:</p>
<p><code>[11.918683, -21.391111, -18.756794, -15.70252, -14.593732, 28.798603, -22.37965, 10.117706, 5.1135015, -8.376111, -25.258512, -7.270096, 0.9224758, 3.4262152, 28.566887, 2.90841, 25.247177, 35.124638, 14.7190695, -37.291008, -4.821145, 33.09956, 47.47553, 11.395653, 9.54897, -5.713372, -32.897644, -18.26301, -5.596691, -18.339537, -25.02614, -23.303043, -3.3603168, 31.69397, 3.0528922, 7.3663263]</code></p>
<p>It is totally different from the one in Python. The model weights seem to be the same, I’m calling model.eval() … not sure what else is missing. Here is my model architecture in case it is helpful:</p>
<pre><code class="lang-auto">class ExplicitSpeechResModel(SerializableModule):
    def __init__(self, n_labels, n_maps):
        super().__init__()
        self.conv0 = nn.Conv2d(1, n_maps, (3, 3), padding=(1, 1), bias=False)
        self.conv1 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(0 // 3)), dilation=int(2**(0 // 3)), bias=False)
        self.bn1 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv2 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(1 // 3)), dilation=int(2**(1 // 3)), bias=False)
        self.bn2 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv3 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(2 // 3)), dilation=int(2**(2 // 3)), bias=False)
        self.bn3 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv4 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(3 // 3)), dilation=int(2**(3 // 3)), bias=False)
        self.bn4 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv5 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(4 // 3)), dilation=int(2**(4 // 3)), bias=False)
        self.bn5 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv6 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(5 // 3)), dilation=int(2**(5 // 3)), bias=False)
        self.bn6 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv7 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(6 // 3)), dilation=int(2**(6 // 3)), bias=False)
        self.bn7 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv8 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(7 // 3)), dilation=int(2**(7 // 3)), bias=False)
        self.bn8 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv9 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(8 // 3)), dilation=int(2**(8 // 3)), bias=False)
        self.bn9 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv10 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(9 // 3)), dilation=int(2**(9 // 3)), bias=False)
        self.bn10 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv11 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(10 // 3)), dilation=int(2**(10 // 3)), bias=False)
        self.bn11 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv12 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(11 // 3)), dilation=int(2**(11 // 3)), bias=False)
        self.bn12 = nn.BatchNorm2d(n_maps, affine=False)
        self.conv13 = nn.Conv2d(n_maps, n_maps, (3, 3), padding=int(2**(12 // 3)), dilation=int(2**(12 // 3)), bias=False)
        self.bn13 = nn.BatchNorm2d(n_maps, affine=False)
        self.output = nn.Linear(n_maps, n_labels)

    def forward(self, x):
        x = x.unsqueeze(1)
        y0 = F.relu(self.conv0(x))
        x = self.bn1(F.relu(self.conv1(y0)))
        y2 = F.relu(self.conv2(x)) + y0
        x = self.bn2(y2)
        x =  self.bn3(F.relu(self.conv3(x)))
        y4 = F.relu(self.conv4(x)) + y2
        x = self.bn4(y4)
        x = self.bn5(F.relu(self.conv5(x)))
        y6 = F.relu(self.conv6(x)) + y4
        x = self.bn6(y6)
        x =  self.bn7(F.relu(self.conv7(x)))
        y8 = F.relu(self.conv8(x)) + y6
        x = self.bn8(y8)
        x = self.bn9(F.relu(self.conv9(x)))
        y10 = F.relu(self.conv10(x)) + y8
        x = self.bn10(y10)
        x =  self.bn11(F.relu(self.conv11(x)))
        y12 = F.relu(self.conv12(x)) + y10
        x = self.bn12(y12)
        x =  self.bn13(F.relu(self.conv13(x)))
        x = x.view(x.size(0), x.size(1), -1) # shape: (batch, feats, o3)
        x = torch.mean(x, 2)
        return self.output(x)
</code></pre>
<p>Sorry for the long post. I really appreciate your help so far, and the sample project you made as well.</p>

username: xta0
cooked: <p>This seems to be related to <a href="https://github.com/pytorch/pytorch/pull/39591" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/39591</a>. Can you manually apply the PR and recompile from the source? Please refer to the section here - <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" rel="nofollow noopener">https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source</a></p>

username: Bryan_Wang
cooked: <p>Thanks. I’ll let you know how it goes!</p>

username: Bryan_Wang
cooked: <p>I’m getting a bunch of errors like:</p>
<pre><code class="lang-auto">Undefined symbols for architecture x86_64:
  "c10::IValue::toModule() const", referenced from:
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ModulePolicy&gt;::next() in TorchModule.o
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ParameterPolicy&gt;::next() in TorchModule.o
  "c10::ivalue::Object::type() const", referenced from:
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ModulePolicy&gt;::valid() const in TorchModule.o
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ModulePolicy&gt;::next() in TorchModule.o
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ParameterPolicy&gt;::valid() const in TorchModule.o
      torch::jit::slot_iterator_impl&lt;torch::jit::detail::ParameterPolicy&gt;::next() in TorchModule.o
</code></pre>
<p>Seems like something went wrong with the initial build. I followed the posted instructions, and added a few steps.</p>
<ol>
<li>Pod install LibTorch 1.4.0 then replace the install folder in Pods/LibTorch with the install generated by the build.</li>
<li>Change C++ compiler version in XCode to C++14.</li>
</ol>
<p>Do you know what the problem might be?</p>
<p>If it is relevant, my cmake version is 3.12.3, and clang version is 11.0.3. I’m using Mac OS Catalina.</p>

username: xta0
cooked: <p>you’re not supposed to use cocoapods to recompile. Can you try</p>
<ol>
<li>Clone pytorch and check out the master branch</li>
<li>Manually apply the changes in the PR (only one liner)</li>
<li>Follow the section <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" rel="nofollow noopener">https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source</a> to recompile from source code and setup XCode</li>
</ol>

username: Bryan_Wang
cooked: <p>I did as you suggested, and am still getting these errors. Seems like Xcode can’t find some header file.</p>
<p><code>Undefined symbols for architecture x86_64:   "torch::jit::Object::find_method(std::__1::basic_string&lt;char, std::__1::char_traits&lt;char&gt;, std::__1::allocator&lt;char&gt; &gt; const&amp;) const", referenced from:       torch::jit::Object::get_method(std::__1::basic_string&lt;char, std::__1::char_traits&lt;char&gt;, std::__1::allocator&lt;char&gt; &gt; const&amp;) const in TorchModule.o   "at::GradMode::is_enabled()", referenced from:       at::AutoGradMode::AutoGradMode(bool) in TorchModule.o   "at::GradMode::set_enabled(bool)", referenced from:       at::AutoGradMode::AutoGradMode(bool) in TorchModule.o       at::AutoGradMode::~AutoGradMode() in TorchModule.o</code></p>
<p>I tried replacing the <code>force_load</code> flag with <code>all_load</code> and <code>Objc</code> but it has the same result. My header search path is <code>$(PROJECT_DIR)/install/include</code>. If I toggle this to “recursive”, the above errors disappear and are replaced by errors like:</p>
<p><code>No member named 'signbit' in the global namespace; did you mean 'sigwait'?</code></p>
<p>Seems like the recursive setting makes it so that cmath is incorrectly loaded. I have been trying to fix this for a while, and I was wondering if you had any suggestions! Thanks for the help.</p>

username: xta0
cooked: <p>Did you add these two guards?</p>
<pre><code class="lang-auto">at::AutoNonVariableTypeMode nonVarTypeModeGuard(true);
torch::autograd::AutoGradMode guard(false);
</code></pre>

username: Bryan_Wang
cooked: <p>Yes, I had those guards. The error is in compile time. Here is a minimal reproduction of the bug, if it helps.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/BryanWBear/PytorchDev" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/7650109?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/BryanWBear/PytorchDev" target="_blank" rel="nofollow noopener">BryanWBear/PytorchDev</a></h3>

<p>repro a compiler bug. Contribute to BryanWBear/PytorchDev development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: xta0
cooked: <p><a class="mention" href="/u/bryan_wang">@Bryan_Wang</a> I’m kinda busy recently, I’ll ask my colleague to followup with you</p>

username: husthyc
cooked: <p>Hi <a class="mention" href="/u/bryan_wang">@Bryan_Wang</a>, as <a class="mention" href="/u/xta0">@xta0</a> mentioned, the wrong prediction issue should have been fixed via PR 39591. Now I’m looking into the problem you have now. Thanks for your patience.</p>

username: husthyc
cooked: <p>Please follow the tutorial <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" class="inline-onebox" rel="noopener nofollow ugc">iOS | PyTorch</a> <strong>very carefully</strong> as <a class="mention" href="/u/xta0">@xta0</a> mentioned. Double check you’ve changed the value of <code>Header Search Paths</code>, <code>other linker flags</code>, and <code>bitcode </code> accordingly before you build it. When you drag <code>install</code> folder to the project, please make sure those three check boxes are checked as the screenshot shows, <strong>this is important</strong>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d94862bf4f5402db8a8fa650d061793150eae143.png" data-download-href="https://discuss.pytorch.org/uploads/default/d94862bf4f5402db8a8fa650d061793150eae143" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d94862bf4f5402db8a8fa650d061793150eae143_2_690x406.png" alt="image" data-base62-sha1="v0aHcvuuP0sNvd3ca0a0FHp4CXh" width="690" height="406" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d94862bf4f5402db8a8fa650d061793150eae143_2_690x406.png, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d94862bf4f5402db8a8fa650d061793150eae143.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d94862bf4f5402db8a8fa650d061793150eae143.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×430 56.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you dragged in successfully, the color of the folder icon of <code>install</code> in the side bar should be <strong>yellow</strong> instead of blue.</p>
<p>I also attached a successfully built project which is based on the minimal project you attached (not sure if PR 39591 is included or not, just for showing you the project file structure and settings). Hopefully it will help. <a href="https://www.dropbox.com/s/sh0plrh2oakbxcj/PytorchDev-Fixed.zip?dl=0" class="inline-onebox" rel="noopener nofollow ugc">Dropbox - File Deleted - Simplify your life</a> The password is <code> pytorchios</code></p>

username: Bryan_Wang
cooked: <p>Thank you so much for your help. I have been traveling for the past two weeks so unable to respond.</p>
<p>Would it be possible for you to reshare the link? I am getting “The owner hasn’t granted you access to this link” with no password prompt.</p>

username: husthyc
cooked: <p><a href="https://www.dropbox.com/s/mqe3pyme2mnh8o6/PytorchDev-Fixed.zip?dl=0" rel="nofollow noopener">https://www.dropbox.com/s/mqe3pyme2mnh8o6/PytorchDev-Fixed.zip?dl=0</a>   BTW, PyTorch 1.6 now has been released, so it is supposed to be working as well if you import PyTorch via Cocoapod. Thanks.</p>

username: Bryan_Wang
cooked: <p>Ah, I’ll do that as well. Thank you.</p>