Running Pytorch models on host x86

Marat: <p>Sorry for possibly stupid question. But my question emerges naturally due to lack (in my opinion) of complete step-by-step documentation for release transition process.</p>
<p>To the best of my knowledge the only way to effectively (using GPU + quantization to int8 + some compiler optimizations) execute PyTorch models on host devices is:</p>
<ol>
<li>Make JIT code from PyTorch models</li>
<li>Save  this model in ONNX format on disk</li>
<li>Make an application or use existing to load and run pytorch model in ONNX format saved previously</li>
</ol>
<p>My question is how much code will be autonomous? What libraries and other environment features will it require to run on host x86 server with GPU on the customer side?</p>

jfix: <p>Not stupid at all! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Right now we support the <a href="https://github.com/pytorch/glow/blob/master/docs/Onnxifi.md" rel="nofollow noopener">ONNXIFI interface</a> which allows PyTorch/Caffe2 to use Glow as an execution backend. Through ONNXIFI (technically FOXI as noted there) the model is passed as an ONNX or C2 proto to Glow, loaded/compiled/quantized/etc., and run on one of our backends.</p>
<p>We also have a <a href="https://github.com/pytorch/glow/pull/3069" rel="nofollow noopener">very basic PR up here</a> (has not yet landed) that creates an actual Glow backend for PyTorch, which goes more directly from PyTorch IR to Glow.</p>

Marat: <p>Building GLOW (<a href="https://github.com/pytorch/glow" rel="nofollow noopener">https://github.com/pytorch/glow</a>) is extreme pain which also includes additional pain of building custom llvm (<a href="https://solarianprogrammer.com/2013/01/17/building-clang-libcpp-ubuntu-linux/" rel="nofollow noopener">https://solarianprogrammer.com/2013/01/17/building-clang-libcpp-ubuntu-linux/</a>). I am still trying to build GLOW first I find issue with libpng&amp;zlib next I have problem with protobuf. Are your suggestion simple?</p>

jfix: <p>I’m sorry you’ve had such a bad experience. If you have feedback we’d happily take it to improve developer experience.</p>
<p>If you’re on Ubuntu 18.04 (like the link you posted says), I believe you should be able to use <code>apt-get</code> to install all of our dependencies. Have you followed the <a href="https://github.com/pytorch/glow#ubuntu" rel="nofollow noopener">instructions on our README</a>? It says it’s been tested on 16.04 at least. Or are you building everything manually?</p>
<p>The instructions for installing and using ONNXIFI are on that page, but I haven’t tried it myself. That page does say it can be a little tricky but I do not know the details there. If you try and run into issues you can always reach out for help.</p>

Marat: <p>I have the following issue with protobuf</p>
<pre><code class="lang-auto">lib/Importer/libImporter.a(caffe2.pb.cc.o):(.data.rel.ro+0xaf0): undefined reference to `google::protobuf::Message::InitializationErrorString[abi:cxx11]() const'
lib/Importer/libImporter.a(caffe2.pb.cc.o):(.data.rel.ro+0xb90): undefined reference to `google::protobuf::Message::GetTypeName[abi:cxx11]() const'
lib/Importer/libImporter.a(caffe2.pb.cc.o):(.data.rel.ro+0xbc8): undefined reference to `google::protobuf::Message::InitializationErrorString[abi:cxx11]() const'
lib/Importer/libImporter.a(caffe2.pb.cc.o):(.data.rel.ro+0xc68): undefined reference to `google::protobuf::Message::GetTypeName[abi:cxx11]() const'
lib/Importer/libImporter.a(caffe2.pb.cc.o):(.data.rel.ro+0xca0): undefined reference to `google::protobuf::Message::InitializationErrorString[abi:cxx11]() const'
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
[214/282] Linking CXX executable bin/char-rnn
ninja: build stopped: subcommand failed.
</code></pre>
<p>And I have following version of protobuf it is 3.6.1 which is &gt;= 2.6.1</p>
<pre><code class="lang-auto">marat@moon:~/glow/build_Debug$ which protoc
/home/marat/anaconda3/bin/protoc
marat@moon:~/glow/build_Debug$ protoc --version
libprotoc 3.6.1
marat@moon:~/glow/build_Debug$
</code></pre>
<p>I also will be very appreciated if you say where did you take llvm-8 because I failed to find it by apt in my ubuntu 16.04</p>
<p>UPDATE</p>
<p>I removed anaconda form $PATH and it helped. But I do not know how it will work: pytorch with version of 3.6.1 and glow compiled with 2.6.1. So it seems that problem on compilation stage accrued due to libraries versions mismatch .</p>

Marat: <p>I also trying to build GLOW with</p>
<pre><code class="lang-auto">cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DGLOW_WITH_CPU=1 -DGLOW_WITH_OPENCL=1 ..
</code></pre>
<p>But recently have some problems</p>
<pre><code class="lang-auto">CMake Error at lib/Backends/OpenCL/CMakeLists.txt:32 (add_library):
  Target "OpenCLBackend" links to target "OpenCL::OpenCL" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?

</code></pre>
<p>That I supposed to do?</p>
<p>I believe that problem come from</p>
<pre><code class="lang-auto">target_link_libraries(OpenCLBackend
                      PUBLIC
                      OpenCL::OpenCL)

</code></pre>
<p>from</p>
<pre><code class="lang-auto">lib/Backends/OpenCL/CMakeLists.txt
</code></pre>
<p>find_package finds OpenCL successfully (no error messages about that)</p>
<p>That is  “OpenCL::OpenCL” ?</p>
<p>UPDATE</p>
<p>This issue solved by using more recent cmake, <strong>cmake 3.5.2 DID NOT WORK</strong> please update cmake minimal required</p>

jfix: <p>Thanks for the feedback! Would you mind posting these problems as GH issues via <a href="https://github.com/pytorch/glow/issues/new" rel="nofollow noopener">this link</a>? We will be sure to fix them.</p>