How to use torchscript C++ with nn.transformer

username: raabuchanan
cooked: <p>Hi</p>
<p>I’m trying to export a transformer model with torchscript for use in C++.<br>
I am currently passing a dummy input of all zeros for debugging. The first time I call forward in C++ I get an output from the transformer fine. The second time, even though I pass the exact same input, I get the error bellow:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  Expected a list element that subtypes 'Tensor' but got an element of type 'Optional[Tensor]'
Exception raised from createList at ../torch/csrc/jit/ir/ir.cpp:1660 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x69 (0x7f55dfefeb29 in /usr/local/include/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xd2 (0x7f55dfefbab2 in /usr/local/include/libtorch/lib/libc10.so)
frame #2: torch::jit::Graph::createList(std::shared_ptr&lt;c10::Type&gt; const&amp;, c10::ArrayRef&lt;torch::jit::Value*&gt;) + 0x185 (0x7f5625fa25f5 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #3: &lt;unknown function&gt; + 0x372f824 (0x7f562615d824 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x3732108 (0x7f5626160108 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #5: torch::jit::differentiate(std::shared_ptr&lt;torch::jit::Graph&gt;&amp;) + 0x237 (0x7f5626162497 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x376a3a7 (0x7f56261983a7 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #7: &lt;unknown function&gt; + 0x376b329 (0x7f5626199329 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #8: &lt;unknown function&gt; + 0x376be8e (0x7f5626199e8e in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #9: &lt;unknown function&gt; + 0x373e764 (0x7f562616c764 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #10: torch::jit::GraphFunction::operator()(std::vector&lt;c10::IValue, std::allocator&lt;c10::IValue&gt; &gt;, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, c10::IValue, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, c10::IValue&gt; &gt; &gt; const&amp;) + 0x3a (0x7f5625e8725a in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #11: torch::jit::Method::operator()(std::vector&lt;c10::IValue, std::allocator&lt;c10::IValue&gt; &gt;, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, c10::IValue, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, c10::IValue&gt; &gt; &gt; const&amp;) + 0x16a (0x7f5625e9670a in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #12: torch::jit::Module::forward(std::vector&lt;c10::IValue, std::allocator&lt;c10::IValue&gt; &gt;) + 0x131 (0x7f5635e8ab01 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #13: vilens::TorchModelWrapper::getBiasTransformer(Eigen::Matrix&lt;double, -1, -1, 0, -1, -1&gt; const&amp;, Eigen::Matrix&lt;double, -1, -1, 0, -1, -1&gt; const&amp;, Eigen::Matrix&lt;double, -1, 1, 0, -1, 1&gt;&amp;) + 0x665 (0x7f5635e8e5d5 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #14: vilens::ImuBiasObserverHandler::createFactor(vilens::State const&amp;, gtsam::Values const&amp;, vilens::FGNodes const&amp;, gtsam::NonlinearFactorGraph&amp;, gtsam::Values&amp;, bool) + 0x4238 (0x7f5635e85808 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #15: vilens::VilensOptimizer::addFactors(std::vector&lt;std::unique_ptr&lt;vilens::Handler, std::default_delete&lt;vilens::Handler&gt; &gt;, std::allocator&lt;std::unique_ptr&lt;vilens::Handler, std::default_delete&lt;vilens::Handler&gt; &gt; &gt; &gt;&amp;, vilens::StateMap&amp;) + 0xfe4 (0x7f5635d49774 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #16: vilens::VilensManager::optimize() + 0x5e (0x7f5635f68c0e in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #17: vilens::VilensManager::optimizer() + 0xbc5 (0x7f5635f6fa85 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #18: &lt;unknown function&gt; + 0xda6b4 (0x7f55dfb6b6b4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #19: &lt;unknown function&gt; + 0x8609 (0x7f56355f3609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #20: clone + 0x43 (0x7f55df9a1133 in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>I am doing the tracing as below:</p>
<pre><code class="lang-auto">    def torch_script_trace(self, module, imu_window_time):
        # Tracing
        exampleUInput = torch.rand(1, 6, 100)
        exampleUInputCuda = exampleUInput.cuda()
        exampleYInput = torch.rand(1, 6, 1)
        exampleYInputCuda = exampleYInput.cuda()

        traced_script_module = torch.jit.trace(
            module, (exampleUInputCuda, exampleYInputCuda))
        return traced_script_module
</code></pre>
<p>I have read this tutorial on Huggingface which uses a torchscript flag but I don’t see a similar thing in nn.transformer. Is there some configuration I am missing to use torchscript with transformers?<br>
Huggingface tutorial: <a href="https://huggingface.co/transformers/v3.1.0/serialization.html" class="inline-onebox" rel="noopener nofollow ugc">Exporting transformers models — transformers 3.1.0 documentation</a></p>
<p>Thank you</p>

username: raabuchanan
cooked: <p>So I eventually resolved this with two changes:</p>
<ol>
<li>Instead of tracing I used annotations <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#converting-to-torch-script-via-annotation" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 1.11.0+cu102 documentation</a>
</li>
<li>I updated to the most recent libtorch. I forget what version I had before but it seems older versions did not support annotations which I think are necessary for transformers.</li>
</ol>

username: varunagrawal
cooked: <p>We had version 1.8. That seems to be key, ensuring that Pytorch and LibTorch have the same (or at least compatible) versions.</p>