In C++ Exporting and Loading Torch Module as iostream fails

Stefan1: <p>Dear all,</p>
<p>I have a problem which I cannot solve. I develop a program in C++ which uses a PyTorch model. I have no clue where to start debugging this, I hope somebody can help me or give me some input so I can continue my search for a solution.</p>
<p>Is this a bug in LibTorch, or am I doing something wrong?</p>
<p><strong>Description</strong><br>
I want to Export my Torch Module and write it using a <code>std::ostream</code>, then I want to read my exported Torch Module using a <code>std::istream</code>. It does not work. Below I give some details:</p>
<p>For clearity, I try to use the following 2 functions:<br>
<em>(from …\include\torch\csrc\jit\serialization\export.h)</em></p>
<pre><code class="lang-auto">TORCH_API void ExportModule(
    const Module&amp; module,
    std::ostream&amp; out,
    const ExtraFilesMap&amp; metadata = ExtraFilesMap(),
    bool bytecode_format = false,
    bool save_mobile_debug_info = false);
</code></pre>
<p><em>(from …\include\torch\csrc\jit\serialization\import.h)</em></p>
<pre><code class="lang-auto">/// Loads a serialized `Module` from the given `istream`.
///
/// The istream must contain a serialized `Module`, exported via
/// `torch::jit::ExportModule` in C++.
TORCH_API Module
load(std::istream&amp; in, c10::optional&lt;c10::Device&gt; device = c10::nullopt);
</code></pre>
<p><strong>To reproduce</strong><br>
To test, I developed a small test script.</p>
<ul>
<li>In Python I have developed and trained a model. I Exported this using <code>Torch.jit.trace(...).save(...)</code>.</li>
<li>Then, I load this TorchScript model successfully in C++ using <code>torch::jit::load(file_path)</code> (so not using the <code>std::istream</code> overload.).</li>
<li>Then, for testing purposes, I export it again using the above denoted <code>torch::jit::ExportModule(module, ostream_out)</code>.</li>
<li>Then, for testing purposes, I import it again using the above denoted <code>module = torch::jit::load(istream_in)</code>
</li>
</ul>
<p>The code is below:</p>
<pre><code class="lang-auto">// Path to file
torchScriptFileFromPython = "C:/Test/MyModule.pte";


// Deserialize the ScriptModule from file as saved in Python (this works flawlessly).
std::cout &lt;&lt; "Loading Torch Script model...\n\n";
torch::jit::script::Module module_python;
try {
	module_python = torch::jit::load(torchScriptFileFromPython);
}
catch (const c10::Error&amp; e) {
	std::cerr &lt;&lt; "Error during loading the model:\n\n";
	std::cerr &lt;&lt; e.msg() &lt;&lt; std::endl &lt;&lt; std::endl;
	std::cerr &lt;&lt; e.backtrace() &lt;&lt; std::endl &lt;&lt; std::endl;
	return -1;
}
std::cout &lt;&lt; "Model succesfully loaded!\n\n";


// Stream the module out (this code runs but I think it creates a corrupted file).
std::string streamedFile = "C:/Test/MyModuleStreamed.pte";
{
	std::filebuf outfb;
	outfb.open(streamedFile, std::ios::out);
	std::ostream os(&amp;outfb);
	try {
		torch::jit::ExportModule(module_python, os);
		// torch::jit::ExportModule(module_python, streamedFile);  // Suppose I would use this line instead of the previous, it would work. However, I want to use the ostream.
	}
	catch (const c10::Error&amp; e) {
		std::cerr &lt;&lt; "Error during export streaming the model:\n\n";
		std::cerr &lt;&lt; e.msg() &lt;&lt; std::endl &lt;&lt; std::endl;
		std::cerr &lt;&lt; e.backtrace() &lt;&lt; std::endl &lt;&lt; std::endl;
		return -1;
	}
	outfb.close();
	std::cout &lt;&lt; "Model succesfully export streamed!\n\n";
}


// Stream the module in (this part raises an exception).
torch::jit::script::Module module;
{
	std::filebuf infb;
	infb.open(streamedFile, std::ios::in);
	std::istream is(&amp;infb);
	try {
		module = torch::jit::load(is);
		//module = torch::jit::load(streamedFile);  // Suppose I would use this line instead of the previous, it would work, provided I did the same in the ExportModule part. However, I want to use the istream.
	}
	catch (const c10::Error&amp; e) {
		std::cerr &lt;&lt; "Error during import streaming the model:\n\n";
		std::cerr &lt;&lt; e.msg() &lt;&lt; std::endl &lt;&lt; std::endl;
		std::cerr &lt;&lt; e.backtrace() &lt;&lt; std::endl &lt;&lt; std::endl;
		return -1;
	}
	infb.close();
	std::cout &lt;&lt; "Model succesfully import streamed!\n\n"; 
}
</code></pre>
<p>As denoted in the code comments, if instead of the <code>istream</code> / <code>ostream</code> versions, I use the overloads which accepts a <code>std::string</code> containing the path, it works. I want the stream however to manipulate the file before writing and undo this when reading.</p>
<p>The relevant headers I use:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;torch/csrc/jit/serialization/export.h&gt;
#include &lt;iostream&gt;
#include &lt;string&gt;
#include &lt;ostream&gt;
#include &lt;istream&gt;
</code></pre>
<p><strong>The Exception</strong><br>
The Exception I get when using the <code>istream</code> / <code>ostream</code> versions is:</p>
<pre><code class="lang-auto">istream reader failed: reading file.
Exception raised from validate at ..\..\caffe2\serialize\istream_adapter.cc:32 (most recent call first):
00007FFDA78C4A2A00007FFDA78C3AF0 c10.dll!c10::detail::LogAPIUsageFakeReturn [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFDA78C458A00007FFDA78C3AF0 c10.dll!c10::detail::LogAPIUsageFakeReturn [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFDA78C578100007FFDA78C3AF0 c10.dll!c10::detail::LogAPIUsageFakeReturn [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFDA78C53F500007FFDA78C3AF0 c10.dll!c10::detail::LogAPIUsageFakeReturn [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFDA78C2FAF00007FFDA78C2F40 c10.dll!c10::Error::Error [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFDA78C1B3600007FFDA78C1A70 c10.dll!c10::detail::torchCheckFail [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350ECD0800007FFD350ECBB0 torch_cpu.dll!caffe2::serialize::IStreamAdapter::validate [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350ECB4200007FFD350ECA80 torch_cpu.dll!caffe2::serialize::IStreamAdapter::read [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E7BDE00007FFD350E7B70 torch_cpu.dll!caffe2::serialize::PyTorchStreamReader::read [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E7EE000007FFD350E7D00 torch_cpu.dll!caffe2::serialize::PyTorchStreamReader::getRecordID [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E24CD00007FFD350C2DD0 torch_cpu.dll!caffe2::Workspace::bookkeeper [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E280300007FFD350C2DD0 torch_cpu.dll!caffe2::Workspace::bookkeeper [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350D304A00007FFD350C2DD0 torch_cpu.dll!caffe2::Workspace::bookkeeper [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E74C100007FFD350E70D0 torch_cpu.dll!caffe2::serialize::PyTorchStreamReader::init [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD350E68E900007FFD350E6850 torch_cpu.dll!caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD37A044FC00007FFD379FFCB0 torch_cpu.dll!torch::jit::readArchiveAndTensors [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD37A021B600007FFD379FFCB0 torch_cpu.dll!torch::jit::readArchiveAndTensors [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD37A0BC2200007FFD379FFCB0 torch_cpu.dll!torch::jit::readArchiveAndTensors [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD379FFB4F00007FFD379FF9B0 torch_cpu.dll!torch::jit::load [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD379FF61500007FFD379FF530 torch_cpu.dll!torch::jit::load [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFD379FF4E100007FFD379FF460 torch_cpu.dll!torch::jit::load [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FF775C039C700007FF775C03600 LibTorchTest.exe!main [D:\Default_Folders\Documents\Development\RepoStefan\PyTorchNetworkv1\LibTorchTest\src\main.cpp @ 86]
00007FF775C0606900007FF775C06030 LibTorchTest.exe!invoke_main [D:\agent\_work\13\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79]
00007FF775C05F0E00007FF775C05DE0 LibTorchTest.exe!__scrt_common_main_seh [D:\agent\_work\13\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
00007FF775C05DCE00007FF775C05DC0 LibTorchTest.exe!__scrt_common_main [D:\agent\_work\13\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 331]
00007FF775C060FE00007FF775C060F0 LibTorchTest.exe!mainCRTStartup [D:\agent\_work\13\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp @ 17]
00007FFE0296703400007FFE02967020 KERNEL32.DLL!BaseThreadInitThunk [&lt;unknown file&gt; @ &lt;unknown line number&gt;]
00007FFE0300265100007FFE03002630 ntdll.dll!RtlUserThreadStart [&lt;unknown file&gt; @ &lt;unknown line number&gt;]

</code></pre>
<p>However, I think the export already writes a corrupted file.<br>
When using the <code>istream</code> approach, I get the same error if I directly try to import the file which was exported by Python.</p>
<p><strong>Expected behavior</strong><br>
The model is exported to file and loaded without errors and then I can use the loaded model to successfully call <code>module.forward(inputs)</code>.</p>
<p><strong>Environment</strong><br>
I’m on Windows 10. The problems occurs both when I build in debug and release mode.<br>
I use the LTS version of LibTorch, version 1.8.1 for CPU only. I also use PyTorch version 1.8.1 in Python.<br>
For now, the model is exported with everything on CPU.</p>
<p><strong>Additional context</strong><br>
Why do I want to use streams to export/load the module?<br>
I need to deploy our solution at a customer. The customer should not be able to simply get the model file, load it and see our network including all trained weights. Therefore, when writing/reading the streams, I want to apply some form of encryption so at least the file on disk is encrypted.</p>
<p>Any other approach to achieve this would be very welcome. However, it will not solve this problem I consider a bug.</p>

dfalbel: <p>Maybe you want to take a look at this post:</p>
<aside class="quote quote-modified" data-post="1" data-topic="93130">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dfalbel/48/8880_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/reading-tensors-in-c-saved-with-torch-save/93130">Reading tensors in C++ saved with `torch.save`</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I am trying to read tensors saved with torch.save() from C++. I tried reproducing this test case: 


That is reading a file created with: 


I modified the test case example to: 
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main()
{
  std::ifstream input_stream("ivalue.pt");
    std::vector&lt;char&gt; input;
    input.insert(
        input.begin(),
        std::istream_iterator&lt;char&gt;(input_stream),
        std::istream_iterator&lt;char&gt;());
    std::cout &lt;&lt; input.size() &lt;&lt; std::endl;
    torch::IVa…
  </blockquote>
</aside>

<p>I had similar issues some time ago and using binary streams fixed the issue.</p>

Stefan1: <p>Thanks!<br>
Last night, I figured this out too. I might be good to add this note to the header although it is kind of logical.</p>
<p>I’m rather new to C++ so I have to figure out some of these things.</p>
<p>For future readers, replace the two lines:</p>
<pre><code class="lang-auto">outfb.open(streamedFile, std::ios::out);
outfb.open(streamedFile, std::ios::in);
</code></pre>
<p>by:</p>
<pre><code class="lang-auto">outfb.open(streamedFile, std::ios::binary | std::ios::out);
outfb.open(streamedFile, std::ios::binary | std::ios::in);
</code></pre>
<p>and it works.</p>
<p>A file is opened in text mode by default and then, on windows, everytime a byte passes which corresponds to the ‘\n’ character, it is replaced by two bytes ‘\r\n’ causing a sort of corrupt file.</p>
<p>How do I close this topic? Or somebody can close it for me.</p>

Steve25: <p>Hi, do you mean infb.open(streamedFile, std::ios::binary | std::ios::in); not  outfb.open(streamedFile, std::ios::binary | std::ios::in);</p>