(libtorch) Save MNIST c++ example's trained model into a file, and load in from another c++ file to use for prediction?

username: tancl
cooked: <p>Hi, I was trying to explore how to train the mnist model in C++, save the model, and having another C++ to load the file and use it as inference system.</p>
<p>I tried the methods in <a href="https://discuss.pytorch.org/t/libtorch-how-to-save-model-in-mnist-cpp-example/34234" class="inline-onebox">(libtorch) How to save model in MNIST cpp example?</a>,</p>
<ol>
<li>Using original mnist.cpp, add 3 lines of codes to save the model:</li>
</ol>
<pre><code class="lang-auto">     torch::serialize::OutputArchive output_archive;
     model.save(output_archive);
     output_archive.save_to("model.pt");
</code></pre>
<ol start="2">
<li>Change the Net by NetImpl as suggested, and save the mode with:</li>
</ol>
<pre><code class="lang-auto">     torch::save(model, "model.pt");
</code></pre>
<p>with both, I am able to compile and run the code and save the model, however, when I am trying to load the model from another C++ file, I get the error of “‘ScriptModule’ object has no attribute ‘forward’”</p>
<p>To simplified the testing, I tried to load the model in Python:</p>
<ol>
<li>Load with torch.load</li>
</ol>
<pre><code class="lang-auto">import torch
# load by torch.load
model = torch.load('model.pt')
## Error loading 
## RuntimeError: model_impl.pt is a zip archive (did you mean to use torch.jit.load()?)
</code></pre>
<ol start="2">
<li>Load by torch.jit.load</li>
</ol>
<pre><code class="lang-auto"># Loaded successfully, but...
model = torch.jit.load('model.pt')
model.eval()

# Python output
ScriptModule(
  (conv1): ScriptModule()
  (conv2): ScriptModule()
  (conv2_drop): ScriptModule()
  (fc1): ScriptModule()
  (fc2): ScriptModule()
)

# When try to have a forward pass, I get the following error.
output = model(torch.ones(1, 1, 28, 28))

## Error : AttributeError: 'ScriptModule' object has no attribute 'forward'
</code></pre>
<p>Apologize if I’ve miss anything else.</p>
<p>Thanks.</p>
<p>Regard,<br>
CL</p>

username: soldierofhell
cooked: <p>You probably use torch::jit::load, but should torch::load</p>

username: tancl
cooked: <p>Hi, thanks for your reply. However, as mentioned in my post, when I use torch::load, I get the error :</p>
<p>Error loading<br>
RuntimeError: model_impl.pt is a zip archive (did you mean to use torch.jit.load()?)</p>

username: soldierofhell
cooked: <p>But this error is from python, not C++</p>

username: tancl
cooked: <p>Hi, Thanks for pointing this out. I thought the behavior are identical but it seems like not.</p>
<p>I tried the following simple lines:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
struct NetImpl : torch::nn::Module {};
TORCH_MODULE(Net);


int main() {
	Net model;	
	torch::load(model, "net.pt");
	auto in = torch::rand({1, 1, 28, 28});
	auto out = model-&gt;forward(in);
    std::cout &lt;&lt; in &lt;&lt; std::endl;
    std::cout &lt;&lt; out &lt;&lt; std::endl;
return 0;
}	
</code></pre>
<p>the compilation gives following error:</p>
<pre><code class="lang-auto">error: ‘struct NetImpl’ has no member named ‘forward’
auto out = model-&gt;forward(in);

</code></pre>
<p>I’ve to define the NetImpl which I used for training on top of the codes to make it works, it seems like the torch::save will only save the parameters but not the network structure? Am I correct?</p>
<p>If so, is there anyway to save everything in C++ so that I could call it directly?</p>
<p>Thanks again.</p>

username: Ziri_Ziri
cooked: <p>Hi everyone,<br>
I am facing the same problem. I created simple cnn  using sequential implementation (torch::nn::SequentialImpl )  i can  actually use   model-&gt;forward(Sometensor) but it crashes when i save the model and load it again with jit.</p>
<p>The root of the problem is unclear since i am getting an  unhandled exception  .</p>
<p>I opened an issue here : <a href="https://github.com/pytorch/pytorch/issues/25142" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25142</a>.</p>

username: tancl
cooked: <p>Hi, I saw that you have this problem solve with torch::save and torch::load, could you share your example on this?</p>
<p>I was using the mnist example, after saving the model using torch::save, in another C++ file, I’ve to define the same model on top of the file before I could use the torch::load.</p>
<p>Thanks.</p>
<p>Regards,<br>
Chin Luh</p>

username: Ziri_Ziri
cooked: <p>My network is a simple CNN like this :</p>
<pre><code class="lang-auto">
	struct NetworkImpl : torch::nn::SequentialImpl {
		NetworkImpl() {


			// Network here 

		};

		TORCH_MODULE(Network);
</code></pre>
<p>while training in loop i save the model like this :</p>
<pre><code class="lang-auto">   for (size_t i = 0; i &lt; options.iterations; ++i) {

   	train(network, *train_loader, optimizer, i + 1, train_size);

   	test(network, *test_loader, test_size);

       /*Save model */
   	torch::save(network, "Path_to_modelSaveFolder\\model.pt");

   }
</code></pre>
<p>then i load it :</p>
<pre><code class="lang-auto">Network net;
torch::load(net,model_path);
</code></pre>
<p>let me know if you still need help.</p>

username: tancl
cooked: <p>Hi, thanks for your reply. Are you performing the saving and loading in the same C++ file? or separate files?</p>
<p>Regards,<br>
Chin Luh</p>

username: yf225
cooked: <p><a class="mention" href="/u/tancl">@tancl</a> There are a few scenarios that C++/TorchScript serialization supports:</p>
<ul>
<li>Save as C++ model, load using torch::load() in C++
<ul>
<li>Requirement: <strong>You need to have the same C++ model class definition available when you use <code>torch::save</code> and <code>torch::load</code>.</strong> The easiest way to achieve this is to put the model class definition in a common header file.</li>
</ul>
</li>
<li>If you want to be able to debug the model in Python, the suggested way is to define the model in Python (and perform debugging), convert it to TorchScript, and then load the model using <code>torch::jit::load</code> in C++ (for details on this process, see <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a>).</li>
</ul>
<p>These are scenarios that C++/TorchScript serialization doesn’t support:</p>
<ul>
<li>Save as C++ model using <code>torch::save</code>, load using <code>torch::jit::load</code> in C++</li>
<li>Save as C++ model using <code>torch::save</code>, load using <code>torch.load</code> in Python</li>
<li>Save as C++ model using <code>torch::save</code>, load using <code>torch.jit.load</code> in Python</li>
</ul>

username: Ziri_Ziri
cooked: <p>Obviously if you save a model you’ll need to load the same one. As <a class="mention" href="/u/yf225">@yf225</a> mentioned , put the model in common header file.</p>

username: tancl
cooked: <p>Hi,</p>
<p>Thanks for the details <a class="mention" href="/u/yf225">@yf225</a> and <a class="mention" href="/u/ziri_ziri">@Ziri_Ziri</a>.</p>
<p>My intention is to integrate libtorch with Scilab, in which the user could define their own model (in any form, different conv layers, etc) and parse in a scilab gateway (C++) and save it into a model which have the information of the network architecture. The model then will be train in next gateway C++ and save it into another model with the trained parameters again. Finally, the trained model will be called in another gateway C++ for inferencing. I am not too sure whether this could be done, as from your explanation, it goes consistence with the documentation, so i was trying my luck if there is any undocumented way or idea on how to do this.</p>
<p>Thanks again.</p>
<p>Regards,<br>
Chin Luh</p>

username: yf225
cooked: <p><a class="mention" href="/u/tancl">@tancl</a> Thanks a lot for the use case information and it’s really helpful. I am thinking of two possible options:<br>
Option 1 - Ask the user to define their C++ model outside of Scilab and in a common header file, and then all Scilab gateways can use this header file to find the definition of the C++ model.<br>
Option 2 - Ask the user to define their TorchScript model outside of Scilab and serialize the model, and then the Scilab gateways can load this TorchScript model and run training / inference with it.</p>
<p>Please let me know if any of these two options would work.</p>

username: tancl
cooked: <p>Hi, thanks for the prompt reply.</p>
<p>I will have C++ gateways to call the libtorch, compiled and link so that it becomes a native function in Scilab. In user end, they will just call sth like : trained_model = torch_train(data, target, model_arch, configs) .</p>
<p>Option 1 - Correct me if I am wrong but  this will require the recompiling the codes every time a model being define in header, so It might be not flexible, unless the model could be parse as the input to the function.</p>
<p>Option 2 - I will explore more of this options, I remember there are some changes in 1.1 and 1.2, will try out this on 1.2 and get back to you on this. My previous test on mnist examples giving error sth like “forward method not define” during runtime, will confirm this again.</p>
<p>Again, thanks for your details reply and useful suggestions.</p>
<p>Regards,<br>
Chin Luh</p>

username: tancl
cooked: <p>Hi,</p>
<p>I was trying the option 2 by using the mnist example, my steps are:</p>
<ol>
<li>Building/Training a model in Python-pyTorch using the python mnist example and save it into torch script using the script compiler method.</li>
</ol>
<pre><code class="lang-auto"># Using the example from https://github.com/pytorch/examples/tree/master/mnist/main.py with following modification
    if (args.save_model):
        my_model = torch.jit.script(model)
        my_model.save("mymodel.pt")

</code></pre>
<ol start="2">
<li>Using the model for inference works fine in C++:</li>
</ol>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;iostream&gt;

int main() {
	//Net model;	
	torch::jit::script::Module model;
	std::string module_path = "mymodel.pt";
	model=  torch::jit::load(module_path);
	
	// Create a vector of inputs.
	std::vector&lt;torch::jit::IValue&gt; inputs;
	inputs.push_back(torch::ones({1, 1, 28, 28}));

	// Execute the model and turn its output into a tensor.
	at::Tensor output = model.forward(inputs).toTensor();
	std::cout &lt;&lt; output &lt;&lt; std::endl;	
    
return 0;
}	
</code></pre>
<ol start="3">
<li>However, I was facing difficulty when I wanted to use the model and train in C++</li>
</ol>
<pre><code class="lang-auto">// Using the example from https://github.com/pytorch/examples/blob/master/cpp/mnist/mnist.cpp, by removing the net definition block on the beginning of the codes, and loading the model previously trained in python by using jit::load:

  //Net model;
  //model.to(device);
 
  torch::jit::script::Module model;
  std::string module_path = "mymodel.pt";
  model = torch::jit::load(module_path);  
  model.to(device);

  auto train_dataset = torch::data::datasets::MNIST(kDataRoot)
                           .map(torch::data::transforms::Normalize&lt;&gt;(0.1307, 0.3081))
                           .map(torch::data::transforms::Stack&lt;&gt;());
  const size_t train_dataset_size = train_dataset.size().value();
  auto train_loader =
      torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
          std::move(train_dataset), kTrainBatchSize);

  auto test_dataset = torch::data::datasets::MNIST(
                          kDataRoot, torch::data::datasets::MNIST::Mode::kTest)
                          .map(torch::data::transforms::Normalize&lt;&gt;(0.1307, 0.3081))
                          .map(torch::data::transforms::Stack&lt;&gt;());
  const size_t test_dataset_size = test_dataset.size().value();
  auto test_loader = torch::data::make_data_loader(std::move(test_dataset), kTestBatchSize);

  torch::optim::SGD optimizer(model.parameter, torch::optim::SGDOptions(0.01).momentum(0.5));

  for (size_t epoch = 1; epoch &lt;= kNumberOfEpochs; ++epoch) {
    train(epoch, model, device, *train_loader, optimizer, train_dataset_size);
    test(model, device, *test_loader, test_dataset_size);
  }

</code></pre>
<p>I get error as below:</p>
<pre><code class="lang-auto"> error: ‘struct torch::jit::script::Module’ has no member named ‘parameter’; did you mean ‘set_parameter’?
   torch::optim::SGD optimizer(model.parameter, torch::optim::SGDOptions(0.01).momentum(0.5));
</code></pre>
<p>I read the torch::jit and it did mentioned on defining the nn.parameters could save the attributes, however, how to make this possible in the model definition replacing nn.Conv2d and nn.Linear?</p>
<p>Thanks again in advance.</p>
<p>Regards,<br>
Chin Luh</p>

username: Edwardmark
cooked: <h1>How to save sth in python-api pytorch, but load it in libtorch?</h1>
<p>How can I save some tensor in python, but load it in libtorch:</p>
<p>I save tensor named piror using python, using the code:</p>
<pre><code class="lang-auto">torch.save(prior,  'prior.pth')
</code></pre>
<p>And I load the tensor in libtorch using C++, by the following code:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; tensorVec;
torch::load(tensorVec, "/app/model/prior.pth");
torch::Tensor priors = tensorVec[0];
</code></pre>
<p>But I got the error:<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  <code>torch::jit::load()</code> received a file from <code>torch.save()</code>, but <code>torch::jit::load()</code> can only load files produced by <code>torch.jit.save()</code> (load at …/torch/csrc/jit/serialization/import.cpp:285)</p>
<p>Why is that? And what should I do to solve the issue? Thanks in advance.</p>
<p><a class="mention" href="/u/yf225">@yf225</a></p>

username: MakGulati
cooked: <p><a class="mention" href="/u/yf225">@yf225</a> I have similar doubt : basically how can i use pytorch trained model subsequently in torchlib for fine-tuning on c++ based device given that i have defined the model first in c++ which is obviously same as pytorch model?</p>