Error in loading the model

krrishabh: <p>When I make a small project to check. It works fine… But as soon as I put the same thing in the Kaldi (<a href="https://github.com/cyfer0618/kaldi-pytorch-rnnlm/blob/master/kaldi/src/pyrnnlm/pytorch-rnnlm.cc" rel="nofollow noopener">Pytorch lattice rescoring</a>) and run the executable file then I receive the same error as mentioned below.</p>
<p><strong>Error</strong></p>
<pre><code class="lang-auto">data/pytorch/rnnlm/newmodel2.pt error loading the model
_ivalue_ INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/api/object.cpp":19, please report a bug to PyTorch.  (_ivalue at /pytorch/torch/csrc/jit/api/object.cpp:19)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x46 (0x7f83b939d666 in /home/rakesh/rishabh_workspace/Garbage/kaldi/tools/libtorch/lib/libc10.so)
frame #1: torch::jit::Object::_ivalue() const + 0xab (0x7f83acdc42cb in /home/rakesh/rishabh_workspace/Garbage/kaldi/tools/libtorch/lib/libtorch_cpu.so)
frame #2: torch::jit::Object::find_method(std::string const&amp;) const + 0x26 (0x7f83acdc43a6 in /home/rakesh/rishabh_workspace/Garbage/kaldi/tools/libtorch/lib/libtorch_cpu.so)
frame #3: &lt;unknown function&gt; + 0x9192f (0x5592eb17892f in lattice-lmrescore-py-rnnlm)
frame #4: &lt;unknown function&gt; + 0x86b78 (0x5592eb16db78 in lattice-lmrescore-py-rnnlm)
frame #5: &lt;unknown function&gt; + 0x867d8 (0x5592eb16d7d8 in lattice-lmrescore-py-rnnlm)
frame #6: &lt;unknown function&gt; + 0x23c38 (0x5592eb10ac38 in lattice-lmrescore-py-rnnlm)
frame #7: __libc_start_main + 0xe7 (0x7f836b610b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: &lt;unknown function&gt; + 0x2340a (0x5592eb10a40a in lattice-lmrescore-py-rnnlm)
</code></pre>
<p>pyrnnlm.cc</p>
<pre><code class="lang-auto">


#include &lt;utility&gt;
#include &lt;fstream&gt;

#include "pyrnnlm/pytorch-rnnlm.h"
#include "util/stl-utils.h"
#include "util/text-utils.h"

// torch::Tensorflow includes were moved after tfrnnlm/tensorflow-rnnlm.h include to
// avoid macro redefinitions. See also the note in tfrnnlm/tensorflow-rnnlm.h.
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;dirent.h&gt;

namespace kaldi {
using std::ifstream;
using py_rnnlm::KaldiPyRnnlmWrapper;
using py_rnnlm::PyRnnlmDeterministicFst;


// read a unigram count file of the OOSs and generate extra OOS costs for words
void SetUnkPenalties(const string &amp;filename,
                     const fst::SymbolTable&amp; fst_word_symbols,
                     std::vector&lt;float&gt; *out) {
  if (filename == "")
    return;
  out-&gt;resize(fst_word_symbols.NumSymbols(), 0);  // default is 0
  ifstream ifile(filename.c_str());
  string word;
  float count, total_count = 0;
  while (ifile &gt;&gt; word &gt;&gt; count) {
    int id = fst_word_symbols.Find(word);
    KALDI_ASSERT(id != -1); // fst::kNoSymbol
    (*out)[id] = count;
    total_count += count;
  }

  for (int i = 0; i &lt; out-&gt;size(); i++) {
    if ((*out)[i] != 0) {
      (*out)[i] = log ((*out)[i] / total_count);
    }
  }
}

// Read pytorch model files
// Done ****
void KaldiPyRnnlmWrapper::ReadPyModel(const std::string &amp;py_model_path,
                                      int32 num_threads) {

  // Need to initialise it
  // torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    //std::cout &lt;&lt; "Model " &lt;&lt; py_model_path.substr(0, py_model_path.size()-1) &lt;&lt; " /newmodel2.pt";
    std::string file("/newmodel2.pt");
    // Load model in the module
    std::string path = py_model_path + file;

    std::string::iterator st = std::remove(path.begin(), path.end(), ' ');

    path.erase(st, path.end());
    std::cout &lt;&lt; path;
    module = torch::jit::load(path);
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; " error loading the model\n";
    //return -1;
    return;
  }

  std::cout &lt;&lt; "Language Model\n\n";

  // (Samrat): Think we need few of these, not all
  word_id_tensor_name_ = "word_id";
  context_tensor_name_ = "context";
  log_prob_tensor_name_ = "log_prob";
  rnn_out_tensor_name_ = "rnn_out";
  rnn_states_tensor_name_ = "rnn_states";
  initial_state_tensor_name_ = "initial_state";
  
}

// Done ****
// Batch_size = 1 they have hard code it
KaldiPyRnnlmWrapper::KaldiPyRnnlmWrapper(
    const KaldiPyRnnlmWrapperOpts &amp;opts,
    const std::string &amp;rnn_wordlist,
    const std::string &amp;word_symbol_table_rxfilename,
    const std::string &amp;unk_prob_file,
    const std::string &amp;py_model_path): opts_(opts) {
  ReadPyModel(py_model_path, opts.num_threads);

  fst::SymbolTable *fst_word_symbols = NULL;
  if (!(fst_word_symbols =
        fst::SymbolTable::ReadText(word_symbol_table_rxfilename))) {
    KALDI_ERR &lt;&lt; "Could not read symbol table from file "
        &lt;&lt; word_symbol_table_rxfilename;
  }

  fst_label_to_word_.resize(fst_word_symbols-&gt;NumSymbols());

  for (int32 i = 0; i &lt; fst_label_to_word_.size(); ++i) {
    fst_label_to_word_[i] = fst_word_symbols-&gt;Find(i);
    if (fst_label_to_word_[i] == "") {
      KALDI_ERR &lt;&lt; "Could not find word for integer " &lt;&lt; i &lt;&lt; " in the word "
          &lt;&lt; "symbol table, mismatched symbol table or you have discoutinuous "
          &lt;&lt; "integers in your symbol table?";
    }
  }

  // first put all -1's; will check later
  fst_label_to_rnn_label_.resize(fst_word_symbols-&gt;NumSymbols(), -1);
  num_total_words = fst_word_symbols-&gt;NumSymbols();

  // read rnn wordlist and then generate ngram-label-to-rnn-label map
  oos_ = -1;
  { // input.
    ifstream ifile(rnn_wordlist.c_str());
    string word;
    int id = -1;
    eos_ = 0;
    while (ifile &gt;&gt; word) {
      id++;
      rnn_label_to_word_.push_back(word);  // vector[i] = word

      int fst_label = fst_word_symbols-&gt;Find(word);
      if (fst_label == -1) { // fst::kNoSymbol
        if (id == eos_)
          continue;

        KALDI_ASSERT(word == opts_.unk_symbol &amp;&amp; oos_ == -1);
        oos_ = id;
        continue;
      }
      KALDI_ASSERT(fst_label &gt;= 0);
      fst_label_to_rnn_label_[fst_label] = id;
    }
  }
  if (fst_label_to_word_.size() &gt; rnn_label_to_word_.size()) {
    KALDI_ASSERT(oos_ != -1);
  }
  num_rnn_words = rnn_label_to_word_.size();

  // we must have an oos symbol in the wordlist
  if (oos_ == -1)
    return;

  for (int i = 0; i &lt; fst_label_to_rnn_label_.size(); i++) {
    if (fst_label_to_rnn_label_[i] == -1) {
      fst_label_to_rnn_label_[i] = oos_;
    }
  }

  AcquireInitialTensors();
  SetUnkPenalties(unk_prob_file, *fst_word_symbols, &amp;unk_costs_);
  delete fst_word_symbols;
}

KaldiPyRnnlmWrapper::~KaldiPyRnnlmWrapper() {
}
// Done
 
void KaldiPyRnnlmWrapper::AcquireInitialTensors() {
  // Status status;
  // get the initial context; this is basically the all-0 tensor
  /*
  (Samrat): Have to figure out get_initial_state(batch_size) ? what should btchsz be ?
  */
  //auto hidden = module.get_method("get_initial_state")({torch::tensor({1})});
  //initial_context_ = hidden.toTensor();

  initial_context_=module.get_method("get_initial_state")({torch::tensor({1})}).toTensor();


  //changed function call name (Samrat)
  auto bosword = torch::tensor({eos_});

  auto hidden = module.get_method("single_step_rnn_out")({initial_context_, bosword});
  initial_cell_ = hidden.toTensor();




  // {
  //   std::vector&lt;torch::Tensor&gt; state;
  //   status = bundle_.session-&gt;Run(std::vector&lt;std::pair&lt;string, torch::Tensor&gt; &gt;(),
  //                          {initial_state_tensor_name_}, {}, &amp;state);
  //   if (!status.ok()) {
  //     KALDI_ERR &lt;&lt; status.ToString();
  //   }
  //   initial_context_ = state[0];
  // }

  // get the initial pre-final-affine layer
  // {
  //   std::vector&lt;torch::Tensor&gt; state;
  //   torch::Tensor bosword(tensorflow::DT_INT32, {1, 1});
  //   bosword.scalar&lt;int32&gt;()() = eos_;  // eos_ is more like a sentence boundary

  //   std::vector&lt;std::pair&lt;string, torch::Tensor&gt; &gt; inputs = {
  //     {word_id_tensor_name_, bosword},
  //     {context_tensor_name_, initial_context_},
  //   };

  //   status = bundle_.session-&gt;Run(inputs, {rnn_out_tensor_name_}, {}, &amp;state);
  //   if (!status.ok()) {
  //     KALDI_ERR &lt;&lt; status.ToString();
  //   }
  //   initial_cell_ = state[0];
  // }
}


/*
// Need to change *****
BaseFloat KaldiPyRnnlmWrapper::GetLogProb(int32 word,
                                          int32 fst_word,
                                          const torch::Tensor &amp;context_in,
                                          const torch::Tensor &amp;cell_in,
                                          torch::Tensor *context_out,
                                          torch::Tensor *new_cell) {
  torch::Tensor thisword(torch::Tensor, {1, 1});

  thisword.scalar&lt;int32&gt;()() = word;

  std::vector&lt;torch::Tensor&gt; outputs;

  std::vector&lt;std::pair&lt;string, torch::Tensor&gt; &gt; inputs = {
    {word_id_tensor_name_, thisword},
    {context_tensor_name_, context_in},
  };

  if (context_out != NULL) {
    // The session will initialize the outputs
    // Run the session, evaluating our "c" operation from the graph
    Status status = bundle_.session-&gt;Run(inputs,
        {log_prob_tensor_name_,
         rnn_out_tensor_name_,
         rnn_states_tensor_name_}, {}, &amp;outputs);
    if (!status.ok()) {
      KALDI_ERR &lt;&lt; status.ToString();
    }

    *context_out = outputs[1];
    *new_cell = outputs[2];
  } else {
    // Run the session, evaluating our "c" operation from the graph
    Status status = bundle_.session-&gt;Run(inputs,
        {log_prob_tensor_name_}, {}, &amp;outputs);
    if (!status.ok()) {
      KALDI_ERR &lt;&lt; status.ToString();
    }
  }

  float ans;
  if (word != oos_) {
    ans = outputs[0].scalar&lt;float&gt;()();
  } else {
    if (unk_costs_.size() == 0) {
      ans = outputs[0].scalar&lt;float&gt;()() - log(num_total_words - num_rnn_words);
    } else {
      ans = outputs[0].scalar&lt;float&gt;()() + unk_costs_[fst_word];
    }
  }

  return ans;
}
*/

/*
  Below is my(Samrat) modified version of the above function only. 
  Replace if you think something is incorrect.
*/


BaseFloat KaldiPyRnnlmWrapper::GetLogProb(int32 word,
                                          int32 fst_word,
                                          const torch::Tensor &amp;context_in,
                                          const torch::Tensor &amp;cell_in,
                                          torch::Tensor *context_out,
                                          torch::Tensor *new_cell) {
  //torch::Tensor thisword(torch::Tensor, {1, 1});
  
  //thisword.scalar&lt;int32&gt;()() = word;
  torch::Tensor thisword = torch::tensor({word});


  //std::vector&lt;torch::Tensor&gt; outputs;

  // std::vector&lt;std::pair&lt;string, torch::Tensor&gt; &gt; inputs = {
  //   {word_id_tensor_name_, thisword},
  //   {context_tensor_name_, context_in},
  // };



  auto outputs = module.get_method("single_step")({context_in, thisword});
  if (context_out != NULL) {
    // The session will initialize the outputs
    // Run the session, evaluating our "c" operation from the graph
    // Status status = bundle_.session-&gt;Run(inputs,
    //     {log_prob_tensor_name_,
    //      rnn_out_tensor_name_,
    //      rnn_states_tensor_name_}, {}, &amp;outputs);

    // if (!status.ok()) {
    //   KALDI_ERR &lt;&lt; status.ToString();
    // }

    *context_out = module.get_method("single_step_rnn_out")({context_in, thisword}).toTensor();
    *new_cell = module.get_method("single_step_rnn_state")({context_in, thisword}).toTensor();
  } //else {
    // Run the session, evaluating our "c" operation from the graph
    // Status status = bundle_.session-&gt;Run(inputs,
    //     {log_prob_tensor_name_}, {}, &amp;outputs);
    // if (!status.ok()) {
    //   KALDI_ERR &lt;&lt; status.ToString();
    // }
  //}

  /*
    (Samrat): Can through error so have to check manually in testLM
    Hopefully expect it to return a float
  */
 
  float log_prob=(float)module.get_method("single_step_log")({context_in, thisword}).toDouble();
  float ans;
  if (word != oos_) {
    //ans = outputs[0].scalar&lt;float&gt;()();
    ans = log_prob;
  } else {
    if (unk_costs_.size() == 0) {
      //ans = outputs[0].scalar&lt;float&gt;()() - log(num_total_words - num_rnn_words);
      ans = log_prob - log(num_total_words - num_rnn_words);
    } else {
      //ans = outputs[0].scalar&lt;float&gt;()() + unk_costs_[fst_word];
      ans = log_prob + unk_costs_[fst_word];
    }
  }

  return ans;
}

// Done *****
const torch::Tensor&amp; KaldiPyRnnlmWrapper::GetInitialContext() const {
  return initial_context_;
}

const torch::Tensor&amp; KaldiPyRnnlmWrapper::GetInitialCell() const {
  return initial_cell_;
}

int KaldiPyRnnlmWrapper::FstLabelToRnnLabel(int i) const {
  KALDI_ASSERT(i &gt;= 0 &amp;&amp; i &lt; fst_label_to_rnn_label_.size());
  return fst_label_to_rnn_label_[i];
}


// Done *****
PyRnnlmDeterministicFst::PyRnnlmDeterministicFst(int32 max_ngram_order,
                                             KaldiPyRnnlmWrapper *rnnlm) {
  KALDI_ASSERT(rnnlm != NULL);
  max_ngram_order_ = max_ngram_order;
  rnnlm_ = rnnlm;

  std::vector&lt;Label&gt; bos;
  const torch::Tensor&amp; initial_context = rnnlm_-&gt;GetInitialContext();
  const torch::Tensor&amp; initial_cell = rnnlm_-&gt;GetInitialCell();

  state_to_wseq_.push_back(bos);
  state_to_context_.push_back(new torch::Tensor(initial_context));
  state_to_cell_.push_back(new torch::Tensor(initial_cell));
  wseq_to_state_[bos] = 0;
  start_state_ = 0;
}

// Done *****
PyRnnlmDeterministicFst::~PyRnnlmDeterministicFst() {
  for (int i = 0; i &lt; state_to_context_.size(); i++) {
    delete state_to_context_[i];
  }
  for (int i = 0; i &lt; state_to_cell_.size(); i++) {
    delete state_to_cell_[i];
  }
}

// Done *****
void PyRnnlmDeterministicFst::Clear() {
  // similar to the destructor but we retain the 0-th entries in each map
  // which corresponds to the &lt;bos&gt; state
  for (int i = 1; i &lt; state_to_context_.size(); i++) {
    delete state_to_context_[i];
  }
  for (int i = 1; i &lt; state_to_cell_.size(); i++) {
    delete state_to_cell_[i];
  }

  state_to_context_.resize(1);
  state_to_cell_.resize(1);
  state_to_wseq_.resize(1);
  wseq_to_state_.clear();
  wseq_to_state_[state_to_wseq_[0]] = 0;
}

// Done *****
fst::StdArc::Weight PyRnnlmDeterministicFst::Final(StateId s) {
  // At this point, we should have created the state.
  KALDI_ASSERT(static_cast&lt;size_t&gt;(s) &lt; state_to_wseq_.size());

  std::vector&lt;Label&gt; wseq = state_to_wseq_[s];
  BaseFloat logprob = rnnlm_-&gt;GetLogProb(rnnlm_-&gt;GetEos(),
                         -1,  // only need type; this param will not be used
                         *state_to_context_[s],
                         *state_to_cell_[s], NULL, NULL);
  return Weight(-logprob);
}

// Done *****
bool PyRnnlmDeterministicFst::GetArc(StateId s, Label ilabel,
                                     fst::StdArc *oarc) {
  KALDI_ASSERT(static_cast&lt;size_t&gt;(s) &lt; state_to_wseq_.size());

  std::vector&lt;Label&gt; wseq = state_to_wseq_[s];
  torch::Tensor *new_context = new torch::Tensor();
  torch::Tensor *new_cell = new torch::Tensor();

  // look-up the rnn label from the FST label
  int32 rnn_word = rnnlm_-&gt;FstLabelToRnnLabel(ilabel);
  BaseFloat logprob = rnnlm_-&gt;GetLogProb(rnn_word,
                                         ilabel,
                                         *state_to_context_[s],
                                         *state_to_cell_[s],
                                         new_context,
                                         new_cell);

  wseq.push_back(rnn_word);
  if (max_ngram_order_ &gt; 0) {
    while (wseq.size() &gt;= max_ngram_order_) {
      // History state has at most &lt;max_ngram_order_&gt; - 1 words in the state.
      wseq.erase(wseq.begin(), wseq.begin() + 1);
    }
  }

  std::pair&lt;const std::vector&lt;Label&gt;, StateId&gt; wseq_state_pair(
      wseq, static_cast&lt;Label&gt;(state_to_wseq_.size()));

  // Attemps to insert the current &lt;lseq_state_pair&gt;. If the pair already exists
  // then it returns false.
  typedef MapType::iterator IterType;
  std::pair&lt;IterType, bool&gt; result = wseq_to_state_.insert(wseq_state_pair);

  // If the pair was just inserted, then also add it to &lt;state_to_wseq_&gt; and
  // &lt;state_to_context_&gt;.
  if (result.second == true) {
    state_to_wseq_.push_back(wseq);
    state_to_context_.push_back(new_context);
    state_to_cell_.push_back(new_cell);
  } else {
    delete new_context;
    delete new_cell;
  }

  // Creates the arc.
  oarc-&gt;ilabel = ilabel;
  oarc-&gt;olabel = ilabel;
  oarc-&gt;nextstate = result.first-&gt;second;
  oarc-&gt;weight = Weight(-logprob);

  return true;
}

}  // namespace kaldi

</code></pre>

glaringlee: <p><a class="mention" href="/u/krrishabh">@krrishabh</a><br>
Can you post this in ‘jit’ category?</p>
<p>And, check what pytorch/libtorch version is used to generate your .pt file and what libtorch version you were used to load it.  This might be a version mismatch as well.</p>

krrishabh: <p>I have change the tag from C++ to jit<br>
libtorch is used to load the model<br>
libtorch Build version : 1.6.0.dev20200501+cu101</p>
<p>Model is saved in Pytorch version : 1.3.1</p>
<p>Moreover, I have made a small project where it is working fine… but in my Kaldi Project the model is not able to get load. (both have same Configuration)</p>

glaringlee: <p><a class="mention" href="/u/krrishabh">@krrishabh</a><br>
I think version mismatch might be the root cause.<br>
If you saved your .pt with 1.3.1, please use the same version of libtorch to load it.<br>
We have a similar issue here,<br>
‘<a href="https://github.com/pytorch/pytorch/issues/39623" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/39623</a>’<br>
please read the comments see how to download libtorch 1.3.1, or if possible, upgrade your pytorch to 1.6dev, or I believe 1.5 is fine.</p>

krrishabh: <p>I have upgrade the Pytorch version to 1.5 but I am getting the same error. Moreover, It was not a problem of version because when I am building a small project it is working correctly.</p>

m4saka: <p><a class="mention" href="/u/krrishabh">@krrishabh</a><br>
Since the error message says “error loading the model”, it is likely the program failed to load the model file in <code>torch::jit::load(path)</code>.</p>
<p>Please make sure <code>data/pytorch/rnnlm/newmodel2.pt</code> exists and it is accessible from the binary.<br>
If you compile the program into <code>build</code> directory, the path might be <code>../data/pytorch/rnnlm/newmodel2.pt</code>.</p>
<p>And the message “<code>_ivalue_ INTERNAL ASSERT FAILED at ...</code>” is shown when an instance of <code>torch::jit::script::Module</code> is not initialized (like the situation that the exception from <code>torch::jit::load()</code> is caught and you reference the uninitialized module instance after that).</p>

ven: <p><a class="mention" href="/u/m4saka">@m4saka</a><br>
Did you figure out this issue? I’ve been suffering the same. torch::jit::load brings the model without a problem as an individual project, but when it is combined with another project, it shows:</p>
<p>error loading the model<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  <em>ivalue</em> INTERNAL ASSERT FAILED at “…/torch/csrc/jit/api/object.cpp”:19, please report a bug to PyTorch.<br>
Exception raised from _ivalue at …/torch/csrc/jit/api/object.cpp:19 (most recent call first):</p>

Martin_JH: <p>I am facing the same problem. In individual consol project it runs without any problem. But with combined project it can’t load the same model.</p>

glaand: <p>Did you find the solution to your problem? i am stuck as wel…</p>