C++ and pytorch inference discrepancy

jonrbates: <p>I’m comparing inference in a traced model using (1) python and (2) c++.  The outputs are different.  Has anyone successfully run GPT-2 inference with c++?</p>
<p>xposted here: <a href="https://github.com/pytorch/pytorch/issues/36891" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/36891</a></p>
<p>python version: <em>3.7.4</em><br>
torch version: <em>1.4.0</em><br>
transformers version: <em>2.6.0</em></p>
<p><strong>Creating the model in python</strong></p>
<pre><code class="lang-auto">from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained("distilgpt2", 
            pad_token_id=tokenizer.eos_token_id,
            torchscript=True)
test_text = 'The bear went over the mountain to see'
tokens_tensor = torch.tensor([tokenizer.encode(test_text)])
traced_model = torch.jit.trace(model, [tokens_tensor])
torch.jit.save(traced_model,"traced_distilgpt2.pt")
</code></pre>
<p><strong>Inference in pytorch</strong></p>
<pre><code class="lang-auto">loaded_model = torch.jit.load("traced_distilgpt2.pt")
outputs = loaded_model(torch.tensor([[464, 6842, 1816,  625,  262, 8598,  284,  766]]))

torch.max(outputs[0],2)
&gt;&gt; torch.return_types.max(values=tensor([[-26.7050, -53.3268, -66.3666, -50.1084, -54.8050, -72.4056, -55.2586,-63.5215]], grad_fn=&lt;MaxBackward0&gt;),
indices=tensor([[ 383,  373,  319,  262, 1353,  290,  262,  262]]))

torch.min(outputs[0][:, :, :],2)
&gt;&gt;torch.return_types.min(
values=tensor([[ -47.3794,  -77.6341,  -95.8365,  -75.3026,  -81.6899, -103.6701, -83.0335,  -93.0259]], grad_fn=&lt;MinBackward0&gt;),
indices=tensor([[  154,  7134, 31204, 22997, 10298, 31204, 22997, 31573]]))
</code></pre>
<p><strong>Inference in c</strong></p>
<pre><code class="lang-auto">int main(int argc, const char* argv[]) {

  torch::jit::script::Module module;
  module = torch::jit::load(argv[1]);
  std::cout &lt;&lt; "Model loaded.\n";
  module.eval(); // just in case?

  torch::Tensor x = torch::tensor({464, 6842, 1816,  625,  262, 8598,  284,  766},
    torch::dtype(torch::kInt64)).reshape({8, 1});
  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back(x);

  // Execute the model and turn its output into a tensor (all_encoder_layers).
  torch::Tensor out = module.forward(inputs).toTuple()-&gt;elements()[0].toTensor();

  auto z = torch::max(out, /*dim=*/2);
  std::cout &lt;&lt; std::get&lt;0&gt;(z) &lt;&lt; '\n';
  std::cout &lt;&lt; std::get&lt;1&gt;(z) &lt;&lt; '\n';

  z = torch::min(out, /*dim=*/2);
  std::cout &lt;&lt; std::get&lt;0&gt;(z) &lt;&lt; '\n';
  std::cout &lt;&lt; std::get&lt;1&gt;(z) &lt;&lt; '\n';

  return 0;
}
</code></pre>
<p>Output from running c exe (with argv[1] = “traced_distilgpt2.pt”) :</p>
<pre><code class="lang-auto">[ Variable[CPULongType]{8,1} ]
-26.7050
-28.0251
-29.0539
-28.2608
-26.3226
-28.0652
-26.8328
-28.5087
[ Variable[CPUFloatType]{8,1} ]
 383
 383
 383
 383
 383
 383
 383
 383
[ Variable[CPULongType]{8,1} ]
-47.3794
-48.2470
-49.5565
-49.0826
-46.3104
-48.5934
-47.3013
-49.1308
[ Variable[CPUFloatType]{8,1} ]
 154
 154
 154
 154
 154
 154
 154
 154
[ Variable[CPULongType]{8,1} ]
</code></pre>
<p><strong>Observation</strong>:<br>
The models agree for the first token (position=0) but for positions 1-7 the encoder outputs disagree.</p>

alanzhai219: <p>what is tokenizer? Could you give a reproduce code? I can’t generate such .pt for debug.</p>

jonrbates: <p>Thanks for the quick reply!</p>
<p>I’m using</p>
<pre><code class="lang-auto">from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2", 
                                          cache_dir=cache_dir, 
                                          torchscript=True)
</code></pre>
<p><strong>Note</strong> I’m using the same tensor to trace the model as for inference</p>
<aside class="quote no-group" data-username="jonrbates" data-post="1" data-topic="77388">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ac8455/48.png" class="avatar"> jonrbates:</div>
<blockquote>
<p>torch.tensor([[464, 6842, 1816, 625, 262, 8598, 284, 766]])</p>
</blockquote>
</aside>

jonrbates: <p>Ok, here is code to reproduce .pt</p>
<pre><code class="lang-auto">import torch
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained("distilgpt2", 
           pad_token_id=50256,
           torchscript=True)
traced_model = torch.jit.trace(model, [torch.tensor([[464, 6842, 1816,  625,  262, 8598,  284,  766]])])
torch.jit.save(traced_model, "traced_distilgpt2.pt")
</code></pre>