Free(): invalid next size (normal) with ported cpp code

username: Ahmed_Abdelaziz
cooked: <p>I ported a CPP code into my PyTorch code. Every time I start training I randomly see this error at random batches during training. I couldn’t debug it and I don’t know what can cause the problem.</p>
<pre><code class="lang-auto">free(): invalid next size (normal)
</code></pre>
<p>I use the below script to compile and port the cpp extension</p>
<pre><code class="lang-auto">from setuptools import setup, Extension
from torch.utils import cpp_extension
import os

setup(name='span_prune_cpp',
      ext_modules=[cpp_extension.CppExtension('span_prune_cpp', ['span_prune.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
</code></pre>
<p>Cpp code if it helps in anyway</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

#include &lt;cstdio&gt;
#include &lt;time.h&gt;
#include &lt;stdlib.h&gt;

#include &lt;iostream&gt;
#include &lt;pybind11/pybind11.h&gt;

using namespace std;


torch::Tensor extract_spans(
    torch::Tensor span_scores,
    torch::Tensor candidate_starts,
    torch::Tensor candidate_ends,
    torch::Tensor num_output_spans,
    int max_sentence_length,
    bool _sort_spans,
    bool _suppress_crossing
) {

   time_t givemetime = time(NULL);

    int num_sentences = span_scores.size(0);
    int num_input_spans = span_scores.size(1);
    int max_num_output_spans = 0;


    for (int i = 0; i &lt; num_sentences; i++) {

      if (num_output_spans[i].item&lt;int64_t&gt;() &gt; max_num_output_spans) {
        max_num_output_spans = num_output_spans[i].item&lt;int64_t&gt;();
      }
    }

    std::vector&lt;std::vector&lt;int&gt;&gt; sorted_input_span_indices(num_sentences,
                                                            std::vector&lt;int&gt;(num_input_spans));

    torch::Tensor output_span_indices = torch::ones({num_sentences, max_num_output_spans});

    for (int i = 0; i &lt; num_sentences; i++) {
      std::iota(sorted_input_span_indices[i].begin(), sorted_input_span_indices[i].end(), 0);
      std::sort(sorted_input_span_indices[i].begin(), sorted_input_span_indices[i].end(),
                [&amp;span_scores, &amp;i](int j1, int j2) {
                 if (j1 &gt;= span_scores.size(1) || j1 &lt; 0 || j2 &gt;= span_scores.size(1) || j2 &lt; 0) {
                    return false;
                 }

                  return span_scores[i][j2].item&lt;int64_t&gt;() &lt; span_scores[i][j1].item&lt;int64_t&gt;();
                });
    }


    for (int l = 0; l &lt; num_sentences; l++) {

      std::vector&lt;int&gt; top_span_indices;
      std::unordered_map&lt;int, int&gt; end_to_earliest_start;
      std::unordered_map&lt;int, int&gt; start_to_latest_end;
      int current_span_index = 0, num_selected_spans = 0;

      while (num_selected_spans &lt; num_output_spans[l].item&lt;int64_t&gt;() &amp;&amp; current_span_index &lt; num_input_spans) {
        int i = sorted_input_span_indices[l][current_span_index];
        bool any_crossing = false;
        if (_suppress_crossing) {
          const int&amp; start = candidate_starts[l][i].item&lt;int64_t&gt;();
          const int&amp; end = candidate_ends[l][i].item&lt;int64_t&gt;();

          for (int j = start; j &lt;= end; ++j) {
            if (j &gt; start) {
              auto latest_end_iter = start_to_latest_end.find(j);
              if (latest_end_iter != start_to_latest_end.end() &amp;&amp; latest_end_iter-&gt;second &gt; end) {
                // Given (), exists [], such that ( [ ) ]
                any_crossing = true;
                break;
              }
            }
            if (j &lt; end) {
              auto earliest_start_iter = end_to_earliest_start.find(j);
              if (earliest_start_iter != end_to_earliest_start.end() &amp;&amp; earliest_start_iter-&gt;second &lt; start) {
                // Given (), exists [], such that [ ( ] )
                any_crossing = true;
                break;
              }
            }
          }
        }
        if (!any_crossing) {
          if (_sort_spans) {
            top_span_indices.push_back(i);
          } else {
            output_span_indices[l][num_selected_spans] = i;
          }
          ++num_selected_spans;
          if (_suppress_crossing) {
            // Update data struct.
            const int&amp; start = candidate_starts[l][i].item&lt;int64_t&gt;();
            const int&amp; end = candidate_ends[l][i].item&lt;int64_t&gt;();
            auto latest_end_iter = start_to_latest_end.find(start);
            if (latest_end_iter == start_to_latest_end.end() || end &gt; latest_end_iter-&gt;second) {
              start_to_latest_end[start] = end;
            }
            auto earliest_start_iter = end_to_earliest_start.find(end);
            if (earliest_start_iter == end_to_earliest_start.end() || start &lt; earliest_start_iter-&gt;second) {
              end_to_earliest_start[end] = start;
            }
          }
        }
        ++current_span_index;
      }


      // Sort and produce span indices.
      if (_sort_spans) {

        std::sort(top_span_indices.begin(), top_span_indices.end(),
                [&amp;candidate_starts, &amp;candidate_ends, &amp;l] (int i1, int i2) {
                 if (i1 &gt;= candidate_starts.size(1) || i1 &lt; 0 || i2 &gt;= candidate_starts.size(1) || i2 &lt; 0) {
                    return false;
                 }
                  if (candidate_starts[l][i1].item&lt;int64_t&gt;() &lt; candidate_starts[l][i2].item&lt;int64_t&gt;()) {
                    return true;
                  } else if (candidate_starts[l][i1].item&lt;int64_t&gt;() &gt; candidate_starts[l][i1].item&lt;int64_t&gt;()) {
                    return false;
                  } else if (candidate_ends[l][i1].item&lt;int64_t&gt;() &lt; candidate_ends[l][i1].item&lt;int64_t&gt;()) {
                    return true;
                  } else if (candidate_ends[l][i1].item&lt;int64_t&gt;() &gt; candidate_ends[l][i1].item&lt;int64_t&gt;()) {
                    return false;
                  } else {
                    return i1 &lt; i2;
                  }
                });



        for (int i = 0; i &lt; num_output_spans[l].item&lt;int64_t&gt;(); ++i) {
          output_span_indices[l][i] = top_span_indices[i];
        }
      }

      // Pad with the last selected span index to ensure monotonicity.
      int last_selected = num_selected_spans - 1;
      if (last_selected &gt;= 0) {
        for (int i = num_selected_spans; i &lt; max_num_output_spans; ++i) {
          output_span_indices[l][i]= output_span_indices[l][last_selected].item&lt;int64_t&gt;();
        }
      }


    }


    return output_span_indices;
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("extract_spans", &amp;extract_spans, "extract_spans");
}

</code></pre>
<p>Please let me know if I can share any additional information. I debugged all tensors fed to the <code>extract_spans</code> function and it holds valid numerical values.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Did you run your program in gdb to know where the error comes from by any chance?</p>

username: Ahmed_Abdelaziz
cooked: <p>I didn’t run it in gdb because it sounds complicated, I think I will have to compile pytorch again, is that right? Plus this is totally random and it occurs after many batches (which takes hours) so I am not quite sure how to approach this. I tried logging progress to a file but still cannot infer the root cause of the issue</p>

username: albanD
cooked: <p>Hi,</p>
<p>No it is quite simple.<br>
If you run your script with <code>python your_script.py --args foo</code> usually. You can do:</p>
<pre><code class="lang-auto">gdb python
# Wait to be in gdb cli
r your_script.py --args foo
# Wait for your script to crash
# Then you can use `bt` to get a stack trace
bt
</code></pre>
<p>This will give us quite a bit of information even without a debug build <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Ahmed_Abdelaziz
cooked: <p>Cool I don’t have gdb installed on my server but will reach out to the server admin. Thanks a lot!</p>

username: Ahmed_Abdelaziz
cooked: <p>I got this stack trace today.</p>
<pre><code class="lang-auto">#0  0x00007ffff7803f47 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff78058b1 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff784e907 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff785597a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ffff785cef4 in free () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007fff5506a425 in __gnu_cxx::new_allocator&lt;int&gt;::deallocate (this=0x7fffffffcfd0, __p=&lt;optimized out&gt;)
    at /usr/include/c++/7/ext/new_allocator.h:125
#6  std::allocator_traits&lt;std::allocator&lt;int&gt; &gt;::deallocate (__a=..., __n=&lt;optimized out&gt;, __p=&lt;optimized out&gt;)
    at /usr/include/c++/7/bits/alloc_traits.h:462
#7  std::_Vector_base&lt;int, std::allocator&lt;int&gt; &gt;::_M_deallocate (this=0x7fffffffcfd0, __n=&lt;optimized out&gt;, __p=&lt;optimized out&gt;)
    at /usr/include/c++/7/bits/stl_vector.h:180
#8  std::_Vector_base&lt;int, std::allocator&lt;int&gt; &gt;::~_Vector_base (this=0x7fffffffcfd0, __in_chrg=&lt;optimized out&gt;)
    at /usr/include/c++/7/bits/stl_vector.h:162
#9  std::vector&lt;int, std::allocator&lt;int&gt; &gt;::~vector (this=0x7fffffffcfd0, __in_chrg=&lt;optimized out&gt;) at /usr/include/c++/7/bits/stl_vector.h:435
#10 extract_spans (span_scores=..., candidate_starts=..., candidate_ends=..., num_output_spans=..., max_sentence_length=&lt;optimized out&gt;,
    _sort_spans=&lt;optimized out&gt;, _suppress_crossing=&lt;optimized out&gt;) at span_prune.cpp:75
#11 0x00007fff5507c0c6 in pybind11::detail::argument_loader&lt;at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool&gt;::call_impl&lt;at::Tensor, at::Tensor (*&amp;)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool), 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, pybind11::detail::void_type&gt; (
    f=&lt;optimized out&gt;, this=0x7fffffffd460)
    at /mnt/data2/hussein/miniconda3/envs/torch-sci/lib/python3.7/site-packages/torch/include/pybind11/cast.h:1931
#12 pybind11::detail::argument_loader&lt;at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool&gt;::call&lt;at::Tensor, pybind11::detail::void_type, at::Tensor (*&amp;)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool)&gt;(at::Tensor (*&amp;)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool)) &amp;&amp; (f=&lt;optimized out&gt;, this=&lt;optimized out&gt;)
    at /mnt/data2/hussein/miniconda3/envs/torch-sci/lib/python3.7/site-packages/torch/include/pybind11/cast.h:1908
#13 pybind11::cpp_function::initialize&lt;at::Tensor (*&amp;)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool), at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool, pybind11::name, pybind11::scope, pybind11::sibling, char [14]&gt;(at::Tensor (*&amp;)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool), at::Tensor (*)(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, bool, bool), pybind11::name const&amp;, pybind11::scope const&amp;, pybind11::sibling const&amp;, char const (&amp;) [14])::{lambda(pybind11::detail::function_call&amp;)#3}::operator()(pybind11::detail::function_call&amp;) const (__closure=&lt;optimized out&gt;, call=...)
    at /mnt/data2/hussein/miniconda3/envs/torch-sci/lib/python3.7/site-packages/torch/include/pybind11/pybind11.h:155
#14 0x00007fff55078690 in pybind11::cpp_function::dispatcher (self=&lt;optimized out&gt;, args_in=0x7ffec673e4b0, kwargs_in=0x0)
    at /mnt/data2/hussein/miniconda3/envs/torch-sci/lib/python3.7/site-packages/torch/include/pybind11/pybind11.h:620
#15 0x00005555556b9c94 in _PyMethodDef_RawFastCallKeywords (method=0x5555574dfb40, self=0x7ffff5f24750, args=0x555625428c58,
    nargs=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;) at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:694
#16 0x00005555556b9db1 in _PyCFunction_FastCallKeywords (func=0x7fff5511ebe0, args=&lt;optimized out&gt;, nargs=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:734
#17 0x00005555557255be in call_function (kwnames=0x0, oparg=7, pp_stack=&lt;synthetic pointer&gt;)
    at /tmp/build/80754af9/python_1588882889832/work/Python/ceval.c:4568
#18 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at /tmp/build/80754af9/python_1588882889832/work/Python/ceval.c:3093
#19 0x000055555566a31b in function_code_fastcall (globals=&lt;optimized out&gt;, nargs=2, args=&lt;optimized out&gt;, co=0x7ffef5921a50)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:283
#20 _PyFunction_FastCallDict (func=&lt;optimized out&gt;, args=0x7fffffffd960, nargs=2, kwargs=&lt;optimized out&gt;)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:322
#21 0x0000555555688b93 in _PyObject_Call_Prepend (callable=0x7ffef5928320, obj=&lt;optimized out&gt;, args=0x7ffec68de050, kwargs=0x7ffec6a20410)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:908
#22 0x000055555567b95e in PyObject_Call (callable=0x7ffec6abbaa0, args=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:245
#23 0x000055555572251a in do_call_core (kwdict=0x7ffec6a20410, callargs=0x7ffec68de050, func=0x7ffec6abbaa0)
    at /tmp/build/80754af9/python_1588882889832/work/Python/ceval.c:4645
#24 _PyEval_EvalFrameDefault (f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at /tmp/build/80754af9/python_1588882889832/work/Python/ceval.c:3191
#25 0x00005555556692b9 in _PyEval_EvalCodeWithName (_co=0x7fff556e6030, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;,
    argcount=&lt;optimized out&gt;, kwnames=0x0, kwargs=0x0, kwcount=&lt;optimized out&gt;, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0,
    name=0x7ffff6c651b0, qualname=0x7fff556e5230) at /tmp/build/80754af9/python_1588882889832/work/Python/ceval.c:3930
#26 0x000055555566a3e5 in _PyFunction_FastCallDict (func=&lt;optimized out&gt;, args=0x7fffffffdcc0, nargs=2, kwargs=&lt;optimized out&gt;)
    at /tmp/build/80754af9/python_1588882889832/work/Objects/call.c:376
</code></pre>
<p>Any idea what is going on here?</p>

username: albanD
cooked: <p>The errors points at the destructor of a <code>std::vector&lt;int&gt;</code>.<br>
Are you doing anything funky with one of these? Like stealing it’s data? Or building it with stolen data? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Ahmed_Abdelaziz
cooked: <p>I don’t think I am doing any illegal operation to it.</p>
<p>I first initialised it (Any btw this is the only <code>std::vector&lt;int&gt;</code> in my code)</p>
<pre><code class="lang-auto">std::vector&lt;int&gt; top_span_indices;
</code></pre>
<p>Push some data into it</p>
<pre><code class="lang-auto">top_span_indices.push_back(i);
</code></pre>
<p>Sort all elements with a custom comparator</p>
<pre><code class="lang-auto">std::sort(top_span_indices.begin(), top_span_indices.end(),
                [&amp;candidate_starts, &amp;candidate_ends, &amp;l] (int i1, int i2) {
                 if (i1 &gt;= candidate_starts.size(1) || i1 &lt; 0 || i2 &gt;= candidate_starts.size(1) || i2 &lt; 0) {
                    return false;
                 }
                  if (candidate_starts[l][i1].item&lt;int64_t&gt;() &lt; candidate_starts[l][i2].item&lt;int64_t&gt;()) {
                    return true;
                  } else if (candidate_starts[l][i1].item&lt;int64_t&gt;() &gt; candidate_starts[l][i2].item&lt;int64_t&gt;()) {
                    return false;
                  } else if (candidate_ends[l][i1].item&lt;int64_t&gt;() &lt; candidate_ends[l][i2].item&lt;int64_t&gt;()) {
                    return true;
                  } else if (candidate_ends[l][i1].item&lt;int64_t&gt;() &gt; candidate_ends[l][i2].item&lt;int64_t&gt;()) {
                    return false;
                  } else {
                    return i1 &lt; i2;
                  }
                });
</code></pre>
<p>And finally copied its data (which I believe is copying by value here)</p>
<pre><code class="lang-auto">for (int i = 0; i &lt; num_output_spans[l].item&lt;int64_t&gt;(); ++i) {
          output_span_indices[l][i] = top_span_indices[i];
        }
</code></pre>
<p>Do you think am doing anything wrong here?</p>

username: Ahmed_Abdelaziz
cooked: <p>Could it be that vector too?<br>
<code>std::vector&lt;std::vector&lt;int&gt;&gt; sorted_input_span_indices(num_sentences, std::vector&lt;int&gt;(num_input_spans));</code></p>

username: albanD
cooked: <p>It’s hard to say.<br>
The way I would debug this is remove pieces of the code until it works fine. And then add them back one by one. Until you find the one thing that makes it fail <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Ahmed_Abdelaziz
cooked: <p>Ok wil give it a try. Thanks!</p>