Where does `torch._C` come from?

acgtyrant: <p>I am read the code of batch normlization, and I find <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L454" rel="nofollow noopener">this line</a>:</p>
<p><code>f = torch._C._functions.BatchNorm(running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled)</code></p>
<p>But I do not find any library called the <code>_C</code>. I do not know where does <code>torch._C.functions.BatchNorm</code> come from.</p>

fmassa: <p>Here is the C source for PyTorch <a href="https://github.com/pytorch/pytorch/tree/master/torch/csrc">https://github.com/pytorch/pytorch/tree/master/torch/csrc</a><br>
And the external libraries that perform the math computation can be found in <a href="https://github.com/pytorch/pytorch/tree/master/torch/lib">https://github.com/pytorch/pytorch/tree/master/torch/lib</a></p>

fmassa: <p>For completeness, the <code>_C</code> comes from <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/Module.cpp#L732-L742">here</a></p>

lmf: <p>hi, fmassa. I want to find the defination of torch._C.CudaFloatTensorBase class.But i didn’t find it in the torch/csrc/Module.cpp.Do you know where is it? Thanks.</p>

smth: <p><a class="mention" href="/u/lmf">@lmf</a> we do these things via C macro expansions:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/pytorch/pytorch/search?utf8=%E2%9C%93&amp;q=TensorBase&amp;type=" target="_blank">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/pytorch/pytorch/search?utf8=%E2%9C%93&amp;q=TensorBase&amp;type=" target="_blank">pytorch/pytorch</a></h3>

<p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


dl4daniel: <p>Thanks for the two links, I tried a lot but I still could not find exactly where <code>torch._C._functions.ConvNd</code> come from, could you give me an exact link? Thanks a lot!</p>

albanD: <p>Hi</p>
<p>The module <code>torch._C._functions</code> is created <a href="https://github.com/pytorch/pytorch/blob/ded3a3b317fbc8b165235c98f238c193e1482ad5/torch/csrc/autograd/functions/init.cpp#L239">here</a>.<br>
The <code>ConvNd</code> class is added on <a href="https://github.com/pytorch/pytorch/blob/ded3a3b317fbc8b165235c98f238c193e1482ad5/torch/csrc/autograd/functions/init.cpp#L248">this</a> line.</p>

dl4daniel: <p>Thanks <a class="mention" href="/u/alband">@albanD</a><br>
One more thing, and very important, is how do I understand this line of code <code>addClass&lt;ConvForward, ConvCtor&gt;(module, ConvClass, "ConvNd", conv_forward_properties);</code> , because I plan to learn C and understand the logic of each function in Torch Neural Network library.</p>
<p>I have to admit at this moment I have not learnt C language, so could you just give me a basic idea of what does this line of code do?</p>
<p>Thanks a lot!</p>

smth: <p><a class="mention" href="/u/dl4daniel">@dl4daniel</a> this is C++. Without knowing C or C++, it’s not easy to give you a basic idea of what’s going on.</p>

dl4daniel: <p>Thanks for your reply! Could you give me a basic picture while assuming I am a C or C++ beginner, is it feasible? thanks</p>

dl4daniel: <p><a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/apaszke">@apaszke</a><br>
Is it possible to debug from pytorch code into torch._C code?</p>
<p>I can use pdbpp to debug pytorch code and check all variables values and it is very convenient for learning. for example, when I want to see what is going on inside ‘self.conv1(x)’, I can step into</p>
<pre><code>27  -&gt;         x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) 
</code></pre>
<p>and eventually, I am taken to the following code, which is the edge between pytorch python and torch._C. I want to be able to continue to debug and checkout variable values inside torch._C code such as <code>ConvNd</code> below. Is it possible? if so, how could I do it? Thanks a lot</p>
<pre><code>50         f = ConvNd(_pair(stride), _pair(padding), _pair(dilation), False,
51                    _pair(0), groups, torch.backends.cudnn.benchmark, torch.backends
52         return f(input, weight, bias)</code></pre>

albanD: <p>Hi,<br>
Yes you can do it but since you are going into cpp land, you will need a cpp debugger.<br>
If you have a cpp ide you may have a debugger in it.<br>
To check stack traces and basic values without any graphical tool you can use gdb, but this is significantly less user friendly.</p>
<p>If you are not used to cpp debugger, it may be simpler to just add prints into the function and recompile pytorch with your prints. You can find <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#c-development-tips">here</a> some cpp dev tips for pytorch.</p>

dl4daniel: <p><a class="mention" href="/u/alband">@albanD</a> thanks a lot!<br>
You are very right about gdb, I spend several hours to install gdb and tried to get it work for running python file, but still<a href="https://stackoverflow.com/questions/46463521/gdb-cant-be-properly-installed-in-mac-sierra" rel="noopener nofollow ugc"> failed to get it work properly</a></p>
<p>I prefer a simple solution to my problem above: how to check or debug pytorch code all the way from python code to C code or C++ code. If I understand you correctly, it seems the simple solution for me is to pick a C/C++ ide with a debuger. I have been using atom for a while. Is it possible to make atom a proper C/C++ ide? if so, can I use it to check or debug pytorch code from python to C and C++ in one go within atom?</p>
<p>right now, it seems I can compile and run C/C++ in atom OK<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/17dd230c571a6189539944737a740f178dac12f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/17dd230c571a6189539944737a740f178dac12f8" title="atom__c_compileOK"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/17dd230c571a6189539944737a740f178dac12f8_2_658x500.png" alt="atom__c_compileOK" data-base62-sha1="3p6LsqdCkg6GJdlR8S1w9DCvKLm" width="658" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/17dd230c571a6189539944737a740f178dac12f8_2_658x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/17dd230c571a6189539944737a740f178dac12f8_2_987x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/17dd230c571a6189539944737a740f178dac12f8_2_1316x1000.png 2x" data-dominant-color="282C33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">atom__c_compileOK</span><span class="informations">1662×1262 197 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, I can’t just compile and debug my pytorch code as if it is a C/C++ file.  Also, although it says “compile and debug”, I don’t see any features like gdb or pdb in atom. What should I do from here? Thanks a lot!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/190cfcda444ba969d7ea7326d3c2a4b77d34f074.png" data-download-href="https://discuss.pytorch.org/uploads/default/190cfcda444ba969d7ea7326d3c2a4b77d34f074" title="atom__c_compileBad"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/190cfcda444ba969d7ea7326d3c2a4b77d34f074_2_690x428.png" alt="atom__c_compileBad" data-base62-sha1="3zBLiaFp1D79bwdz9rUPgIziYAI" width="690" height="428" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/190cfcda444ba969d7ea7326d3c2a4b77d34f074_2_690x428.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/190cfcda444ba969d7ea7326d3c2a4b77d34f074_2_1035x642.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/190cfcda444ba969d7ea7326d3c2a4b77d34f074_2_1380x856.png 2x" data-dominant-color="473D42"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">atom__c_compileBad</span><span class="informations">2026×1258 454 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>I have never used Atom, so I can’t really help you here unfortunately. You may be able to find more information on google though <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

dl4daniel: <p>Thanks for your patience! one last question, put atom aside, for a c/c++ ide with debuger, we can use C/C++ debugger to debug a pytorch file like “neural_network_tutoral.py”, for both python code and C code like <code>ConvNd</code>  under the hood, right?</p>

albanD: <p>I am not aware of any IDE that would be able to do that. I think you will have to have two different debuggers.</p>

dl4daniel: <p>Thanks, so two debugers (one for python, one for C/C++) for the same pytorch tutorlal in python. This is interesting.</p>

fmassa: <p>The same thing happens in numpy if you want to debug C code there.</p>

dl4daniel: <p>Thanks, could you give me a simple demo on how to debug C code in a numpy code example?</p>

fmassa: <p>Well, you usually want to debug in C when you have segmentation faults, as you don’t have any information that was returned from C.</p>
<p>If you want to debug from python, you can use <a href="https://docs.python.org/2/library/pdb.html"><code>pdb</code></a>.</p>
<p><strong>Note:</strong> in most cases you don’t need to go into gdb to debug a python program, because the libraries were designed to give meaningful error messages and not segfault. In the numpy case that I will show below, I will use an unsafe numpy function to obtain the segfault, but that should not happen with normal numpy code. The same applies for pytorch, so if you observe segmentation faults, please let us know <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>If you want to go into debugging C, you can use gdb for example. Say you have a python script <code>example.py</code> that gives a problem. Here I’ll post an example in numpy, called <code>example.py</code>:</p>
<pre><code class="lang-python">import numpy as np
from numpy.lib.stride_tricks import as_strided

a = np.array([1, 2, 3])
b = as_strided(a, shape=(2,), strides=(20,))
# accessing invalid memory, will segfault
b[1] = 1
</code></pre>
<p>If you try running it in python, it will give segmentation fault. Try running <code>python example.py</code>.</p>
<p>To run it on gdb, you can do something like</p>
<pre><code class="lang-auto">gdb python
</code></pre>
<p>and then, once it enters gdb, you do</p>
<pre><code class="lang-auto">run example.py
</code></pre>
<p>This will run the program and, when it crashes, it will give you the possibility to inspect where it crashed by running <code>bt</code> (from backtrace). You can find more information on <a href="https://www.gnu.org/software/gdb/">gdb options online</a>.</p>