The pytorch blog "A Tour of PyTorch Internals" is out-of-date. How to know more about the pytorch internal

username: Ke_Bai
cooked: <p>I am start to read the source code of pytorch recently. But I am totally lost. In the tutorial, Part one introduces the “Tensor type”, but “csrc/generic/Tensor.cpp” disappeared. I only found “Tensor.h” in Aten library and in pytorch/torch/lib/THD/master_worker/worker/dispatch/ (But the later folder is claimed “dead” now)</p>
<p>I have read short introductions to the internal of pytorch in the document and tutorial. Now I only know that ATen is the tensor library of pytorch (The source code of this is easier to read since they have clear meanings). pytorch/crsc is the integration part of c++ and python.</p>
<p>I also have tried to find some hint from the GitHub and read the pull record. But it is quite hard.</p>
<p>I am familiar with the basic grammar of C++ but am bad at organization of the code. My own past C++ code only includes several xxx.h and xxx.c. So I am so confused when facing such a big project. Can anyone give me some suggestions to read the source code?</p>
<p>Thanks!</p>

username: SimonW
cooked: <p>Sorry for the post being out of date. Currently I don’t think there is a good guide on this topic especially since that we are in the middle of a big refactoring. We may have something once PyTorch 1.0 is out. May I ask why you want to read the source code?</p>

username: JiamingSun
cooked: <p>Hi, I’m also interested in reading the source code of PyTorch. As a C++/CUDA newbie, I also found it’s hard to start digging into the code.</p>
<p>The reasons are mainly because I want to understand deeper about PyTorch(and Python) and become a better user of it. I’m having a hard time figuring out where the variables will be in(CPU/GPU, which GPU etc.) and how the variables are passed(reference or copy) in memory when I’m writing data augmentation code on GPU. I’m also interested in the way how PyTorch interacts with Python and C++/CUDA code. And since it’s such a great library, maybe I could become a contributor when I have a better understanding of the internals.</p>
<p>It would be nice if there’s a developer getting started guide available.</p>
<p>Thanks for writing this amazing library:grinning:</p>

username: SimonW
cooked: <p>Thanks for explaining!</p>
<p>For a very quick and high-level description:</p>
<ol>
<li>
<p>Tensor operations are implemented in <a href="https://github.com/pytorch/pytorch/tree/master/aten">the C++ Tensor library called ATen</a>, which also includes some old code from Torch time (we are rewriting those for PyTorch 1.0!). It’s important to note that ATen doesn’t have autograd functionality (at least not currently) and only provides a Tensor API and operators on Tensors of various types, e.g., one of them being <code>CPUFloatTenor</code>.</p>
<p>Since there are many different Tensor types and different operators, <strong>a lot</strong> of codegen is applied here, which unfortunately makes the code harder to read for new contributors. But we are constantly improving upon it!</p>
</li>
<li>
<p>To connect ATen with Python, we have a lot of C++ code living under <a href="https://github.com/pytorch/pytorch/tree/master/torch/csrc"><code>torch/csrc</code></a>, implementing things like <code>autograd</code>, <code>jit</code>, <code>dtype</code> <code>device</code> <code>layout</code> objects, etc. In particular, for <code>autograd</code>, we subclass each of the ATen tensor types with something called <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/VariableType.h"><code>VariableType</code></a> that implements functionalities to track computation graph (i.e., history), which <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.h">the <code>autograd</code> engine</a> will use to back-prop through and compute gradients.</p>
<p>Similar to ATen, we also use some codegen to generate the operator interface on <code>VariableTypes</code>, and more importantly generate the backward pass for each operator. The mapping between forward and backward functions are listed in <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml">a yaml file</a>, with many backward functions implemented in <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/Functions.cpp"><code>Functions.cpp</code></a>. Note that all operations in <code>Functions.cpp</code> are operating on <code>Variable</code>s, i.e., Tensors with <code>VariableType</code>, which means that if a function, e.g. <code>my_op_backward</code>, is implemented using existing operations that support <code>autograd</code>, the double backward also works out of the box! (Of course one may also implement a custom double backward for efficiency reasons.)</p>
</li>
<li>
<p>Finally, the connection between C++ land (i.e., <code>torch/csrc</code>) and Python is mostly implemented with <a href="https://docs.python.org/2/c-api/index.html">Python C API</a> and <a href="https://github.com/pybind/pybind11">pybind11</a>.</p>
</li>
</ol>

username: SimonW
cooked: <p>Oh I just realized that you are not OP LOL. But hope my reply helps!</p>

username: christianperone
cooked: <p>I have made available a <a href="https://speakerdeck.com/perone/pytorch-under-the-hood" rel="nofollow noopener">slide deck with some details about PyTorch internals</a> in case you’re interested.</p>
<p>There are also two other articles I wrote about it in case you’re interested:</p>
<ul>
<li><a href="http://blog.christianperone.com/2018/10/pytorch-1-0-tracing-jit-and-libtorch-c-api-to-integrate-pytorch-into-nodejs/" rel="nofollow noopener">PyTorch 1.0 tracing JIT and LibTorch C++ API to integrate PyTorch into NodeJS</a></li>
<li><a href="http://blog.christianperone.com/2018/03/pytorch-internal-architecture-tour/" rel="nofollow noopener">PyTorch – Internal Architecture Tour</a></li>
</ul>
<p>Hope it helps !</p>

username: baldassarre.fe
cooked: <p>The slide deck says “currently unavailable”. Can you check if it’s a problem with the slides or with the website?</p>

username: christianperone
cooked: <p>It was something temporary with the website probably, because I can now access it without problems.</p>

username: andreiliphd
cooked: <p>“A tour of PyTorch internals” is a great article!</p>
<p>Article in short:</p>
<ol>
<li>We get a generic C++ tensor and customize it for a particular type of tensor  with <code>split_types</code> function which is available here <a href="https://github.com/pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/tools/setup_helpers/split_types.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/tools/setup_helpers/split_types.py</a> The tutorial mention that it’s called from setup.py I have not found it there anymore.</li>
<li>The same logic for C++ headers.</li>
<li>Then there is a cwrap that creates C++ methods for particular objects. It uses YAML file to do that. cwrap reads YAML and generate the code for methods.</li>
<li>Tutorial mentions CPython but on Github I found pybind11 to handle interactions of Python and C++.</li>
</ol>
<p>Could FB make "A Tour of PyTorch Internals (Part III)? We really need updated version. And <code>split_types</code> and <code>cwrap</code> make life really complicated. I don’t know about others but for that I would definitely wait even 30 minutes for PyTorch to build to finally know what happens.</p>
<p>UPDATE: Well it’s already an hour since I build PyTorch from the source.</p>

username: kmario23
cooked: <p>In case someone is looking for a new blog post on this, here’s one from <a class="mention" href="/u/ezyang">@ezyang</a></p>
<p><a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="nofollow noopener"><code>PyTorch Internals for potential contributors</code></a></p>