How can I modify c++ files in pytorch?

jinwoosorj: <p>How can I modify c++ files such as caffe2 or csrc c files? I can only find header files so that I can’t try to modify to test what I want.</p>
<p>Pytorch is calling methods from c++ files and I want to make changes in the c++ files such as caffe2/serialize/inline_container to test what I want to see.</p>
<p>I am totally lost as a starter and its been 2 weeks I am trying to get familiar with this software…</p>
<p>I beg your generousity.</p>

ptrblck: <p>You can find the <code>inline_container.cc</code> file <a href="https://github.com/pytorch/pytorch/blob/e4f3e5434f4e980d9b6b40d59d75b2a8c0a4c86c/caffe2/serialize/inline_container.cc">here</a> and it will have the same relative location in your locally closed GitHub repository.<br>
Manipulate the file locally and rebuild PyTorch afterwards.</p>

jinwoosorj: <p>Thanks for answering! I was trying to follow the path according to the github path, but there are only headers… where can I find cpp files? I am using anaconda, would it be affecting?</p>

ptrblck: <p>You would need to <code>git clone</code> the repository, manipulate the file, and build PyTorch from source.<br>
You cannot manipulate C++ files in a pre-built binary and expect to see changes without compiling the code again.</p>

jinwoosorj: <p>So… if I just download pytorch from their website in local or conda, I cannot find the cpp files, but headers? I have to clone their github and build pytorch by myself?</p>

ptrblck: <p>Yes, since C++ is a compiled language you would need to clone the repository and rebuild PyTorch from source. Any changes in <code>.cpp</code> files won’t be visible in the compiled binary.</p>

jinwoosorj: <p>Thanks a lot! And could you please let me know any resources about how to recompile those c++ files after making changes to apply for pytorch?</p>

ptrblck: <p>You can follow <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a> from the repository.</p>

amgb20: <p>Hello,</p>
<p>I’m revisiting this discussion because I’m tackling a similar challenge. I’m working on a specialized version of a GRU, incorporating an internal attention mechanism within the cell. The obstacle I face is that the mathematical operations seem to be implemented in C++ rather than Python, complicating direct modifications in the Python layer.</p>
<p>It seems that I would need to clone the raw pytorch repo from github,make my changes, and then compile it?</p>
<p>Assuming that’s the case, I have a concern: my existing setup depends on PyTorch installed via pip. How would my existing codebase recognize and utilize the locally modified version of PyTorch? What steps are required to ensure that my modifications are acknowledged by my current PyTorch-dependent code?</p>
<p>Thank you for your insights.</p>

ptrblck: <p>You would uninstall the already installed PyTorch binary and would then follow the linked build instructions (after manipulating the code). You might want to use incremental build during the development stage to avoid full rebuilds. The <a href="https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md">contributing guide</a> gives you more information about it.</p>

amgb20: <p>thank you very much for your help.</p>
<p>I found this tutorial as well: <a href="https://CPP_extension" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>. Would it work?</p>

ptrblck: <p>Yes, writing a custom extension would also work.</p>

amgb20: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="187864" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes, since C++ is a compiled language you would need to clone the repository and rebuild PyTorch from source. Any changes in <code>.cpp</code> files won’t be visible in the compiled binary.</p>
</blockquote>
</aside>
<p>If I decide to download pytorch from source and modify my GRU gates, would building c files of my pytorch the only thing to do? Should I expect to change some kernels if I had a gate to my GRU?</p>

ptrblck: <p>I assume you want to manipulate the internal <code>GRUCell</code> implementation from <a href="https://github.com/pytorch/pytorch/blob/b5488cbe64fa9a183b298b4ef432ff35b70d3d79/aten/src/ATen/native/RNN.cpp#L755">here</a>. If so, you would need to manipulate this C++ file and rebuild PyTorch from source.</p>

amgb20: <p>Thank you for your answer.</p>
<p>I now have installed the source code of PyTorch. I have <code>pip install --verbose /pytoch</code> in my venv.</p>
<p>Now, if I understood well, I would need to go in <a href="https://github.com/pytorch/pytorch/blob/b5488cbe64fa9a183b298b4ef432ff35b70d3d79/aten/src/ATen/native/RNN.cpp#L755" rel="noopener nofollow ugc">here</a> and do my changes to the gate. After this I would need to redo <code>pip install --verbose /pytoch</code> in my venv? right? If everything went well, while calling nn.GRU, it should call the modified gru cell within the C++ file?</p>
<p>While changing the GRU cell, is there a cpu RNN.cpp and a cuda GPU RNN.cpp implementation? Inside the GRU cell, should I worry about changing kernels, tensors, etc? I just need to implement the following where the red lines are the added bit.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/386af4192122c00b89c83ec295a3576ce0b43962.png" alt="Screenshot from 2024-04-02 23-43-51" data-base62-sha1="835SSDdHudaEGRE7Lr0y4E1gxCq" width="449" height="243"></p>

sprakashdash: <p>Follow this link [<a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">Build from Source - PyTorch</a>] to build from source. Generally when you want to make core edits the cleanest way is to have the repo locally using <code>git clone</code> , then do necessary changes and the follow the same instruction to build from source again. As in your case you are changing the c++ files (changing python files won’t require building from source)</p>

amgb20: <p>Hello I am to the point of changing the C++ code located in <a href="https://github.com/pytorch/pytorch/blob/b5488cbe64fa9a183b298b4ef432ff35b70d3d79/aten/src/ATen/native/RNN.cpp#L755" rel="noopener nofollow ugc">here</a> for the diagram implementation above. Since there are 2 more gates (ReLU and Softmax), I am ensure how to implement it in C++.</p>
<p>I would really much appreciate if somebody could help me implement it as I am ensure how to manipulate the c++ code. I come from a python background and I am a bit lost.</p>
<p>Cheers <a class="mention" href="/u/sprakashdash">@sprakashdash</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

sprakashdash: <p>Well if you are terrified of c++ scripts, you can try modifying the python script located <a href="https://github.com/pytorch/pytorch/blob/eff1e4899c7c89f8a8fc8f6ff6bed06dd8d2ec8a/torch/nn/modules/rnn.py#L369" rel="noopener nofollow ugc">here</a>.<br>
Do note that I have shared a specific branch’s version so check the path and try to locate the same block in your local pytorch version. And by the way pytorch works for both CPU and GPU when you make changes to this code (in most of the cases).<br>
Best</p>

amgb20: <p>Hi,</p>
<p>Sorry for the delay.</p>
<p>I am still convinced that I can do it by modifying the raw c++ file. i will guide you through what I have done so you can fully understand and hopefully guide me through the right direction.</p>
<p>So I re-iterate my problem: I would like to change the internal structure of a GRU cell to implement what we call a recurrent attention unit (RAU) (<a href="https://www.sciencedirect.com/science/article/pii/S0925231222013339?via%3Dihub" rel="noopener nofollow ugc">here</a>) as it can be seen in the above image. It is just a matter of adding those two gates (ReLU and Softmax).</p>
<ol>
<li>
<p>I have downloaded pytorch from source by executing the following command: <code>git clone --recursive https://github.com/pytorch/pytorch</code> → <code>cd pytorch</code> → <code>git submodule sync</code> → <code>git submodule update --init --recursive</code>. After this I executed: <code>export _GLIBCXX_USE_CXX11_ABI=1</code>.</p>
</li>
<li>
<p>When the installation was finished, I went to <strong>pytorch/aten/src/ATen/native/RNN.cpp</strong> and I have changed the following structure to implement my RAU:</p>
</li>
</ol>
<pre><code class="lang-auto">template &lt;typename cell_params&gt;
struct GRUCell : Cell&lt;Tensor, cell_params&gt; {
  using hidden_type = Tensor;
 hidden_type operator()(
      const Tensor&amp; input,
      const hidden_type&amp; hidden,
      const cell_params&amp; params,
      bool pre_compute_input = false) const override {
    if (input.is_cuda() || input.is_xpu() || input.is_privateuseone()) {
      TORCH_CHECK(!pre_compute_input);
      auto igates = params.matmul_ih(input);
      auto hgates = params.matmul_hh(hidden);
      auto result = at::_thnn_fused_gru_cell(
          igates, hgates, hidden, params.b_ih(), params.b_hh());
      // Slice off the workspace argument (it's needed only for AD).
      return std::move(std::get&lt;0&gt;(result));
    }
    const auto chunked_igates = pre_compute_input
        ? input.unsafe_chunk(5, 1)
        : params.linear_ih(input).unsafe_chunk(5, 1);
    auto chunked_hgates = params.linear_hh(hidden).unsafe_chunk(5, 1);
    const auto reset_gate =
        chunked_hgates[0].add_(chunked_igates[0]).sigmoid_();
    const auto input_gate =
        chunked_hgates[1].add_(chunked_igates[1]).sigmoid_();
    const auto new_gate =
        chunked_igates[2].add(chunked_hgates[2].mul_(reset_gate)).tanh_();
    const auto attention_gate_ReLU =
        chunked_hgates[3].add(chunked_igates[3]).relu_();
    const auto attention_gate_softmax =
        at::softmax(chunked_hgates[4].add(chunked_igates[4]), /*dim=*/-1);
    auto gru_normal = (hidden - new_gate).mul_(input_gate).add_(new_gate);
    auto rau = gru_normal + attention_gate_ReLU.mul(attention_gate_softmax);
#warning "C Preprocessor got here!"
    return rau; 
  }
};
</code></pre>
<p>I have also added the following dependencies for softmax:</p>
<pre><code class="lang-auto">include &lt;ATen/ops/_log_softmax.h&gt;
#include &lt;ATen/ops/_log_softmax_backward_data_native.h&gt;
#include &lt;ATen/ops/_log_softmax_native.h&gt;
#include &lt;ATen/ops/_masked_softmax_backward_native.h&gt;
#include &lt;ATen/ops/_masked_softmax_native.h&gt;
#include &lt;ATen/ops/_softmax.h&gt;
#include &lt;ATen/ops/_softmax_backward_data_native.h&gt;
#include &lt;ATen/ops/_softmax_native.h&gt;
#include &lt;ATen/ops/empty.h&gt;
#include &lt;ATen/ops/empty_like.h&gt;
#include &lt;ATen/ops/log_softmax.h&gt;
#include &lt;ATen/ops/log_softmax_native.h&gt;
#include &lt;ATen/ops/softmax.h&gt;
#include &lt;ATen/ops/softmax_native.h&gt;
#include &lt;ATen/ops/special_log_softmax_native.h&gt;
#include &lt;ATen/ops/special_softmax_native.h&gt;
</code></pre>
<ol>
<li>I have created an virtual environment in which i will build the pytorch module:<br>
<code>python3 -m venv ./test2</code><br>
<code>(test2) $ pip install --verbose pytorch/.</code></li>
</ol>
<p>When I do this everything compiles very well and the warning message within the GRUCell struct fires on meaning that it compiles the right RNN.cpp file and not other dependencies.</p>
<p>Within my test2 venv I have two files which have been created: one <strong>torch</strong> and the other one <strong>torch-2.4.0a0+git8aa08b8.dist-info</strong></p>
<p>Now that everything is find I have checked that the torch module within my test2 venv was effectively communicating to my local source pytorch with my changes:</p>
<pre><code class="lang-auto">(test2) python3
&gt;&gt; import torch
&gt;&gt; print(torch.__version__)
2.4.0a0+git8aa08b8
</code></pre>
<p>it means that it is actually taking into account the local version of pytorch that I have downloaded.</p>
<p>Now let’s go in the python code side.</p>
<p>The goal is that when I call my <strong>nn.GRU</strong> that it takes into account my new GRU cell modified in my local pytorch version. Because the gates have changed from 3 to 5, I would need, within <strong>rnn.py</strong> to modified this line:</p>
<pre><code class="lang-auto">elif mode == 'GRU':
            gate_size = 3 * hidden_size
</code></pre>
<p>to this line:</p>
<pre><code class="lang-auto">elif mode == 'GRU':
            gate_size = 5 * hidden_size
</code></pre>
<p>However, it throws me an error.</p>
<p>My question is that i don’t know if I have missed any steps and I need help to understand what I am doing wrong.</p>
<p>cheers</p>

ptrblck: <aside class="quote no-group" data-username="amgb20" data-post="19" data-topic="187864">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amgb20/48/68926_2.png" class="avatar"> amgb20:</div>
<blockquote>
<p>However, it throws me an error.</p>
</blockquote>
</aside>
<p>What kind of error are you seeing as your workflow sounds valid?</p>