Autograd issues with time series LSTM model in libtorch

rutledgea20: <p>I’ve been attempting to learn libtorch by converting this time sequence prediction model to c++: <a href="https://github.com/pytorch/examples/tree/main/time_sequence_prediction" rel="noopener nofollow ugc">examples/time_sequence_prediction at main · pytorch/examples (github.com)</a></p>
<p>Using this page as a reference for C++ syntax: <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html#the-discriminator-module" rel="noopener nofollow ugc">Using the PyTorch C++ Frontend — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>
<p>So far I believe I have successfully set up the model:</p>
<pre><code class="lang-auto">struct LSTMPredictorImpl : torch::nn::Module {
    LSTMPredictorImpl(int n_hidden = 51) :
        lstm1(torch::nn::LSTMCell(1, n_hidden)),
        lstm2(torch::nn::LSTMCell(n_hidden, n_hidden)),
        linear1(torch::nn::Linear(n_hidden, 1))
    {
        register_module("lstm1", lstm1);
        register_module("lstm2", lstm2);
        register_module("linear1", linear1);
        this-&gt;n_hidden = n_hidden;
    }
    torch::Tensor forward(torch::Tensor x, int future = 0) {
        int n_samples = x.sizes()[0];
        std::vector&lt;torch::Tensor&gt; outputs;
        std::tuple&lt;torch::Tensor, torch::Tensor&gt; hc_t1(
            torch::zeros({n_samples, n_hidden}), 
            torch::zeros({n_samples, n_hidden}));
        std::tuple&lt;torch::Tensor, torch::Tensor&gt; hc_t2(
            torch::zeros({n_samples, n_hidden}), 
            torch::zeros({n_samples, n_hidden}));

        torch::Tensor output;
        std::vector&lt;torch::Tensor&gt; separated = x.split(1, 1);
        for (torch::Tensor input_t : separated) {
            hc_t1 = lstm1(input_t, hc_t1);
            hc_t2 = lstm2(std::get&lt;0&gt;(hc_t1), hc_t2);
            output = linear1(std::get&lt;0&gt;(hc_t2));
            outputs.push_back(output);
        }
        for (int i = 0; i &lt; future; i++) {
            hc_t1 = lstm1(output, hc_t1);
            hc_t2 = lstm2(std::get&lt;0&gt;(hc_t1), hc_t2);
            output = linear1(std::get&lt;0&gt;(hc_t2));
            outputs.push_back(output);
        }
        torch::TensorList intermediate = torch::TensorList(separated);
        torch::Tensor final_output = torch::cat(intermediate, 1);
        return final_output;
    }
    torch::nn::LSTMCell lstm1, lstm2;
    torch::nn::Linear linear1;
    int n_hidden;
};
TORCH_MODULE(LSTMPredictor);
</code></pre>
<p>I have generated sine wave training data with noise in the format of a 2nd degree Tensor, where each row contains data for a single sine wave and there are 100 rows.</p>
<p>Here is my function that I use to try to train on that data:</p>
<pre><code class="lang-auto">void time_series_train(torch::Tensor data) {
    auto data_sizes = data.sizes();
    torch::Tensor train_input = data.index({
        torch::indexing::Slice(3),
        torch::indexing::Slice(0, -1)
    });
    torch::Tensor train_target = data.index({
        torch::indexing::Slice(3),
        torch::indexing::Slice(1)
    });
    torch::Tensor test_input = data.index({
        torch::indexing::Slice(0, 3),
        torch::indexing::Slice(0, -1)
    });
    torch::Tensor test_target = data.index({
        torch::indexing::Slice(0, 3),
        torch::indexing::Slice(1)
    });

    LSTMPredictor predictor;
    torch::optim::LBFGS LSTM_optimizer(
        predictor-&gt;parameters(), torch::optim::LBFGSOptions(0.8)
    );
    int n_training_steps = 10;
    for (int i = 0; i &lt; n_training_steps; i++) {
        auto closure = [predictor, train_input, train_target, i] () mutable {
            predictor-&gt;zero_grad();
            torch::Tensor output = predictor-&gt;forward(train_input);
            torch::Tensor loss = torch::mse_loss(output, train_target);
            cout &lt;&lt; "Current step: " &lt;&lt; i &lt;&lt; ", loss: " &lt;&lt; loss.item&lt;float&gt;() &lt;&lt; endl;
            
            loss.backward();
            return loss;
        };
        LSTM_optimizer.step(closure);
    }
}
</code></pre>
<p>When I run this function, it reaches the cout statement inside my closure lambda function, and after that I get this output:</p>
<blockquote>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  element 0 of tensors does not require grad and does not have a grad_fn<br>
Exception raised from run_backward at …/torch/csrc/autograd/autograd.cpp:91 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x3e (0x7fce15506f0e in /(home)/.local/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x5c (0x7fce154e17ce in /(home)/.local/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x3efaef0 (0x7fcdffab6ef0 in /(home)/.local/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span> torch::autograd::backward(std::vector&lt;at::Tensor, std::allocator&lt; at::Tensor&gt; &gt; const&amp;, std::vector&lt;at::Tensor, std::allocator&lt; at::Tensor&gt; &gt; const&amp;, c10::optional, bool, std::vector&lt;at::Tensor, std::allocator&lt; at::Tensor&gt; &gt; const&amp;) + 0x5c (0x7fcdffab8efc in /(home)/.local/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0x3f54e5e (0x7fcdffb10e5e in (home)/.local/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> at::Tensor::_backward(c10::ArrayRef&lt; at::Tensor&gt;, c10::optional&lt; at::Tensor&gt; const&amp;, c10::optional, bool) const + 0x48 (0x7fcdfd4109e8 in /(home)/.local/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#6:</span> at::Tensor::backward(at::Tensor const&amp;, c10::optional, bool, c10::optional&lt;c10::ArrayRef&lt; at::Tensor&gt; &gt;) const + 0x11d (0x43e915 in ./PyTorchTest)<br>
frame <span class="hashtag-raw">#7:</span> ./PyTorchTest() [0x43ae88]<br>
frame <span class="hashtag-raw">#8:</span> ./PyTorchTest() [0x43cd2e]<br>
frame <span class="hashtag-raw">#9:</span> torch::optim::LBFGS::step(std::function&lt;at::Tensor ()&gt;) + 0x27a (0x7fce0066f50a in /(home)/.local/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#10:</span> time_series_train(at::Tensor) + 0x87b (0x43b8c5 in ./PyTorchTest)<br>
frame <span class="hashtag-raw">#11:</span> main + 0x10d (0x43ac9d in ./PyTorchTest)<br>
frame <span class="hashtag-raw">#12:</span> __libc_start_main + 0xf5 (0x7fcdfadb0555 in /lib64/libc.so.6)<br>
frame <span class="hashtag-raw">#13:</span> ./PyTorchTest() [0x43a8c9]<br>
Aborted (core dumped)</p>
</blockquote>
<p>From what I can tell, the only important line in all that is " element 0 of tensors does not require grad and does not have a grad_fn". I’ve tried disabling autograd globally by putting this line in various locations:</p>
<pre><code class="lang-auto">torch::NoGradGuard no_grad;
</code></pre>
<p>But that did not work, probably because I have no clue what I’m doing when it comes to autograd. I suspect maybe it’s something to do with my closure lambda function being setup weird, as I couldn’t find anything definitive online for passing closure functions in libtorch for c++.</p>
<p>Any solutions or general tips on ways I’m doing libtorch incorrectly would be appreciated, Thank you.</p>

rutledgea20: <p>It seems my issue was that in the forward pass function I created, tensors were being initialized without require_grad() set to true. Adding the line:</p>
<pre><code class="lang-auto">final_output.requires_grad_(true);
</code></pre>
<p>right before returning in the forward pass function stops the error from showing up.</p>