Correct way storing states inside one forward pass

username: Knievel
cooked: <p>What is the recommended/most efficient way of keeping and updating a buffer of states that I can access, all inside the forward pass of my network.</p>
<p>I wrote a custom RNN layer for multi dimensional sequences, but let’s take a 2D image as example. The layer takes two hidden states (for each dimension) and the input (pixel value) and returns a new hidden state. In the network I have to build a list of these hidden states to use them for the later computations.</p>
<p>Currently I’m initializing a tensor of the size of my images that holds all hidden states with</p>
<pre><code class="lang-auto">context_buffer = torch.empty(dim1, dim2, batch_size, self.hidden_size)
</code></pre>
<p>and then update it with</p>
<pre><code class="lang-auto">context_buffer.data[t1, t2] = h_t
</code></pre>
<p>I use <code>.data</code> because without it the backward pass would fail, but I don’t feel like this is the ideal solution. The network does actually learn, but the backward pass takes ages.</p>
<p>What is the best way to handle this? I looked at pytorch buffers but I feel like they are for storing data shared between several forward passes. Do I just have to change the way I assign the states? Concat them to a initial state instead of filling them in?</p>
<p>Any help appreciated!</p>

username: Knievel
cooked: <p>Update:</p>
<p>I now tried appending each new hidden state h_t along one column of the image with</p>
<pre><code class="lang-auto">states = torch.cat((states, h_t.unsqueeze(0)))
</code></pre>
<p>and then append this tensor to the other columns with</p>
<pre><code class="lang-auto">context_buffer = torch.cat((context_buffer, states.unsqueeze(0)))
</code></pre>
<p>This also works, but is even slower.</p>
<p>For reference, when passing a batch of 32 images (84x64) the average times are:</p>
<ol>
<li>Method: 0.5s Forward, 11s Backward</li>
<li>Method: 0.67s Forward, 13s Backward</li>
</ol>
<p>I really hope someone knows a better solution to this…</p>

username: albanD
cooked: <p>Hi,</p>
<p>You don’t want to use <code>.data</code> as it breaks the computational graph and so your gradients won’t be correct.<br>
If you just want to keep a bunch of states, the simplest is to keep them in a python list.<br>
There is no speed advantage in reusing a single buffer in most cases as the custom allocator we use will make the tensors creations almost free.</p>
<p>For the very slow backward, keep in mind that if you do 10 different forwards through your network, it will need to do 10 different backwards which can take a while as the computational graph is quite big.<br>
If you don’t want gradient to flow back, you can use <code>.detach()</code> to prevent that.</p>

username: Knievel
cooked: <p>Hey,</p>
<p>thanks for the answer, it definitely helps, but also confused me a little.</p>
<p>If I store the computed states in a simple python list, I need to detach them first. Does this not break the backward pass when I use them again for the next time step(s)?</p>
<p>This is probably not a PyTorch question, I guess I’ll go wrap my head around the formulas again…</p>

username: albanD
cooked: <p>You don’t have to detach() them before putting them in the list <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Be careful though to detach everything that should be and make sure the list empties out not to leak a lot of tensors !</p>

username: Knievel
cooked: <p>Reviving this thread, because I now encounter a problem that I think is related to your last sentence.</p>
<p>Same project, I’m now using a simple python list to store my states and then generate a tensor from it with torch.stack, for following computations.</p>
<p>Now during training I need loads of memory. Current runs with multiple layers and larger states fail on my universities cluster because they run out of memory (251GB are available on one node).</p>
<p>What exactly do you mean by</p>
<blockquote>
<p>make sure the list empties out not to leak a lot of tensors</p>
</blockquote>
<p>And how do I achieve this?</p>

username: Knievel
cooked: <p>From this <a href="https://discuss.pytorch.org/t/memory-leak-when-appending-tensors-to-a-list/25937">thread</a> I found out that I apparently store the whole computational graph in my list with each iteration.</p>
<p>How am I supposed to handle this if I need the states for the gradient computation? Can I still detach them at some point?</p>