Libtorch memory on training

ghfkddorl: <p>I created a PointNet module with libtorch. It has a point cloud input as {min batch size, 3, pointCount(about 500000)} shape. But, when I training the model, It consumes so many memory.</p>
<p>As I know, on training, ‘conv, linear, batchNormalize…’ hold it’s input tensor for backward. If input tensor shape is {4, 3, 500000} and float32 type, it has 24000000 (=4 x 3 x 500000 x 4) byte on memory. is it true? And if it is True. can I change storage for the values with libtorch?</p>

ptrblck: <p>It is true that some layers need to store intermediate tensors to be able to calculate the gradients during the backward pass.<br>
If you are running out of memory you could either decrease the batch size (or number of points, if possible). I don’t know, if e.g. <code>torch.utils.checkpoint</code> can be used in <code>libtorch</code>, but that might be another approach to trade compute for memory.</p>

ghfkddorl: <p>Thanks for your reply. I think <code>torch.utils.checkpoint</code> is useful for me.</p>
<p>But I have some questions of meomory use of torch. For monitoring memory useage of torch, I run my custom model which contains code below in both way <code>model-&gt;eval()</code> and <code>model-&gt;train()</code>.</p>
<pre><code class="lang-cpp">printShape(x);  [4, 3, 40695]
printMemory();
x = conv1x1(x);
printShape(x);  //[4, 64, 40695]
printMemory(); //increase about 41mb
x = bn1(x);
printShape(x);  //[4, 64, 40695]
printMemory(); //increase about 41mb
x = func::relu(x);
printShape(x);  //[4, 64, 40695]
printMemory(); //increase about 41mb
</code></pre>
<p>But there is no difference between two way and a non-paramters module <code>relu</code> also takes some memory. It looks like all intermediate tensors always not be freed for some reason, until call <code>backward()</code> (and I run with a CPU, not GPU). Now, I am tring to know why tensor not released even if not used. can you help me?</p>

ghfkddorl: <p>I solved my problem.</p>
<pre><code class="lang-auto">torch::AutoGradMode enable_grad(false);
</code></pre>
<p>but I wonder that below result</p>
<pre><code class="lang-auto">data = data.set_requires_grad(false);
auto req_grad = data.requires_grad(); // flase

data = conv1(data);
auto req_grad = data.requires_grad(); // true
</code></pre>

ptrblck: <p>Even though relu doesn’t use parameters, the intermediate tensors might be stored, if they are needed to calculate the gradients in the backward pass.<br>
The output of <code>conv1</code> would require gradients, if the gradient calculation wasn’t disabled for the code block, so could you double check if the <code>AutoGradMode</code> is indeed set to <code>false</code> for this calculation?</p>

ghfkddorl: <p>Yes, <code>AutoGradMode</code> is set to <code>false</code>. and then, <code>data.requires_grad ()</code> always return <code>false</code>. My wonder is that <code>conv1(data)</code> return requires gradient, when <code>AutoGradMode = true</code> and input data not requires gradient. I thought that <code>inpudata.requires_grad(false)</code> is enougth to prevent creating gradient. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

ptrblck: <p>Usually the input data doesn’t require gradients and the output of a layer with trainable parameters will require gradients.<br>
Since <code>AutoGradMode</code> was set to <code>true</code>, the result is thus expected.</p>
<p>Here is a small code snippet in Python:</p>
<pre><code class="lang-python">x = torch.randn(1, 1)
print(x.requires_grad)
&gt; False

lin = nn.Linear(1, 1)
out = lin(x)
print(out.requires_grad)
&gt; True
</code></pre>
<p>This would be a very simple example of the “standard” use case, where you don’t need to calculate gradients in the input tensor.<br>
As long as you don’t disable the gradient calculation Autograd will create the computation graph (and will thus create outputs with <code>requires_grad=True</code>) for operations using trainable parameters.</p>