How to free CPU memory after inference in libtorch?

username: zcm
cooked: <p>Hi,<br>
Here’s my question: I is inferring image on GPU in libtorch.  it occupies large amount of CPU memory(2G+), when I run the code as fallow:</p>
<pre><code class="lang-auto">output = net.forward({ imageTensor }).toTensor();
</code></pre>
<p>Until the end of the main function, the CPU memory remains unfreed. I alse try to run “c10::cuda::CUDACachingAllocator::emptyCache();”, but nothing happened.<br>
Wath can I do to free CPU memory?<br>
<a class="mention" href="/u/tom">@tom</a> homas V <a class="mention" href="/u/alband">@albanD</a></p>

username: tom
cooked: <p>Two things:</p>
<ul>
<li>be sure to have autograd disabled,</li>
<li>put output in a local C++ scope so it goes out of scope.<br>
Also, the network parameters would still be around, right?</li>
</ul>
<p>That should help.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zcm
cooked: <p>Thank you for your attention!<br>
My simplified version of the code is as follow:</p>
<pre><code class="lang-auto">bool segMain(vtkImageData* vtkImageDataPixel,vtkImageData* outImageData, std::string modelPath)
{
	auto pixelDataShape = vtkImageDataPixel-&gt;GetDimensions();
	auto spacing = vtkImageDataPixel-&gt;GetSpacing();
	int imageDims[3];
	vtkImageDataPixel-&gt;GetDimensions(imageDims);
	short* oriPixelPointer = static_cast&lt;short*&gt;(vtkImageDataPixel-&gt;GetScalarPointer());   //get data pointer

	auto device = torch::kCUDA;     // set cuda
	torch::jit::script::Module module; // load model
	module = torch::jit::load(modelPath);

	module.to(device);
	module.eval();

	at::set_num_threads(1);

	at::Tensor imageTensor = torch::from_blob(oriPixelPointer, { imageDims[2], imageDims[0], imageDims[1] }, torch::kShort).to(torch::kFloat32);
	imageTensor = torch::flip(imageTensor, { 0,1,2 });
	auto imageTensorNorm = (imageTensor + 1024.0);
	at::Tensor modelOutputTensor = torch::ones({ imageDims[2], imageDims[0], imageDims[1] }, torch::kFloat32);
	at::Tensor imageTensorSlicer, modelOutputTensorSlicer;

	for (int i = 0; i &lt; imageDims[2]; i++)
	{
		imageTensorSlicer = imageTensorNorm[i].unsqueeze(0).unsqueeze(0);
		imageTensorSlicer = imageTensorSlicer.to(device);     // predict
		modelOutputTensorSlicer = module.forward({ imageTensorSlicer }).toTensor();
		modelOutputTensorSlicer = torch::one_hot(torch::argmax(torch::softmax(modelOutputTensorSlicer, 1), 1), 2).permute({ 0, 3, 1, 2 }).detach();
		modelOutputTensor[i] = modelOutputTensorSlicer[0][1].data();
	}
	
	modelOutputTensor = torch::flip(modelOutputTensor, { 0,1,2 });
	modelOutputTensor = modelOutputTensor.to(torch::kCPU);

	modelOutputTensor = modelOutputTensor.to(torch::kShort);
	vtkSmartPointer&lt;vtkImageData&gt; vtkModelOutputImageData = vtkImageData::New();
	vtkModelOutputImageData-&gt;SetDimensions(pixelDataShape[0], pixelDataShape[1], pixelDataShape[2]);
	vtkModelOutputImageData-&gt;SetExtent(0, pixelDataShape[0] - 1, 0, pixelDataShape[1] - 1, 0, pixelDataShape[2] - 1);
	vtkModelOutputImageData-&gt;AllocateScalars(VTK_SHORT, 1);
	vtkModelOutputImageData-&gt;SetSpacing(spacing[0], spacing[1], spacing[2]);
	short* pixelPointer = static_cast&lt;short*&gt;(vtkModelOutputImageData-&gt;GetScalarPointer());   //get data pointer
	memcpy(pixelPointer, modelOutputTensor.data_ptr(), sizeof(short) * pixelDataShape[0] * pixelDataShape[1] * pixelDataShape[2]);

	c10::cuda::CUDACachingAllocator::emptyCache();
	
	outImageData-&gt;DeepCopy(vtkModelOutputImageData);
	free(vtkModelOutputImageData);
	
	return true;

}


int main()
{
	std::cout &lt;&lt; "main func start run ..." &lt;&lt; std::endl;
	std::string dataDirPath = "D:\\sourceData.mhd";
	std::string modelPath = "C:\\net.pt";

	auto imageReader = vtkSmartPointer&lt;vtkMetaImageReader&gt;::New();
	imageReader-&gt;SetFileName(dataDirPath.c_str());
	imageReader-&gt;Update();
	vtkImageData* vtkImageData = imageReader-&gt;GetOutput();

	auto outImage = vtkImageData::New();
	auto r = segMain(vtkImageData, outImage, modelPath);

	outImage-&gt;Delete();
	cin.clear();
	cin.sync();

	Sleep(10000);

	cout &lt;&lt; " stop " &lt;&lt; endl;
}
</code></pre>
<p>I think I have done the two things. but until it run to “cout &lt;&lt; " stop " &lt;&lt; endl;” , the CPU memory have not been released. And it will take up the same amount of memory, when I run segMain twice.<br>
As far as know,  the network parameters and the tensor is  similar to a c++ smart pointer. But the CPU memory remains unfreed, even after a long sleep.<br>
Thank you for the response.</p>
<p>Best regards<br>
cm</p>