Getting out of memory in validation even using torch.no_grad()

username: kalpeshjp89
cooked: <p>My training code running good with around 8GB but when it goes into validation, it show me out of memory for 16GB GPU. I am using model.eval() and torch.no_grad() also but getting same. Here is my testing code for reference of testing which I am using in validation.</p>
<p>def test(self):<br>
self.netG1.eval()<br>
self.netG2.eval()<br>
with torch.no_grad():<br>
self.SR = self.netG1(self.var_L)<br>
self.LR_est = self.netG2(self.SR)<br>
self.DR = self.netG2(self.var_H)<br>
self.HR_est = self.netG1(self.DR)<br>
self.netG1.train()<br>
self.netG2.train()</p>
<p>def get_current_visuals(self, need_HR=True):<br>
out_dict = OrderedDict()<br>
<span class="hashtag">#out_dict</span>[‘LR’] = self.var_L.detach()[0].float().cpu()<br>
out_dict[‘SR’] = self.SR.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘LR_est’] = self.LR_est.detach()[0].float().cpu()<br>
if need_HR:<br>
out_dict[‘HR’] = self.var_H.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘DR’] = self.DR.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘HR_Est’] = self.HR_est.detach()[0].float().cpu()<br>
return out_dict</p>
<p>And my validation code is as bellow.<br>
if current_step % opt[‘train’][‘val_freq’] == 0:<br>
avg_psnr = 0.0<br>
idx = 0<br>
for val_data in val_loader:<br>
idx += 1<br>
img_name = os.path.splitext(os.path.basename(val_data[‘LR_path’][0]))[0]<br>
img_dir = os.path.join(opt[‘path’][‘val_images’], img_name)<br>
util.mkdir(img_dir)</p>
<pre><code>                model.feed_data(val_data)
                model.test()

                visuals = model.get_current_visuals()
                sr_img = util.tensor2img(visuals['SR'])  # uint8
                gt_img = util.tensor2img(visuals['HR'])  # uint8
</code></pre>
<p>.<br>
.<br>
.<br>
.<br>
.<br>
Waiting for any solution. Thanks in advance.</p>

username: ptrblck
cooked: <p>Which step is causing the OOM issue and is the mmeory usage 8GB, when you enter the <code>test</code> method?<br>
Did you increase the batch size of <code>self.var_L</code> (and other inputs)?</p>

username: kalpeshjp89
cooked: <p>Thank you for your quick response.<br>
When training step is executed, it consumes less than 8 GB. But when it goes into validation, at “model.test()” step it throws error. And particular in testing code, it is in “self.LR_est = self.netG2(self.SR)” line.</p>
<p>Currently I stopped using two models in validation and now it is working(with only one model), but using two models it give me error.</p>
<p>In test method, it iteratively consumes more and more memory until all 16GB and then throwing error.</p>
<p>In testing, I am using batch size of 1 only.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kalpeshjp89" data-post="3" data-topic="73305">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/aca169/48.png" class="avatar"> kalpeshjp89:</div>
<blockquote>
<p>In test method, it iteratively consumes more and more memory until all 16GB and then throwing error.</p>
</blockquote>
</aside>
<p>This points towards a potential storage of a tensor, which is attached to the computation graph.<br>
I cannot find anything obviously wrong in the posted code snippet, but could you check, if you are storing some outputs with a valid <code>.grad_fn</code> in a list/dict etc.?</p>

username: kalpeshjp89
cooked: <p>def get_current_visuals(self, need_HR=True):<br>
out_dict = OrderedDict()<br>
<span class="hashtag">#out_dict</span>[‘LR’] = self.var_L.detach()[0].float().cpu()<br>
out_dict[‘SR’] = self.SR.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘LR_est’] = self.LR_est.detach()[0].float().cpu()<br>
if need_HR:<br>
out_dict[‘HR’] = self.var_H.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘DR’] = self.DR.detach()[0].float().cpu()<br>
<span class="hashtag">#out_dict</span>[‘HR_Est’] = self.HR_est.detach()[0].float().cpu()<br>
return out_dict</p>
<p>This is my code for storing data from validation. I used detach here also.</p>

username: Oktai15
cooked: <p>Please, check this thread: <a href="https://discuss.pytorch.org/t/increase-the-cuda-memory-twice-then-stop-increasing/28462/8" class="inline-onebox">Increase the CUDA memory twice then stop increasing</a></p>