How do I put all the data into nn.dataparallel during eval() mode?

username: TanmDL
cooked: <p>During training nn.dataparallel() working perfectly but eval mode it is showing this bellow error,<br>
RuntimeError: CUDA out of memory. Tried to allocate 29.21 GiB (GPU 0; 7.93 GiB total capacity; 361.73 MiB already allocated; 6.15 GiB free; 34.27 MiB cached)</p>
<p>Example:<br>
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
if torch.cuda.is_available:<br>
torch.cuda.manual_seed_all(1)</p>
<p>class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model, self).<strong>init</strong>()</p>
<pre><code>    self.feature = nn.Sequential(
        nn.Linear(int(np.prod(img_shape)), 512),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Linear(512, 10),
        nn.LeakyReLU(0.2, inplace=True))
    self.linear = nn.Sequential( nn.Linear(10, 1))


def forward(self, img):
    img_flat = img.view(img.shape[0], -1)
    img_flat = self.feature(img_flat)
    validity = self.linear(img_flat)
    #print("Discrimiantor model")
    #print("\tIn Model: input size", img.size(),
    #      "output size", validity.size())
    #print("#########################################################")
    return validity
</code></pre>
<p>if torch.cuda.device_count() &gt; 1:<br>
print(“Let’s use”, torch.cuda.device_count(), “GPUs!”)</p>
<h1>dim = 0 [30, xxx] -&gt; [10, …], [10, …], [10, …] on 3 GPUs</h1>
<p>model = nn.DataParallel(model)<br>
Training is perfectly working.<br>
######################## Eval mode<br>
model.eval()<br>
input = torch.randn(10000000,784).to(device)<br>
feature = model.module.feature(input)<br>
print(feature)</p>
<p>I am currently using 4 GPUs with 8GB RAM.</p>

username: ptrblck
cooked: <p>Did you also call <code>model.module.feature</code> during training? It seems to be a custom method, so that I think <code>nn.DataParallel</code> won’t be applied.<br>
Also, wrap your code in a <code>with torch no_grad()</code> block to save some memory during evaluation.</p>

username: TanmDL
cooked: <p>" Did you also call <code>model.module.feature</code> during training?" , No, I am using normal training loss. It is working perfectly.<br>
Do I need to wrap the “input” with dataloader during testing?</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will call the wrapped module directly (<code>__call__</code> first, then inside it <code>forward</code>).<br>
If you call some custom methods from the model, data parallel won’t be used, so instead call your model directly again.</p>

username: TanmDL
cooked: <p>sorry. I couldn’t get it. Please give me a simple example so that I can understand.</p>

username: TanmDL
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,  “If you call some custom methods from the model, data parallel won’t be used, so instead call your model directly again.”. I am planning to take two outputs from the forward method of the model i.e. validity and feature. Would it help me to solve the problem? I am confused.</p>

username: TanmDL
cooked: <p>same error.  “RuntimeError: CUDA out of memory. Tried to allocate 29.21 GiB (GPU 0; 7.93 GiB total capacity; 361.89 MiB already allocated; 6.17 GiB free; 14.11 MiB cached)”<br>
I have tried with the below approach,<br>
model.eval()<br>
with torch.no_grad():<br>
input = torch.randn(10000000,784).to(device)<br>
_, feature = model(input)<br>
print(feature.size())<br>
feature_data = feature.data.cpu().numpy()</p>

username: ptrblck
cooked: <p>In that case your input might be just too large. How many GPUs are you using and were you able to feed 1e7 samples before?</p>

username: TanmDL
cooked: <p>I am currently using 4 GPUs with 8GB RAM (4*8=32GB).<br>
“were you able to feed 1e7 samples before?” No.  This is a simple example of my original code where I need to feed almost 20GB data into the model for extracting features. Is it possible with 4GPUs ? Is there anyway to parallelize the data over all three GPUs and extracting features from the model (cuda:‘0’)?</p>