The model is too big to run on a single GPU with batch1

xingjici: <p>Hi.<br>
I am running a NAS (neural network search). The capacity of these models are too large even with batch_size =1 accounting for 18GB memory for a single GPU.<br>
I have 4GPUs with with 12GB memory (48 GB totaly), how to run these code?</p>

ptrblck: <p>Depending on the model architecture, you could try to apply model sharding as shown in <a href="https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239/2">this exmaple</a>.<br>
This approach would store submodules on different devices and transfer the output to the corresponding device in the <code>forward</code> method.</p>
<p>Would this be possible or is a single layer already creating the OOM issue?</p>

xingjici: <p>Thank you for your reply.  In network perimeter training , the gpu0 were fully used, but gpu1,2,3 half but it works.  when network architecture searching, it also create oom. In another words:<br>
epoch &lt; 5 :Data (4,3,224,224) GPU0:12GB GPU1~3 5GB<br>
epoch &gt;=5:  Data(4,3,224,224) and Architect[(20,8),(12,4,3)]<br>
How to do that or relax the gpu0 burden into others?</p>

ptrblck: <p>How did you split the model for the first runs?<br>
Did you manually “chopped” the model in different parts and pushed each to a different device?<br>
If so, try to move more layers/submodules to other devices.</p>
<p>I’m not sure, how the <code>Architect[(20, 8), (12, 4, 3)]</code> class works exactly, but I assume it’s creating different submodules?</p>

xingjici: <p>actually, the code is a implementation of autodeeplab. I follow your instruction to modify my code:</p>
<pre><code class="lang-auto">class Architect_search(nn.Module):
    def __init__(self,archi_search,data_model):
        super(Architect_search, self).__init__()
        self.archi_model = nn.DataParallel(archi_search,device_ids=[0,1]).to('cuda:0')
        self.data_model = nn.DataParallel(data_model,device_ids=[2,3]).to('cuda:2')
    def forward(self, data_image,search_image ):
        search_result = self.archi_search(data_image)
        train_result = self.data_model(search_image)
        return search_result,train_result
</code></pre>
<p>bug got errors:<br>
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:2<br>
specifically, the architect_search governs part of parameters of data_model.</p>
<pre><code class="lang-auto">class Architect_search (nn.Module) :
    def __init__(self, data_model, args):
        super(Architect_search, self).__init__()
        self.network_momentum = args.momentum
        self.network_weight_decay = args.weight_decay
        self.optimizer = torch.optim.Adam(model.arch_parameters(),
            lr=args.arch_lr, betas=(0.5, 0.999), weight_decay=args.arch_weight_decay)
        self.data_model = data_model


    def forward(self, input_valid):

            logits = self.model(input_valid)

            return logits
</code></pre>
<p>data_model is a common nn.Module class. arch_parameters will return [(20,8),(12,4,3)]<br>
20 potential connection x 8 opperation types every block.  12 layers x4blockx3concatnation<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/98b5286b778653db25e41c31b0ef318a052e784c.png" data-download-href="https://discuss.pytorch.org/uploads/default/98b5286b778653db25e41c31b0ef318a052e784c" title="WX20190716-111419%402x"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/98b5286b778653db25e41c31b0ef318a052e784c_2_690x159.png" alt="WX20190716-111419%402x" data-base62-sha1="lMUHG5oDedKxlaf6pGIhuNmnwCg" width="690" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/98b5286b778653db25e41c31b0ef318a052e784c_2_690x159.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/98b5286b778653db25e41c31b0ef318a052e784c_2_1035x238.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/98b5286b778653db25e41c31b0ef318a052e784c_2_1380x318.png 2x" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20190716-111419%402x</span><span class="informations">2108×486 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If you use different modules on different GPUs, you would also have to transfer the tensors in the <code>forward</code> to the device as given in the example.<br>
In your case <code>search_result</code> should be transferred to <code>'cuda:2'</code>.</p>

xingjici: <p>Thank you . It works.</p>