Finding maximal batch size according to GPU size

username: GMarzinotto
cooked: <p>Hello everyone,</p>
<p>I have been working on a code to train a neural network.<br>
and right now I’m working on a feature that finds the maximum batch size that can fit into memory<br>
for a given model and a training set.</p>
<p>So here is my code:</p>
<pre><code class="lang-auto">def get_free_memory():
    import GPUtil
    CUDA_VISIBLE_DEVICES = os.environ.get('CUDA_VISIBLE_DEVICES')
    memory = 0
    for GPU in GPUtil.getGPUs():
        if CUDA_VISIBLE_DEVICES is None or str(GPU.id) in CUDA_VISIBLE_DEVICES: 
            memory += GPU.memoryFree
    return memory


def no_free_mem( mem_per_sample, available ):
    return 5*np.array(mem_per_sample).max() &gt; available

def main():
    model = PytorchModel(config_network, config_inputs, config_outputs, "")
    model = model.cuda()

    if torch.cuda.device_count() &gt; 1:
        model = torch.nn.DataParallel(model)

    max_len = config_network["max_sequence_length"]

    x = get_first_sample(X, max_len, model.inputs_cfg)
    y = get_first_sample(Y, max_len, model.outputs_cfg, output=True)
    
    optimizer = optim.Adam(filter(lambda param: param.requires_grad, model.parameters()))

    moremem = True
    batch_size = 1
    prev_freemem = get_free_memory()
    mem_per_sample = [0]
    optimizer.zero_grad()
    
    while moremem:
        y_pred, _, _ = model(x)
        freemem = get_free_memory()
        if no_free_mem( mem_per_sample, freemem ): break
        
        loss, _ = model.loss(y_pred, y)
        
        freemem = min(freemem, get_free_memory())
        if no_free_mem( mem_per_sample, freemem ): break

        loss.backward()
        freemem = min(freemem, get_free_memory())
        if no_free_mem( mem_per_sample, freemem ): break

        optimizer.step()
        freemem = min(freemem, get_free_memory())
        if no_free_mem( mem_per_sample, freemem ): break

        if prev_freemem - freemem &gt; 0:
            mem_per_sample.append(prev_freemem - freemem)

        if no_free_mem( mem_per_sample, freemem ): break
    
        batch_size += 1
        prev_freemem = min(prev_freemem, freemem)

        x = insert_sample(x)
        y = insert_sample(y)
        
    print("GUESSING batch_size, ", batch_size)
</code></pre>
<p>I compute how much GPU memory is available at each step of the forward and backward passes<br>
And I expand the batch size iteratively until memory saturation. I keep track of how much memory is required for each sample in order to predict if I’ll be having enough memory to insert a new sample on the batch.<br>
In my head, this should work. however, this code gets unstable behavior.<br>
Sometimes it works fine and sometimes it crashes due to out of memory errors.<br>
I have tried to increase the margin of required memory on the stop condition, it crashes less but  nothing seems to work 100%<br>
So I guess I must be missing something important here.</p>
<p>A pattern that I have observed is that crashes are specially frequent when I do multi-task learning.<br>
And they occur in the loss.backward step most of the time.</p>
<p>Does someone could help me fix this problem or provide with another way to estimate memory-optimal batch sizes?<br>
I have found some equations in this forum but none of them worked for me in all the architectures I tested.</p>
<p>Thank you very much!<br>
Gabriel M</p>

username: shahensha
cooked: <p>Hi Gabriel (<a class="mention" href="/u/gabriel">@gabriel</a> ),</p>
<p>Another issue that you should consider while implementing such a thing is that in many models in neural networks, batch_size is a very sensitive parameters which affects the performance. It would be one thing to find out the best batch size for the entire training purpose and then keep it constant. But since you are changing it at every step, it might lead to instability in performance as well. I would advice you to test this hypothesis as well.</p>

username: YichengWang
cooked: <p>In the inference phase, adaptive batchsize is also necessary.</p>

username: YichengWang
cooked: <p>I have tried to achieve the similar feature like yours, and I encountered similar problem.</p>
<p>I guess the major problems are as follows:</p>
<ol>
<li>In the inference phase or BP phase, some tensors/grads can be deleted when corresponding specific computations are finished. However, in the beginning, pytorch does not know when to delete which tensor/grad, as it is a dynamic-graph framework, so the memory cost is high. After several passes, pytorch knows the architecture of CNNs, and delete tensors/grads as soon as possible in subsequent passes, so the memory cost is low.</li>
<li>PyTorch chooses base computation method according to batchsize and other situations, so the memory cost is not only related to batchsize.</li>
</ol>

username: ecolss
cooked: <p>Hi <a class="mention" href="/u/yichengwang">@YichengWang</a>, regarding what you said in 1, looks like after a few passes, pytorch will do the deletion automatically, is this confirmed somewhere in the pytorch documentation?</p>
<p>Also what is the motivation that we need to tune batch size when training a dnn model?<br>
Is it because we want to use 100% of the GPU memory so that it can speed up the training process?<br>
What if I don’t use 100% of the GPU, but say 80%, then my model would still converge, but it will take more batches to finish the training?</p>

username: TriKri
cooked: <p><a class="mention" href="/u/ecolss">@ecolss</a> I think it is to speed up the training process, yes.</p>
<p>Why would you want to use only 80% of the GPU’s memory?</p>

username: TriKri
cooked: <p><a class="mention" href="/u/yichengwang">@YichengWang</a> But can PyTorch really know whether a tensor will be used before it has been destroyed? I mean, depending on how the Python code is written, a tensor may be unused for a long time before its destruction in almost all passes, but suddenly in one pass it is used right before it is destroyed. If then PyTorch has decided that it can free up the memory on the GPU for that tensor early on, it has fucked up. Right?</p>