Training multiple machines in one GPU

Adrian_Briceno_Aguil: <p>Hi!</p>
<p>So I have a question, I have this recursive algorithm to train multiple Neural Networks in different domains, but same architecture , every time you go deeper into the algorithm the less training data you have, meaning it should be faster training the deeper you go.</p>
<p>I tested my algorithm with a small(2000) training set to be sure that it worked. And it does work! But now i want to try it with more real world training set, with size 262144(512x512). Since I knew I was going to work with the GPU so I tried to carefully move variables to the GPU only if I needed it.</p>
<p>When I ran it,  I saw that the first network I trained(with the biggest training set of 512x512 samples) trains quite fast it takes 130s. However, next one that is going to be trained using only 60% of the data to train it, takes  significantly way more(I have taken the time to see how much it actually takes because it so long). But if I test the network using that 60% of the data to train it, it takes approximately 50s.</p>
<p>Now that you have the context my question is the following. Is there any way that the first network I trained slows the second one?</p>
<p>I will show you the NN and the my training functions:</p>
<pre><code class="lang-auto">class MLPflat(nn.Module):
    def __init__(self,in_dim: int, out_dim: int, N, H):
      super().__init__()
      assert(N &gt; 0)
      assert(H &gt; 0)

      net = [nn.Linear(in_dim, H),nn.BatchNorm1d(H), nn.LeakyReLU()]
      
      for _ in range(N-1): # make N layers
        net += [nn.Linear(H, H),nn.BatchNorm1d(H), nn.LeakyReLU()]
        
      net += [nn.Linear(H,out_dim,bias=False)]
      self.model = nn.Sequential(*net)

    def forward(self, x):
      x = self.model(x)
      output = x
      return output
</code></pre>
<p>I use N(hidden layers) = 5 and H = 64.</p>
<pre><code class="lang-auto">def train(phi,train_loader,epochs,criterion,optimizer):
    fit_start_time = time.time()

    for epoch in range(epochs):
        batch = 0
        for x_batch, y_batch in train_loader:
            optimizer.zero_grad()
            print(x_batch.shape)
            y_pred = phi(x_batch.to(device))
            
        
            loss = criterion(y_pred.squeeze(), y_batch.to(device).squeeze())

            loss.backward(retain_graph=True)# I retain the graph because I get an error if I don't do it.
            optimizer.step() 
            batch+=1
    
    fit_end_time = time.time()
    print("Total time = %f" % (fit_end_time - fit_start_time))
    
</code></pre>
<p>I tried to do is to get rid any tensor inside of the GPU I don’t need. However it didn’t seem to improve.</p>
<p>I think that this problem is not really apparent in with really small training sets, but only comes when I train it with big amounts of training samples.</p>
<p>I would love to hear if this problem sounds like anything you have encounter before. If this is not enough let me know and I could elaborate more on my algorithm.</p>

ptrblck: <p>I tried to reproduce your issue locally, but it seems the performance scales alright:</p>
<pre><code class="lang-python">def train(model, data, target, epochs, criterion, optimizer):
    torch.cuda.synchronize()
    fit_start_time = time.time()

    for epoch in range(epochs):
        optimizer.zero_grad()
        output = model(data)        
        loss = criterion(output, target)
        loss.backward()
        optimizer.step() 
        
    torch.cuda.synchronize()
    fit_end_time = time.time()
    time_per_iter = ((fit_end_time - fit_start_time) / epochs) * 1000
    print('{:.3f}ms/iter'.format(time_per_iter))
    return time_per_iter

model = MLPflat(10, 10, 5, 64).cuda()
data = torch.randn(2000, 10).cuda()
target = torch.randn(2000, 10).cuda()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# warmup
train(model, data, target, 10, criterion, optimizer)
t0 = train(model, data, target, 100, criterion, optimizer)

data = torch.randn(20000, 10).cuda()
target = torch.randn(20000, 10).cuda()
train(model, data, target, 10, criterion, optimizer)
t1 = train(model, data, target, 100, criterion, optimizer)
</code></pre>
<p>The 2000 samples run takes approx. <code>6.671ms/iter</code>, while the 20000 samples take <code>35.873ms/iter</code>.<br>
What timings are you seeing?</p>

Volker_Siegel: <aside class="quote no-group" data-username="Adrian_Briceno_Aguil" data-post="1" data-topic="102468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adrian_briceno_aguil/48/12377_2.png" class="avatar"> Adrian_Briceno_Aguil:</div>
<blockquote>
<p>I have taken the time to see how much it actually takes because it so long</p>
</blockquote>
</aside>
<p>Could you tell the order of magnitude of the time until you gave up?<br>
Something like it took some minutes, some hours, or some days -  it may be obvious to many readers, but not for all.<br>
As a half-joking example of context dependance of it:<br>
A statistician and an astrophysicist have significantly different intuitition for<br>
“takes significantly way more”.</p>

Adrian_Briceno_Aguil: <p>Sorrry for the late reply! I thought I put notifications on. Yes you are right, I just ran it today for one 1 recursive depth(this trains 5 different networks of shapes (5 hidden layers,64 neurons) )This takes more than hour. Comparing it to the low resolution would be 12s.</p>
<p>I ran a test of the GPU Memory allocation and it seems that the problem is that I’m giving a lot of to the GPU and not erasing un necessary data.</p>
<p>You can see the spike of of memory allocation in this graph<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/8361d0a8e3c7a1cfd24ee3c7f87d05632bbaf129.png" alt="image_2020_11_19T16_53_01_374Z" data-base62-sha1="iKg7dGQr7gcTVDroDxx7MXdWt2p" width="494" height="385"></p>

Adrian_Briceno_Aguil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for the late reply thank you! It should scale quite good, however since is a recursive algorithm it seems that I’m something bad with the memory allocation of the GPU since it seems it is increasing with every iteration(of the algorithm).</p>
<p>I’m trying to erase stuff that is not necessary, but I can’t seem to find what is it exactly. I will try to print for each line of code I have something like this:</p>
<p><code>str(torch.cuda.memory_allocated(device)/100000 )</code></p>
<p>So I know what is the difference of memory and try to see where is exactly it is.</p>

Adrian_Briceno_Aguil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/volker_siegel">@Volker_Siegel</a> I just found the culprit. It seems the evaluating of this happens after I evaluate it:</p>
<pre><code class="lang-auto">phi.eval()
inTrainTodevice = inTrain.to(device) # GPU
print("Memory Allocated before Evaluating " + str(torch.cuda.memory_allocated(device)/100000 ) + " MGB")
yPredictThisLevel = phi(inTrainTodevice[:,:indim]) # GPU
print("Memory Allocated after Evaluating " + str(torch.cuda.memory_allocated(device)/100000 ) + " MGB")
</code></pre>
<p>The memory allocation jumps like this:</p>
<pre><code class="lang-auto">Memory Allocated after dividing of batches 0.43008 MGB
Memory Allocated after Training 1.58208 MGB
Memory Allocated before Evaluating 33.03936 MGB
Memory Allocated after Evaluating 6083.32288 MGB
</code></pre>
<p>But I don’t understand why this is the case, since is just a simple evaluation. Do you have any ideas why this would be the case?</p>

ptrblck: <p>I assume you don’t want to calculate the gradients during evaluation, so you could wrap the evaluation code into a <code>with torch.no_grad()</code>, which would avoid storing the intermediate tensors and would not attach the computation graph to the output tensors.<br>
Are you using a larger batch size or generally a larger input during evaluation, which could explain the memory increase?</p>

Adrian_Briceno_Aguil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks! yes yesterday evening made the this change and it help a lot. And now it the memory allocation stays stable.</p>
<p>But now I have another odd problem. I still don’t understand what exactly overloads the memory and what doesn’t. For example: I do a simple difference between two tensor and it allocates so much memory that it breaks. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23c23e002bd1ebab46964e94fee86434284b28c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/23c23e002bd1ebab46964e94fee86434284b28c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23c23e002bd1ebab46964e94fee86434284b28c1_2_690x121.png" alt="image" data-base62-sha1="56kS9uTTPvObYAIFrlAAAIfPjLX" width="690" height="121" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23c23e002bd1ebab46964e94fee86434284b28c1_2_690x121.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23c23e002bd1ebab46964e94fee86434284b28c1_2_1035x181.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23c23e002bd1ebab46964e94fee86434284b28c1.png 2x" data-dominant-color="F1F0F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1054×186 42.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check the shapes of <code>wlast</code> and <code>fun</code>? You might be broadcasting the tensors (which might not be what you want) and thus the memory could increase significantly.<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(10, 1)
y = torch.randn(1, 10)
diff = x - y
print(diff.shape)
&gt; torch.Size([10, 10])
</code></pre>

Adrian_Briceno_Aguil: <p>Ok it seems that I was doing something wrong: I will fix it to see if that helps.<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1944691a805d5f2b464b735ab31f83e3e51fa3a.png" alt="image" data-base62-sha1="tU1tarSrmu1EGGOAnuFdWah21iq" width="437" height="33"></p>
<p>What does it mean broadcasting the tensors? I apologize for my ignorance.</p>

ptrblck: <aside class="quote no-group" data-username="Adrian_Briceno_Aguil" data-post="12" data-topic="102468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adrian_briceno_aguil/48/12377_2.png" class="avatar"> Adrian_Briceno_Aguil:</div>
<blockquote>
<p>What does it mean broadcasting the tensors? I apologize for my ignorance.</p>
</blockquote>
</aside>
<p>PyTorch uses broadcasting to expand smaller dimensions in your tensors, so that they would match the expected tensor shapes for the current operation, if possible.<br>
The <a href="https://numpy.org/doc/stable/user/basics.broadcasting.html">numpy doc</a> explains it pretty well.</p>
<p>In your example I assume you would like to subtract the tensors elementwise.<br>
However, since the first tensor has an additional dimension, broadcasting will be applied as seen in this better example:</p>
<pre data-code-wrap="python"><code class="lang-python"># expected
x = torch.arange(10)
y = torch.arange(10)
diff = x - y
print(diff)
&gt; tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

# your code
x = torch.arange(10).view(10, 1)
y = torch.arange(10)
diff = x - y
print(diff)
&gt; tensor([[ 0, -1, -2, -3, -4, -5, -6, -7, -8, -9],
          [ 1,  0, -1, -2, -3, -4, -5, -6, -7, -8],
          [ 2,  1,  0, -1, -2, -3, -4, -5, -6, -7],
          [ 3,  2,  1,  0, -1, -2, -3, -4, -5, -6],
          [ 4,  3,  2,  1,  0, -1, -2, -3, -4, -5],
          [ 5,  4,  3,  2,  1,  0, -1, -2, -3, -4],
          [ 6,  5,  4,  3,  2,  1,  0, -1, -2, -3],
          [ 7,  6,  5,  4,  3,  2,  1,  0, -1, -2],
          [ 8,  7,  6,  5,  4,  3,  2,  1,  0, -1],
          [ 9,  8,  7,  6,  5,  4,  3,  2,  1,  0]])
</code></pre>
<p>As you can see, the second approach subtracts the <code>y</code> from each row in <code>x</code> and broadcasts the row value to match the dimensions.<br>
This would increase the memory usage to store the result and might not be what you want to use.</p>