Dataset size and limited shared memory

Crispolo: <p>I’m trying to train a network on Colab, but I have a problem of memory.</p>
<p>The training cannot start because I obtain the following message:<br>
<code> RuntimeError: DataLoader worker (pid 12945) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.</code></p>
<p>I’m new to PyTorch and Colab and I’m not sure the problem is really the size of the data or maybe something else in the code.</p>
<p>I use a dataset of 47721 images, about 3.25 GB.</p>
<p>I create three dataloader:</p>
<ul>
<li>training 60%</li>
<li>validation 20%</li>
<li>test 20%</li>
</ul>
<p>For training I use minbatch of size 32.</p>
<p>I use the free version of Colab, which has about 12 GB of RAM. When I start the runtime about 5 GB are already occupied, but 7 are free.</p>
<p>As model I use a pretrained GoogLeNet.</p>
<p>I’m not sure if maybe I’m doing something wrong when I create the dataloader, here below you can find the code:</p>
<pre><code class="lang-auto">def getDataLoader(dataset, batchSize=BATCH_SIZE, shuffle=False, dropLast=False):
  print('Splitting dataset into train and validation datasets...')
  trainDs, validDs = randomSplitDataset(dataset)

  validDataLoader = DataLoader(validDs,
                               batch_size=(len(dataset)
                               if batchSize is None
                               else batchSize),
                               shuffle=shuffle,
                               num_workers=WORKERS,
                               drop_last=dropLast)
  
  trainDataLoader = DataLoader(trainDs,
                               batch_size=(len(dataset)
                               if batchSize is None
                               else batchSize),
                               shuffle=shuffle,
                               num_workers=WORKERS,
                               drop_last=dropLast)

  return trainDataLoader, validDataLoader
</code></pre>
<p>Please let me know if there is something else I can share, in order to understand if the problem is my code or really the size of the data.</p>

ptrblck: <p>Try to decrease the <code>num_workers</code> as each of them would use shared memory which might be causing this issue.<br>
If this doesn’t help, the crash might be unrelated to shared memory usage.</p>

Crispolo: <p>I tried with <code>num_workers</code> equal 0, but nothing change.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="171135">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>the crash might be unrelated to shared memory usage</p>
</blockquote>
</aside>
<p>What do you mean?<br>
What could be the cause or how can I identify it?</p>

Crispolo: <p>I found the problem!</p>
<p>I don’t remember why, but at the beginning of every training step I did the following:</p>
<pre><code class="lang-auto">def train(model, source_loader, target_loader,

    print("Training Started")
    model.train()
    results = [] # append loss values at each epoch

    source = list(enumerate(source_loader))
    target = list(enumerate(target_loader))
    train_steps = min(len(source), len(target))

    # start batch training
    for step in tnrange(train_steps):
        _, (source_data, source_label) = source[step]
        _, (target_data, _) = target[step] # unsupervised learning
        if cuda:
            # move to device
            source_data = source_data.cuda()
            source_label = source_label.cuda()
            target_data = target_data.cuda()

        ...
</code></pre>
<p>These two operations filled up the memory.</p>
<pre><code class="lang-auto">    source = list(enumerate(source_loader))
    target = list(enumerate(target_loader))
</code></pre>
<p>I rewrote the code in this way</p>
<pre><code class="lang-auto">def train(model, trainDataloader, optimizer, epoch, cuda=False):
  print('Training started')
  model.train()
  results = [] # append loss values at each epoch

  step = 0

  # start batch training
  # go over dataloader batches, labels
  for sourceData, sourceLabel in trainDataloader:
    if cuda:
        # move to device
        sourceData = sourceData.cuda()
        sourceLabel = sourceLabel.cuda()

    ...
</code></pre>
<p>Now everything works as it should.</p>