Out of Memory issue with multi GPUs

antsthebul: <p>I am new to ML, Deep Learning, and Pytorch. I am not sure why, but changing my batch size and image size has no effect whatsoever on the allocated memory</p>
<blockquote>
<p>Tried to allocate 25.15 GiB (GPU 1; 47.54 GiB total capacity; 25.15 GiB already allocated; 21.61 GiB free; 25.16 GiB reserve</p>
</blockquote>
<p>I am using to A6000 x 2 GPUS. I made a gist of the code, but if prefered I can post it here. <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8" class="inline-onebox" rel="noopener nofollow ugc">Simple Distrbuted CNN · GitHub</a> . The goal is to create a simple CNN that can detect the illuminated light on traffic lights. I feel like the training data is not being split across the GPUs but all data is being trained on both GPUs simultaneuously, can anyone provide assistance? I am open to criticism if you have any tips on the layout or functionality of the code.</p>
<p>Also i feel like there could be a general rule of thumb (or at least a ‘general’ range) that an image size of (x,x) and CNN with  a certain amount of conv layers (or number of paramters) will require at least X amount of GB. What Ive noticed before is that I trained a Magic the Gathering Card classifier with CNN (distrbuted with the same code and it works)  and I can never seem to build over 3 layers before reaching out of memory following the 32/64 conv layer input/output format and image size 72x72 which i know is extremely small</p>
<p>Ive used the code from the following links:</p>
<ul>
<li>Learing how to use DDP -  <a href="https://pytorch.org/tutorials/beginner/ddp_series_intro.html?utm_source=youtube&amp;utm_medium=organic_social&amp;utm_campaign=tutorial" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel in PyTorch - Video Tutorials — PyTorch Tutorials 2.1.1+cu121 documentation</a></li>
<li>Optimizing memory/speed ( Towards Data Science: Optimize PyTorch Performance for Speed and Memory Efficiency (2022))</li>
</ul>

eqy: <p>I’m seeing a few things that seem unusual. Are you trying to train separate models or a single model? For the single model case I would expect that you would want to use DDP/<a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">Distributed Data Parallel</a> in conjunction with the distributed sampler.</p>
<p>For a reference implementation using the components I would recommend checking out the ImageNet example:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/examples/blob/main/imagenet/main.py">
  <header class="source">

      <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" target="_blank" rel="noopener nofollow ugc">pytorch/examples/blob/main/imagenet/main.py</a></h4>


      <pre><code class="lang-py">import argparse
import os
import random
import shutil
import time
import warnings
from enum import Enum

import torch
import torch.backends.cudnn as cudnn
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.nn.parallel
import torch.optim
import torch.utils.data
import torch.utils.data.distributed
import torchvision.datasets as datasets
import torchvision.models as models
import torchvision.transforms as transforms
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Additionally the model used looks highly unusual, in particular:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29">
  <header class="source">

      <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29</a></h4>

  <h5>gpu_multi.py</h5>
  <pre><code class="Python">import csv
import datetime
import os
import time

import pandas as pd
import albumentations as A
from pathlib import Path

from dotenv import load_dotenv</code></pre>
    This file has been truncated. <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">show original</a>
  <h5>light_dataset.py</h5>
  <pre><code class="Python">import cv2
from torch.utils.data.dataset import Dataset
from config import *


class LightDataset(Dataset):
  def __init__(self, image_data, labels, transform=None):
    self.image_paths = image_data.ravel() 
    self.y_labels = labels.ravel()
    self.transform = transform</code></pre>
    This file has been truncated. <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">show original</a>
  <h5>light_model.py</h5>
  <pre><code class="Python">import torch
import torch.nn as nn 
from config import *


def conv_block(in_channel, out_channel, kernel=5):
    return nn.Sequential(
        nn.Conv2d(in_channels=in_channel, out_channels=out_channel, kernel_size=kernel, padding="same", bias=False),
        nn.ReLU()  , # 178, 128, CONV_1_OUTPUT (32)     
        nn.MaxPool2d(kernel_size=2, stride=1), # 177, 127 , 32</code></pre>
    This file has been truncated. <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
    There are more than three files. <a href="https://gist.github.com/Antsthebul/f1d970cd19a965ac58d39572c9f370d8#file-light_model-py-L29" target="_blank" rel="noopener nofollow ugc">show original</a>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
which is an unusually large linear layer for a classification model.</p>
<p>You may want to take a look at the implementations for some standard torchvision models <a href="https://pytorch.org/vision/stable/models.html#classification" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.15 documentation</a> to get a feel for typical CNN classification architectures.</p>

antsthebul: <p>Im trying to train a single model. I was trying to build a model resembling <a href="https://github.com/rasbt/machine-learning-book/blob/main/ch14/ch14_part2.ipynb" class="inline-onebox" rel="noopener nofollow ugc">machine-learning-book/ch14_part2.ipynb at main · rasbt/machine-learning-book · GitHub</a> which uses a 4 convolution layers to train a classifier. I see in the example that the linear layer is not as large as the model I created. I switched to using batchnorm instead of dropout, but Ill see why the linear layer is so large and go from there. Thanks for taking a look.</p>

antsthebul: <p>Thanks <a class="mention" href="/u/eqy">@eqy</a> my FC layer was waayy too large as mentioned. I just reduced the size to some arbitrary low and I was able to get the model running again.</p>
<p>The stride paramter I had set in the MaxPool was creating an absurd amoutn of paramters. I will look more into how to use this layer correctly</p>