Building simple, pre-tested NN but running into memory problems

Edwin_Meyers: <p>Thank you very much in advance for your help.</p>
<p>I want to train an image classifying NN but am running into the following memory Error:</p>
<p><code>RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 123681636352 bytes. Error code 12 (Cannot allocate memory)</code></p>
<p>I found this error strange because I am running on an ml.t2.xlarge SageMaker instance (lots of memory). Here’s a bit of context on the code:</p>
<p>I wrote a python script to:<br>
i) load a file list (csv file) into the dataloader<br>
ii) convert input images from TIFF to Numpy to Torch<br>
iii) permute channels (H, W, C vs C, H, W)<br>
iv) crop picture into smaller picture to avoid memory problems (dataset has 380GB of images, each .TIFF images is about ~25MB)</p>
<p>Here is the python script:</p>
<pre><code class="lang-auto">class load_csv(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
    
    def __len__(self):
        return len(self.annotations)
    
    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        image = torch.from_numpy(tiff.imread(img_path)).permute(2,0,1).float()
        #Image.MAX_IMAGE_PIXELS = None
                
       image.transform = transforms.RandomResizedCrop(224)
        
        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))
        
        #if self.transform:
        #    image = self.transform(image)
        
        return (image, y_label)
</code></pre>
<p>Now, I’m running into a memory-related problem that I don’t know how to resolve. I am running a ml.t2.xlarge SageMaker notebook with plenty of memory and images being loaded into the dataloader have supposedly undergone a “transforms.RandomResizedCrop(224)” transformation and should be smaller.</p>
<p>Why am I running into this problem?</p>
<p>Please find project repo with CNN.ipynb and python script csv_loader.py where <strong>getitem</strong> class is defined for dataloader (<a href="https://github.com/EddieFroufou/Capstone-Project-Pancreatic-Cancer-Detector" rel="nofollow noopener">here</a>).</p>
<p>Thank you very much again,<br>
Edwin</p>

Edwin_Meyers: <p>FOLLOW UP:</p>
<p>I added <em>num_workers=3</em> to the dataloaders hoping it would help.</p>
<p>I still memory get a memory related error. I ran this on a fresh kernel with no operations on it. Here is the error I get. :</p>
<pre><code class="lang-auto">RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 257, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/home/ec2-user/SageMaker/csv_loader.py", line 21, in __getitem__
    image = torch.from_numpy(tiff.imread(img_path)).permute(2,0,1).float()
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 10856909952 bytes. Error code 12 (Cannot allocate memory)
</code></pre>

Edwin_Meyers: <p>[SOLUTION]</p>
<p>The image was a multi-level TIFF file. If the file level is not specified, the first level is taken by default. In this case, each first-level image was between 25 to 50MB, sometimes more. The Dataloader batch size needed to be reduced to 1 in order not to generate errors. In the end, I chose 2nd level images to train and test the model.</p>
<p>Blessings</p>