Saving Numpy Arrays in the Most Compressed Way

username: MUnique
cooked: <p>Hello everyone,</p>
<p>I have some large NumPy arrays (4000 , 200 , 200 , 20). I was looking for the best way to save them in the most compressed way. I’ve tried .npz format saving but it takes about 200MB for each. These Numpy arrays would be the inputs to my CNN and due to the large size of the arrays, I am struggling with “Cuda out of memory” problem.</p>

username: ptrblck
cooked: <p>The “CUDA out of memory” issue is raised, if your GPU is running our of memory and is unrelated to the storage of numpy arrays.<br>
Try to reduce the batch size for your input tensors to reduce the memory usage or alternatively use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

username: MUnique
cooked: <p>The batch size is set to 1. A large part of it happens when I call my network (Model), before even getting to the forward pass:<br>
In the debugging window:<br>
y = Model(x)<br>
-&gt; Class Model(Module):<br>
def <strong>init</strong>(self):<br>
# some initializations<br>
def forward (self, input):<br>
# A large part of memory have been used before the forward pass. Here<br>
tensor = torch.max(input, dim=2, keepdim=False)</p>

username: ptrblck
cooked: <p>If the OOM error is raised before the forward pass is executed, this would mean that either your model or the current batch is already too large for your GPU.<br>
Are you pushing the complete dataset to the device?</p>

username: MUnique
cooked: <p>No, every epoch only one of these arrays (4000,200,200,20) is fed to the network. There are 200 of these arrays in total in the datatset. The network itself is not a large one. It has 10 convolutional layers in total.</p>

username: ptrblck
cooked: <p>I don’t quite understand the use case, as you’ve mentioned the batch size is 1.<br>
Which dimension of your input would correspond to the batch size and which shape are you feeding to the model?<br>
If the mentioned shape is fed in each epoch to the model, when does the OOM error occur?</p>

username: MUnique
cooked: <p>The dimension of the input is (1, 4000, 200, 200, 20) in which the first one is the batch size. Here is my watchdog window for each command during debugging:<br>
y = Model(x)      ######## Watchdog GPU Memory Usage: 1085MiB<br>
-&gt; Class Model(Module):<br>
def  <strong>init</strong> (self):<br>
…<br>
def forward (self, input):<br>
tensor = input  ######## Watchdog: 8279MiB</p>
<p>Seems that OOM happens when the GPU memory usage is 8469MiB in a layer between but the largest part of the memory is used before getting into the forward pass which I guess makes the problem.</p>

username: ptrblck
cooked: <p>Since you’ve mentioned you are using a CNN, I assume you are using <code>nn.Conv3d</code> layers and the input has a channel dimension of 4000?<br>
Could you post your model architecture and post your GPU (including the memory)?</p>

username: MUnique
cooked: <p>The input is actually like a video in which 4000 is the number of frames, spatial window size is 200 for both horizontal and vertical and 20 is the number of channels. I am using two Tesla V100 PCIe 16GB GPUs.<br>
The model architeture is a stack of two of these layers:<br>
1DTemporalConv-1DChannelconv-2DSpatialConv-2DSpatialMaxPooling</p>
<p>I also tried to test the network without training and the same problem happens during testing.</p>

username: MUnique
cooked: <p>Is it possible that the size of the input array in each epoch (200MB) cause the OOM problem? Seems that Variable(input.cuda()) causes the problem before getting into the forward pass.</p>

username: ptrblck
cooked: <p>We would still need to see the model to debug it further.<br>
Could you post it with all necessary arguments and shapes to reproduce the OOM?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="MUnique" data-post="11" data-topic="86441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/d78d45/48.png" class="avatar"> MUnique:</div>
<blockquote>
<p>Is it possible that the size of the input array in each epoch (200MB) cause the OOM problem?</p>
</blockquote>
</aside>
<p><code>4000*200*200*20*16bits</code> ~= 6GB, so no mysteries there. 200mb is zipped .npz size, you can’t train a network on that representation.</p>