Memory allocation error when I have enough memory!

username: adamIsStuck
cooked: <p>Hi there I get the following error when trying to train my model…</p>
<p>RuntimeError: [enforce fail at CPUAllocator.cpp:68] . DefaultCPUAllocator: can’t allocate memory: you tried to allocate 839818240 bytes. Error code 12 (Cannot allocate memory)</p>
<p>This error is seen quite a lot on internet forums however I do not believe the memory it is trying to allocate is too much in this case as 839818240 bytes = 0.8 GB roughly…</p>
<p>Running the model on Google Cloud Pytorch VM…</p>
<p>Here is the full error message…</p>
<p>Thanks the help!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/695822d2dcff87b96e9bad070f4615c36ad2178e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/695822d2dcff87b96e9bad070f4615c36ad2178e" title="Screenshot 2022-02-22 at 13.50.39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695822d2dcff87b96e9bad070f4615c36ad2178e_2_690x414.jpeg" alt="Screenshot 2022-02-22 at 13.50.39" data-base62-sha1="f1UXPlvCXTdXltS6XOQs5Z2UOrY" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695822d2dcff87b96e9bad070f4615c36ad2178e_2_690x414.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695822d2dcff87b96e9bad070f4615c36ad2178e_2_1035x621.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695822d2dcff87b96e9bad070f4615c36ad2178e_2_1380x828.jpeg 2x" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-02-22 at 13.50.39</span><span class="informations">1920×1153 230 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The failing call tries to allocate ~0.78GB, but note that the script (or other processes) might have already allocated memory on the host.<br>
Check the host RAM usage during the lifetime of your script and check its peak usage before the OOM is raised.</p>

username: adamIsStuck
cooked: <p>Hi there,</p>
<p>thankyou for the reply! I have printed RAM usage at each line of my script and the output has been attached below. What is confusing me here is that it does not use anymore than 20% of the max RAM at the time when the error occurs! The error seems to raise when I run the line ‘trainer.fit(model)’ where trainer is an instance of Trainer() class in the pytorch_lightning library.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45ffb6b38c64ed7801e33caa618cce821f9bf6a3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/45ffb6b38c64ed7801e33caa618cce821f9bf6a3" title="Screenshot 2022-02-23 at 14.23.06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45ffb6b38c64ed7801e33caa618cce821f9bf6a3_2_690x391.jpeg" alt="Screenshot 2022-02-23 at 14.23.06" data-base62-sha1="9ZeObFzr4HfOEfcbvt8csHVtEbh" width="690" height="391" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45ffb6b38c64ed7801e33caa618cce821f9bf6a3_2_690x391.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45ffb6b38c64ed7801e33caa618cce821f9bf6a3_2_1035x586.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45ffb6b38c64ed7801e33caa618cce821f9bf6a3_2_1380x782.jpeg 2x" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-02-23 at 14.23.06</span><span class="informations">1920×1090 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: adamIsStuck
cooked: <p>I should also mention that ‘RAM memory (run)’ is the RAM used from running the running script and ‘RAM memory’ without the ‘(run)’ is the RAM usage from the script that loads the data we are training on.</p>

username: adamIsStuck
cooked: <p>I am a bit confused about the fact that the built-in ‘Trainer()’ class is throwing the error. Why might this be? Does this mean I will struggle to solve the problem as we cannot change this script?</p>

username: ptrblck
cooked: <p>I guess the <code>Trainer</code> class tries to allocate more memory internally and is then running into the issue.<br>
You might thus want to add more debugging print statements to the actual model definition (and/or <code>Dataset</code>) to see where this memory is being allocated (I don’t know what the <code>Trainer</code> class is doing in detail).</p>