So much virtual memory used during the training

Ardeal: <p>Hi,</p>
<p>I am training yolov3 code from: <a href="https://github.com/ultralytics/yolov3" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ultralytics/yolov3: YOLOv3 in PyTorch &gt; ONNX &gt; CoreML &gt; TFLite</a>,<br>
the code could be correctly trained.  My computer configuration is:<br>
CPU i9-9900k with 32GB RAM<br>
GPU 2080Ti with 11GB memory</p>
<p>the following message is printed during the training:</p>
<pre data-code-wrap="python"><code class="lang-python">
Using torch 1.7.1+cu101 CUDA:0 (GeForce RTX 2080 Ti, 11264.0MB)
Namespace(adam=False, batch_size=8, bucket='', cache_images=False, cfg='', data='data/coco128.yaml', device='', 
epochs=300, evolve=False, exist_ok=False, global_rank=-1, hyp='data/hyp.scratch.yaml', image_weights=False, 
img_size=[640, 640], local_rank=-1, log_imgs=16, multi_scale=False, name='exp', noautoanchor=False, nosave=False, 
notest=False, project='runs/train', rect=False, resume=False, save_dir='runs\\train\\exp', single_cls=False, sync_bn=False, 
total_batch_size=8, weights='yolov3.pt', workers=8, world_size=1)
Start Tensorboard with "tensorboard --logdir runs/train", view at http://localhost:6006/
Hyperparameters {'lr0': 0.01, 'lrf': 0.2, 'momentum': 0.937, 'weight_decay': 0.0005, 'warmup_epochs': 3.0, 
'warmup_momentum': 0.8, 'warmup_bias_lr': 0.1, 'box': 0.05, 'cls': 0.5, 'cls_pw': 1.0, 'obj': 1.0, 'obj_pw': 1.0, 
'iou_t': 0.2, 'anchor_t': 4.0, 'fl_gamma': 0.0, 'hsv_h': 0.015, 'hsv_s': 0.7, 'hsv_v': 0.4, 'degrees': 0.0, 'translate': 0.1, 
'scale': 0.5, 'shear': 0.0, 'perspective': 0.0, 'flipud': 0.0, 'fliplr': 0.5, 'mosaic': 1.0, 'mixup': 0.0}

......

Model Summary: 333 layers, 61949149 parameters, 61949149 gradients, 156.4 GFLOPS
Transferred 440/440 items from yolov3.pt
Optimizer groups: 75 .bias, 75 conv.weight, 72 other
Scanning 'C:\doc\dataset\COCO\labels\train2014.cache' for images and labels... 122218 found, 0 missing, 0 empty, 0 corrupted: 100%|██████████| 122218/122218 [00:00&lt;?, ?it/s]
Scanning 'C:\doc\dataset\COCO\labels\train2014' for images and labels... 42219 found, 0 missing, 0 empty, 0 corrupted: 100%|██████████| 42219/42219 [01:12&lt;00:00, 582.44it/s]
New cache created: C:\doc\dataset\COCO\labels\train2014.cache
Scanning 'C:\doc\dataset\COCO\labels\train2014.cache' for images and labels... 42219 found, 0 missing, 0 empty, 0 corrupted: 100%|██████████| 42219/42219 [00:00&lt;?, ?it/s]
Analyzing anchors... anchors/target = 4.41, Best Possible Recall (BPR) = 0.9943
Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
NumExpr defaulting to 8 threads.
Image sizes 640 train, 640 test
Using 8 dataloader workers
Logging results to runs\train\exp
Starting training for 300 epochs...
     Epoch   gpu_mem       box       obj       cls     total   targets  img_size
	0/299     7.32G   0.03043   0.02528  0.009495   0.06521        83       640:  26%|██▌       | 3982/15278 [18:47&lt;49:17,  3.82it/s]


</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb94c79d48cfb5d7f3327ddd0b1cc54b2af29a50.png" data-download-href="https://discuss.pytorch.org/uploads/default/cb94c79d48cfb5d7f3327ddd0b1cc54b2af29a50" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb94c79d48cfb5d7f3327ddd0b1cc54b2af29a50.png" alt="image" data-base62-sha1="t2XGvzirTX2fWyWqNcrMXWyfnfG" width="616" height="500" data-dominant-color="F7F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">869×705 24.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>during the training, I capture the RAM usage.<br>
There is 32GB RAM and 17.8GB available. virtual memory assigned is 95.8GB and used is 79.5GB.</p>
<p>According to my understanding, if physical memory is not sufficient, CPU will apply virtual memory. in my case, there are 17.8GB physical memory available, why is so much virtual memory used? how is it used?</p>