RuntimeError: CUDA out of memory. Tried to allocate 62.50 MiB (GPU 0; 3.81 GiB total capacity; 2.53 GiB already allocated; 30.50 MiB free; 4.14 MiB cached)

shakti_bangare: <p>Traceback (most recent call last):<br>
File “tools/train.py”, line 96, in <br>
main()<br>
File “tools/train.py”, line 92, in main<br>
logger=logger)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/apis/train.py”, line 59, in train_detector<br>
_non_dist_train(model, dataset, cfg, validate=validate)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/apis/train.py”, line 121, in _non_dist_train<br>
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmcv/runner/runner.py”, line 349, in run<br>
epoch_runner(data_loaders[i], **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmcv/runner/runner.py”, line 255, in train<br>
self.model, data_batch, train_mode=True, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/apis/train.py”, line 37, in batch_processor<br>
losses = model(**data)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 141, in forward<br>
return self.module(*inputs[0], **kwargs[0])<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/models/detectors/base.py”, line 80, in forward<br>
return self.forward_train(img, img_meta, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/models/detectors/two_stage.py”, line 88, in forward_train<br>
x = self.extract_feat(img)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/models/detectors/two_stage.py”, line 75, in extract_feat<br>
x = self.backbone(img)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/models/backbones/hrnet.py”, line 450, in forward<br>
x = self.layer1(x)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 92, in forward<br>
input = module(input)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/mmdet-0.6rc0+5da6db5-py3.6.egg/mmdet/models/backbones/hrnet.py”, line 99, in forward<br>
out = self.bn1(out)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py”, line 76, in forward<br>
exponential_average_factor, self.eps)<br>
File “/home/hil/anaconda3/envs/hrnet/lib/python3.6/site-packages/torch/nn/functional.py”, line 1623, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled</p>

ptrblck: <p>You are running out of memory and need to reduce the memory usage e.g. via lowering the batch size of your inputs.</p>

shakti_bangare: <p>Hi , Thanks<br>
I have single NVIDIA T600 gpu of 4GB memory.<br>
I am using PE_faster_rcnn_hrnetv2p_w18_1x.py as a custom config file and running the following command:  python tools/train.py configs/hrnet/PE_faster_rcnn_hrnetv2p_w18_1x.py</p>
<p>PE_faster_rcnn_hrnetv2p_w18_1x.py file has following content:</p>
<h1><a name="model-settings-1" class="anchor" href="#model-settings-1"></a>model settings</h1>
<p>model = dict(<br>
type=‘FasterRCNN’,<br>
pretrained=‘hrnetv2_pretrained/hrnetv2_w18_imagenet_pretrained.pth’,<br>
backbone=dict(<br>
type=‘HighResolutionNet’,<br>
extra=dict(<br>
stage1=dict(<br>
num_modules=1,<br>
num_branches=1,<br>
block=‘BOTTLENECK’,<br>
num_blocks=(4,),<br>
num_channels=(64,),<br>
fuse_method=‘SUM’),<br>
stage2=dict(<br>
num_modules=1,<br>
num_branches=2,<br>
block=‘BASIC’,<br>
num_blocks=(4, 4),<br>
num_channels=(18, 36),<br>
fuse_method=‘SUM’),<br>
stage3=dict(<br>
num_modules=4,<br>
num_branches=3,<br>
block=‘BASIC’,<br>
num_blocks=(4, 4, 4),<br>
num_channels=(18, 36, 72),<br>
fuse_method=‘SUM’),<br>
stage4=dict(<br>
num_modules=3,<br>
num_branches=4,<br>
block=‘BASIC’,<br>
num_blocks=(4, 4, 4, 4),<br>
num_channels=(18, 36, 72, 144),<br>
fuse_method=‘SUM’))),<br>
neck=dict(<br>
type=‘HRFPN’,<br>
in_channels=[18, 36, 72, 144],<br>
out_channels=256),<br>
rpn_head=dict(<br>
type=‘RPNHead’,<br>
in_channels=256,<br>
feat_channels=256,<br>
anchor_scales=[8],<br>
anchor_ratios=[0.5, 1.0, 2.0],<br>
anchor_strides=[4, 8, 16, 32, 64],<br>
target_means=[.0, .0, .0, .0],<br>
target_stds=[1.0, 1.0, 1.0, 1.0],<br>
use_sigmoid_cls=True),<br>
bbox_roi_extractor=dict(<br>
type=‘SingleRoIExtractor’,<br>
roi_layer=dict(type=‘RoIAlign’, out_size=7, sample_num=2),<br>
out_channels=256,<br>
featmap_strides=[4, 8, 16, 32]),<br>
bbox_head=dict(<br>
type=‘SharedFCBBoxHead’,<br>
num_fcs=2,<br>
in_channels=256,<br>
fc_out_channels=1024,<br>
roi_feat_size=7,<br>
num_classes=81,<br>
target_means=[0., 0., 0., 0.],<br>
target_stds=[0.1, 0.1, 0.2, 0.2],<br>
reg_class_agnostic=False))</p>
<h1><a name="model-training-and-testing-settings-2" class="anchor" href="#model-training-and-testing-settings-2"></a>model training and testing settings</h1>
<p>train_cfg = dict(<br>
rpn=dict(<br>
assigner=dict(<br>
type=‘MaxIoUAssigner’,<br>
pos_iou_thr=0.7,<br>
neg_iou_thr=0.3,<br>
min_pos_iou=0.3,<br>
ignore_iof_thr=-1),<br>
sampler=dict(<br>
type=‘RandomSampler’,<br>
num=256,<br>
pos_fraction=0.5,<br>
neg_pos_ub=-1,<br>
add_gt_as_proposals=False),<br>
allowed_border=0,<br>
pos_weight=-1,<br>
smoothl1_beta=1 / 9.0,<br>
debug=False),<br>
rcnn=dict(<br>
assigner=dict(<br>
type=‘MaxIoUAssigner’,<br>
pos_iou_thr=0.5,<br>
neg_iou_thr=0.5,<br>
min_pos_iou=0.5,<br>
ignore_iof_thr=-1),<br>
sampler=dict(<br>
type=‘RandomSampler’,<br>
num=512,<br>
pos_fraction=0.25,<br>
neg_pos_ub=-1,<br>
add_gt_as_proposals=True),<br>
pos_weight=-1,<br>
debug=False))<br>
test_cfg = dict(<br>
rpn=dict(<br>
nms_across_levels=False,<br>
nms_pre=2000,<br>
nms_post=2000,<br>
max_num=2000,<br>
nms_thr=0.7,<br>
min_bbox_size=0),<br>
rcnn=dict(<br>
score_thr=0.05, nms=dict(type=‘nms’, iou_thr=0.5), max_per_img=1)<br>
# soft-nms is also supported for rcnn testing<br>
# e.g., nms=dict(type=‘soft_nms’, iou_thr=0.5, min_score=0.05)<br>
)</p>
<h1><a name="note-3" class="anchor" href="#note-3"></a>NOTE:</h1>
<h1><a name="dataset-settings-4" class="anchor" href="#dataset-settings-4"></a>dataset settings</h1>
<h1><a name="if-you-use-zip-format-to-store-all-images-of-coco-please-use-cocozipdataset-5" class="anchor" href="#if-you-use-zip-format-to-store-all-images-of-coco-please-use-cocozipdataset-5"></a>if you use zip format to store all images of coco, please use CocoZipDataset</h1>
<p>dataset_type = ‘PEDataset’<br>
data_root = ‘…/images/’<br>
img_norm_cfg = dict(<br>
mean=[127.5], std=[127.5], to_rgb=False)</p>
<h1><a name="else-6" class="anchor" href="#else-6"></a>else</h1>
<h1><a name="dataset_type-cocodataset-7" class="anchor" href="#dataset_type-cocodataset-7"></a>dataset_type = ‘CocoDataset’</h1>
<h1><a name="data_root-datacoco-8" class="anchor" href="#data_root-datacoco-8"></a>data_root = ‘data/coco/’</h1>
<h1><a name="img_norm_cfg-dictmean123675-11628-10353-std58395-5712-57375-to_rgbtrue-9" class="anchor" href="#img_norm_cfg-dictmean123675-11628-10353-std58395-5712-57375-to_rgbtrue-9"></a>img_norm_cfg = dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)</h1>
<p>data = dict(<br>
imgs_per_gpu=4,<br>
workers_per_gpu=0,<br>
train=dict(<br>
type=dataset_type,<br>
ann_file=data_root + ‘bbox.json’,<br>
img_prefix=data_root + ‘train/’,<br>
img_scale=(1333, 800),<br>
img_norm_cfg=img_norm_cfg,<br>
size_divisor=32,<br>
flip_ratio=0,<br>
with_mask=False,<br>
with_crowd=False,<br>
with_label=True),<br>
val=dict(<br>
type=dataset_type,<br>
ann_file=data_root + ‘bbox.json’,<br>
img_prefix=data_root + ‘train/’,<br>
img_scale=(1333, 800),<br>
img_norm_cfg=img_norm_cfg,<br>
size_divisor=32,<br>
flip_ratio=0,<br>
with_mask=False,<br>
with_crowd=False,<br>
with_label=True),<br>
test=dict(<br>
type=dataset_type,<br>
ann_file=data_root + ‘test_names.json’,<br>
img_prefix=data_root + ‘test/’,<br>
img_scale=(1333, 800),<br>
img_norm_cfg=img_norm_cfg,<br>
size_divisor=32,<br>
flip_ratio=0,<br>
with_mask=False,<br>
with_label=False,<br>
test_mode=True))</p>
<h1><a name="optimizer-10" class="anchor" href="#optimizer-10"></a>optimizer</h1>
<h1><a name="if-you-use-8-gpus-for-training-please-change-lr-to-002-11" class="anchor" href="#if-you-use-8-gpus-for-training-please-change-lr-to-002-11"></a>if you use 8 GPUs for training, please change lr to 0.02</h1>
<p>optimizer = dict(type=‘SGD’, lr=0.001, momentum=0.9, weight_decay=0.0001)<br>
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))</p>
<h1><a name="learning-policy-12" class="anchor" href="#learning-policy-12"></a>learning policy</h1>
<p>lr_config = dict(<br>
policy=‘step’,<br>
warmup=‘linear’,<br>
warmup_iters=500,<br>
warmup_ratio=1.0 / 3,<br>
step=[8, 11])<br>
checkpoint_config = dict(interval=1)</p>
<h1><a name="yapfdisable-13" class="anchor" href="#yapfdisable-13"></a>yapf:disable</h1>
<p>log_config = dict(<br>
interval=50,<br>
hooks=[<br>
dict(type=‘TextLoggerHook’),<br>
dict(type=‘TensorboardLoggerHook’)<br>
])</p>
<h1><a name="yapfenable-14" class="anchor" href="#yapfenable-14"></a>yapf:enable</h1>
<h1><a name="runtime-settings-15" class="anchor" href="#runtime-settings-15"></a>runtime settings</h1>
<p>total_epochs = 1<br>
dist_params = dict(backend=‘nccl’)<br>
log_level = ‘INFO’<br>
work_dir = ‘./work_dirs/faster_rcnn_hrnetv2p_w18_1x/’<br>
load_from = None<br>
resume_from = None<br>
workflow = [(‘train’, 1)]</p>