Running PHOSA from Facebook FAIR on a GPU with 8G RAM, RuntimeError: CUDA out of memory. Tried to allocate 750.00 MiB (GPU 0; 7.80 GiB total capacity; 5.64 GiB already allocated; 380.44 MiB free; 6.30 GiB reserved in total by PyTorch)

Mona_Jalal: <p>I am running <a href="https://github.com/facebookresearch/phosa" rel="noopener nofollow ugc">PHOSA git repo</a> on a GeFrce 2070 GPU however, I get CUDA out of memory error. Could you please guide how I should fix this?</p>
<pre><code class="lang-auto">(base) mona@mona:~/research/phosa$ python demo.py --filename input/000000038829.jpg

Bad key savefig.frameon in file /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 421 ('savefig.frameon : True')
You probably need to get an updated matplotlibrc file from
https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template
or from the matplotlib source distribution

Bad key verbose.level in file /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 472 ('verbose.level  : silent      # one of silent, helpful, debug, debug-annoying')
You probably need to get an updated matplotlibrc file from
https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template
or from the matplotlib source distribution

Bad key verbose.fileo in file /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 473 ('verbose.fileo  : sys.stdout  # a log filename, sys.stdout or sys.stderr')
You probably need to get an updated matplotlibrc file from
https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template
or from the matplotlib source distribution
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In /home/mona/anaconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: 
The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
2020-12-06 20:56:58,748 INFO     Calling with args: Namespace(class_name='bicycle', filename='input/000000038829.jpg', lw_collision=None, lw_depth=None, lw_inter=None, lw_inter_part=None, lw_scale=None, lw_scale_person=None, lw_sil=None, mesh_index=0, output_dir='output')
2020-12-06 20:57:05,605 INFO     Loading checkpoint from detectron2://PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2020-12-06 20:57:05,607 INFO     URL https://dl.fbaipublicfiles.com/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl cached in /home/mona/.torch/fvcore_cache/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2020-12-06 20:57:08,837 INFO     Reading a file from 'Detectron2 Model Zoo'
2020-12-06 20:57:08,888 INFO     Some model parameters or buffers are not found in the checkpoint:
  roi_heads.mask_head.point_head.fc1.{weight, bias}
  roi_heads.mask_head.coarse_head.reduce_spatial_dim_conv.{weight, bias}
  roi_heads.mask_head.coarse_head.prediction.{weight, bias}
  roi_heads.mask_head.point_head.fc2.{bias, weight}
  roi_heads.mask_head.point_head.fc3.{weight, bias}
  roi_heads.mask_head.coarse_head.coarse_mask_fc2.{weight, bias}
  roi_heads.mask_head.point_head.predictor.{weight, bias}
  roi_heads.mask_head.coarse_head.coarse_mask_fc1.{weight, bias}
2020-12-06 20:57:08,889 INFO     The checkpoint state_dict contains keys that are not used by the model:
  roi_heads.mask_coarse_head.reduce_spatial_dim_conv.{weight, bias}
  roi_heads.mask_coarse_head.coarse_mask_fc1.{weight, bias}
  roi_heads.mask_coarse_head.coarse_mask_fc2.{weight, bias}
  roi_heads.mask_coarse_head.prediction.{weight, bias}
  roi_heads.mask_point_head.fc1.{weight, bias}
  roi_heads.mask_point_head.fc2.{weight, bias}
  roi_heads.mask_point_head.fc3.{weight, bias}
  roi_heads.mask_point_head.predictor.{weight, bias}
/home/mona/anaconda3/lib/python3.7/site-packages/detectron2/modeling/roi_heads/fast_rcnn.py:124: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629403081/work/torch/csrc/utils/python_arg_parser.cpp:766.)
  filter_inds = filter_mask.nonzero()
WARNING: You are using a SMPL model, with only 10 shape coefficients.
loss: 1.05e+04:   0%|▏                        | 1/200.0 [00:00&lt;03:00,  1.10it/s]Traceback (most recent call last):
  File "demo.py", line 145, in &lt;module&gt;
    main(get_args())
  File "demo.py", line 121, in main
    instances=instances, class_name=args.class_name, mesh_index=args.mesh_index
  File "/home/mona/research/phosa/phosa/pose_optimization.py", line 405, in find_optimal_poses
    num_initializations=num_initializations,
  File "/home/mona/research/phosa/phosa/pose_optimization.py", line 303, in find_optimal_pose
    loss_dict, sil = model()
  File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/phosa/phosa/pose_optimization.py", line 186, in forward
    image = self.keep_mask * self.renderer(verts, self.faces, mode="silhouettes")
  File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/renderer.py", line 76, in forward
    return self.render_silhouettes(vertices, faces, K, R, t, dist_coeffs, orig_size)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/renderer.py", line 151, in render_silhouettes
    images = nr.rasterize_silhouettes(faces, self.image_size, self.anti_aliasing)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/rasterize.py", line 434, in rasterize_silhouettes
    return rasterize_rgbad(faces, None, image_size, anti_aliasing, near, far, eps, None, False, True, False)['alpha']
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/rasterize.py", line 317, in rasterize_rgbad
    image_size, near, far, eps, background_color, return_rgb, return_alpha, return_depth)(*inputs)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/rasterize.py", line 265, in forward
    self.return_rgb, self.return_alpha, self.return_depth)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/rasterize.py", line 98, in forward
    alpha_map = RasterizeFunction.forward_alpha_map(ctx, alpha_map, face_index_map)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/rasterize.py", line 197, in forward_alpha_map
    alpha_map[face_index_map &gt;= 0] = 1
RuntimeError: CUDA out of memory. Tried to allocate 750.00 MiB (GPU 0; 7.80 GiB total capacity; 5.64 GiB already allocated; 380.44 MiB free; 6.30 GiB reserved in total by PyTorch)
loss: 1.05e+04:   0%|▏                        | 1/200.0 [00:01&lt;03:51,  1.16s/it]
</code></pre>

Majid_Al: <p>Probably your batch size is too large? and the model is too big for your cuda?</p>

Mona_Jalal: <p>changed both batch_size=500 lines in pose_optimization.py to 8 and now it’s working.</p>