Increasing the batch size from 64 to 512 is not decreasing the runtime in 2D to 3D lifting from RGB images

username: Mona_Jalal
cooked: <p>[ a double-post of <a href="https://github.com/facebookresearch/phosa/issues/38" class="inline-onebox" rel="noopener nofollow ugc">How to speed up the rendering of 3D lifting from 2D images? · Issue #38 · facebookresearch/phosa · GitHub</a> ]<br>
I am noticing that despite increasing the batch size to 512 and moving to a 12GB GPU, I am not gaining speedup really. I have changed both occurrences of batch_size=XYZ in file <code>phosa/pose_optimization.py</code>  from XYZ to 512.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L103" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L103" target="_blank" rel="noopener nofollow ugc">facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L103</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="93" style="counter-reset: li-counter 92 ;">
<li>"""</li>
<li>
</li>
<li>def __init__(</li>
<li>    self,</li>
<li>    ref_image,</li>
<li>    vertices,</li>
<li>    faces,</li>
<li>    textures,</li>
<li>    rotation_init,</li>
<li>    translation_init,</li>
<li class="selected">    batch_size=1,</li>
<li>    kernel_size=7,</li>
<li>    K=None,</li>
<li>    power=0.25,</li>
<li>    lw_chamfer=0.5,</li>
<li>):</li>
<li>    assert ref_image.shape[0] == ref_image.shape[1], "Must be square."</li>
<li>    super(PoseOptimizer, self).__init__()</li>
<li>
</li>
<li>    self.register_buffer("vertices", vertices.repeat(batch_size, 1, 1))</li>
<li>    self.register_buffer("faces", faces.repeat(batch_size, 1, 1))</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L265" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L265" target="_blank" rel="noopener nofollow ugc">facebookresearch/phosa/blob/207a6b1754b80115dbbcb9d8abb9005b11e01f1f/phosa/pose_optimization.py#L265</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="255" style="counter-reset: li-counter 254 ;">
<li>    plt.show()</li>
<li>
</li>
<li>
</li>
<li>def find_optimal_pose(</li>
<li>    vertices,</li>
<li>    faces,</li>
<li>    mask,</li>
<li>    bbox,</li>
<li>    square_bbox,</li>
<li>    image_size,</li>
<li class="selected">    batch_size=500,</li>
<li>    num_iterations=50,</li>
<li>    num_initializations=2000,</li>
<li>    lr=1e-3,</li>
<li>):</li>
<li>    ts = 1</li>
<li>    textures = torch.ones(faces.shape[0], ts, ts, ts, 3, dtype=torch.float32).cuda()</li>
<li>    x, y, b, _ = square_bbox</li>
<li>    L = max(image_size)</li>
<li>    K_roi = compute_K_roi((x, y), b, L, focal_length=FOCAL_LENGTH)</li>
<li>    # Stuff to keep around</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Do you have any other recommendations that could perhaps make it faster? How can I actually check to make sure batch size of 512 is set via pytorch? What other techniques should be used to get speedup?</p>
<p>Thank you,<br>
Mona</p>
<p>Here’s what I see when I run the demo code for one single image, it takes like 3-5 min:</p>
<pre><code class="lang-auto">(phosa) [jalal@goku phosa]$ python demo.py --filename input/bike_ride2.jpg --class_name bicycle
2021-05-11 17:06:08,654 INFO     Calling with args: Namespace(class_name='bicycle', filename='input/bike_ride2.jpg', lw_collision=None, lw_depth=None, lw_inter=None, lw_inter_part=None, lw_scale=None, lw_scale_person=None, lw_sil=None, mesh_index=0, output_dir='output')
2021-05-11 17:06:11,258 INFO     Loading checkpoint from detectron2://PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2021-05-11 17:06:11,291 INFO     URL https://dl.fbaipublicfiles.com/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl cached in /home/grad3/jalal/.torch/fvcore_cache/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2021-05-11 17:06:14,962 INFO     Reading a file from 'Detectron2 Model Zoo'
WARNING: You are using a SMPL model, with only 10 shape coefficients.
class_name:  bicycle
  0%|                                                                                         | 0/200.0 [00:00&lt;?, ?it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.88e+03:  25%|████████████████                                                | 50/200.0 [00:46&lt;02:15,  1.10it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.88e+03:  50%|███████████████████████████████▌                               | 100/200.0 [01:32&lt;01:31,  1.10it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.88e+03:  75%|███████████████████████████████████████████████▎               | 150/200.0 [02:18&lt;00:45,  1.10it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.88e+03: 100%|███████████████████████████████████████████████████████████████| 200/200.0 [03:04&lt;00:00,  1.08it/s]
  0%|                                                                                                    | 0/200.0 [00:00&lt;?, ?it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.97e+03:  25%|██████████████████▊                                                        | 50/200.0 [00:49&lt;02:26,  1.03it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.97e+03:  50%|█████████████████████████████████████                                     | 100/200.0 [01:39&lt;01:37,  1.03it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 5.97e+03:  75%|███████████████████████████████████████████████████████▌                  | 150/200.0 [02:28&lt;00:48,  1.04it/s]PoseOptimizer(
  (pool): MaxPool2d(kernel_size=7, stride=1, padding=3, dilation=1, ceil_mode=False)
  (renderer): Renderer()
)
loss: 4.59e+03: 100%|██████████████████████████████████████████████████████████████████████████| 200/200.0 [03:17&lt;00:00,  1.01it/s]
Loss 77.4604: 100%|██████████████████████████████████████████████████████████████████████████████| 400/400 [03:21&lt;00:00,  1.99it/s]
2021-05-11 17:16:02,046 INFO     Saved rendered image to output/bike_ride2.jpg.
2021-05-11 17:16:02,067 INFO     Saved top-down image to output/bike_ride2_top.jpg.

</code></pre>

username: eqy
cooked: <p>A crude test you can try is to simply inspect the amount of GPU memory used in something like <code>nvidia-smi</code> or <code>gpustat</code>. If the memory usage is not increasing, then something is suspicious. Alternatively, you can try increasing the batch sizes until you run out of GPU memory (which should happen).</p>