Tips for increasing Inference FPS

username: emcp
cooked: <p>For background I am currently utilizing a version of this project <a href="https://github.com/jwyang/faster-rcnn.pytorch/tree/pytorch-1.0" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jwyang/faster-rcnn.pytorch at pytorch-1.0</a></p>
<p>basically Faster R CNN implementation to run against a webcam , or set of jpg on disk…</p>
<p>My question is , any basic tips for increasing the FPS when running my demo mode</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/jwyang/faster-rcnn.pytorch/blob/pytorch-1.0/demo.py#L96">
  <header class="source">

      <a href="https://github.com/jwyang/faster-rcnn.pytorch/blob/pytorch-1.0/demo.py#L96" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/jwyang/faster-rcnn.pytorch/blob/pytorch-1.0/demo.py#L96" target="_blank" rel="noopener nofollow ugc">jwyang/faster-rcnn.pytorch/blob/pytorch-1.0/demo.py#L96</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="86" style="counter-reset: li-counter 85 ;">
          <li>                      default=1, type=int)</li>
          <li>  parser.add_argument('--checkpoint', dest='checkpoint',</li>
          <li>                      help='checkpoint to load network',</li>
          <li>                      default=10021, type=int)</li>
          <li>  parser.add_argument('--bs', dest='batch_size',</li>
          <li>                      help='batch_size',</li>
          <li>                      default=1, type=int)</li>
          <li>  parser.add_argument('--vis', dest='vis',</li>
          <li>                      help='visualization mode',</li>
          <li>                      action='store_true')</li>
          <li class="selected">  parser.add_argument('--webcam_num', dest='webcam_num',</li>
          <li>                      help='webcam ID number',</li>
          <li>                      default=-1, type=int)</li>
          <li>
          </li>
<li>  args = parser.parse_args()</li>
          <li>  return args</li>
          <li>
          </li>
<li>lr = cfg.TRAIN.LEARNING_RATE</li>
          <li>momentum = cfg.TRAIN.MOMENTUM</li>
          <li>weight_decay = cfg.TRAIN.WEIGHT_DECAY</li>
          <li>
      </li>
</ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’m wondering if I should come up with a buffering strategy… try to compile the code with pytorch from source?  switch to ONNX and load the model in C++ version of pytorch or some other inference focused runtime?</p>
<p>I’ve customized a stream which sends frames over a network socket… and thinking about switching to a compression technique to improve FPS (I currently am getting about 5 FPS on a 600x700px image of type JPG)… but really new to this space or how to speed things up</p>
<p>I’m already running latest PyTorch 1.10 and CUDA supported by my machine 11.3</p>
<p>RTX 3090 + Ryzen 9 5950x with 64 GB of ECC 3200mhz RAM so I cannot really give it much more on the hardware side</p>
<p>Thanks</p>