Is it possible to enable apex (opt_level=O1) only while inferencing on a model, purely trained on FP32 (without apex)

Sarthak_Vora: <p>I want to run a few inference results on a pre-trained model trained purely on FP32 (without apex or amp). My main aim is to get faster results while inferencing and <strong>not</strong> while training.  Hence, ideally, I want to initialize mixed precision learning after the model is trained and before the model is inputted with unseen data. How is it possible to do the same using NVIDIA apex library ? It’d be great if some code snippets can be attached too.</p>
<p>Secondly, for some model trained and inferred purely on FP32, let’s say <code>modelA</code> and a model trained on FP32 (without apex or amp) but inferred with apex (opt_level=‘O1’), let’s say <code>modelB</code>, how would the <strong>inference execution time</strong> of the below code for both the models differ ?</p>
<pre><code class="lang-auto">modelA = modelA.to(device) # to be inferred without apex
modelB = modelB.to(device) # to be inferred with apex
tensor = torch.rand(1,C,H,W).to(device) # random tensor for testing

with torch.no_grad():
    modelA.eval()
    modelA(tensor) # inferencing
    # Calculate cuda_time for the execution

with torch.no_grad():
    modelB.eval()
    # Initialize apex (opt_level='O1') code snippets for faster inferencing
    modelB(tensor) # inferencing
    # Calculate cuda_time for the execution

</code></pre>

ptrblck: <p><code>apex.amp</code> is deprecated and you should use the native mixed-precision utility via <code>torch.cuda.amp</code> as described <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a>.</p>
<p>With that being said, yes, it’s possible to activate <code>autocast</code> only for the inference. The <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">docs</a> give you some examples and in particular you can skip the training utils (e.g. the <code>GradScaler</code>).</p>

mohit117: <p>Thankyou for this reply. I will go through the links you mentioned. But wanted to as one question.</p>
<p>We are interested in fast inference speed more than fast training. Does mixed precison using <code>torch.cuda.amp</code> allows faster inference also or speeds up only training?</p>

ptrblck: <p>As said before, it’s possible to activate <code>autocast</code> during inference and the actual speedup depends on the model architecture, cudnn version, GPU, etc.</p>

mohit117: <p>Thank you for your reply, it really works!!<br>
I did several experiments with Resnet50 for input tensors of different spatial resolutions like below:</p>
<pre><code class="lang-auto">with torch.no_grad():
  model.eval()

  start = torch.cuda.Event(enable_timing=True)
  end = torch.cuda.Event(enable_timing=True)

  model(tensor) #warmup

  start.record()
  with torch.cuda.amp.autocast(enabled=True):
    for i in range(10):    
        model(tensor)
  end.record()
  torch.cuda.synchronize()
  print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/10))
</code></pre>
<p>with <code>autocast</code> in each case, I got at least 2x speedup during inference, which is really appreciable. But could you please tell why this speedup? What magic is PyTorch doing under the hood to get this speedup?</p>

ptrblck: <p>From the <a href="https://developer.nvidia.com/automatic-mixed-precision">NVIDIA post</a>:</p>
<h4>
<a name="benefits-of-mixed-precision-training-1" class="anchor" href="#benefits-of-mixed-precision-training-1"></a>Benefits of Mixed precision training</h4>
<ul>
<li>Speeds up math-intensive operations, such as linear and convolution layers, by using <a href="https://developer.nvidia.com/tensor-cores">Tensor Cores.</a>
</li>
<li>Speeds up memory-limited operations by accessing half the bytes compared to single-precision.</li>
<li>Reduces memory requirements for training models, enabling larger models or larger minibatches.</li>
</ul>