akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do lite model have performance drop?

Hui_Liu: <p>Hi,</p>
<p>I would like to deploy my pytorch model on mobile devices.</p>
<p>Based on here: <a href="https://pytorch.org/mobile/home/#deployment-workflow" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a>. I need to quantize my model and convert to ptl file.<br>
it seems like I can do inference using quantized model (<a href="https://discuss.pytorch.org/t/how-to-load-quantized-model-for-inference/140283" class="inline-onebox">How to load quantized model for inference</a>) so that I can evaluate whether the performance is dropped.</p>
<p>But I don’t know how to make sure there is no performance drop from quantized/optimized model to ptl file? Or it is guaranteed that conversion from quantized model to ptl has no performance drop?</p>
<p>Thanks!!</p>

marksaroufim: <p>Quantized models will typically have a degradation in accuracy, it’s up to you to decide if the perf speedups are worth that degradation. This is a good blog on the topic <a href="https://pytorch.org/blog/quantization-in-practice/" class="inline-onebox" rel="noopener nofollow ugc">Practical Quantization in PyTorch | PyTorch</a></p>

Hui_Liu: <p>Thanks Mark! Do you know if there is any way to inference on ptl file in colab/python? So that I can have a better idea of how much the performance is dropped?<br>
I think I might just try fp32 first, which I assume should be no precision loss, but I am not sure if there could be performance drop due to different implementation of pyTorch operations ( e.g, the implementation in the lite_interpreter)</p>
Output values are different when eval() mode

yongjun_Hong: <p>Hello.</p>
<p>When I do following, o1 and o3 are different (Isn’t it should be the same values?)</p>
<ul>
<li>Model contains only BatchNorm2d</li>
<li>If model is not updated in train mode (like below, no optimizer step), I think o1 and o3 should be same.</li>
<li>o2 can be different from o1/o3 due to BatchNorm2d</li>
<li>When is it possible o1 and o3 different ? (I think I did something wrong, but I can’t still figure it out)</li>
</ul>
<p>Thanks!</p>
<pre><code class="lang-auto">x = torch.randn(1,3,224,224)

model = Model()
model.eval()

o1 = model(x)

model.train()
o2 = model(x)

model.eval()
o3 = model(x)
</code></pre>

ptrblck: <p>The outputs are computes as:</p>
<ul>
<li>
<code>o1</code> will be computed using the running stats from batchnorm layers to normalize the corresponding input activations.</li>
<li>
<code>o2</code> will be computed using the activation stats in batchnorm layers to normalize the input activations. The running stats of batchnorm layers will be updated.</li>
<li>
<code>o3</code> will use the same approach as <code>o1</code>, but with the updated running stats from the previous forward pass.</li>
</ul>
<p>Let me know, if this clarifies the use case.</p>

yongjun_Hong: <p>Yes, I think you are right.<br>
I didn’t catch the batch norm running stats would be updated at forward pass.<br>
So, o1 and o3 surely different due to updated batchnorm statistics at o2.</p>
<p>Thanks!</p>
With the PyTorch Distributed training model, code changes affect running applications

ckx506772099: <p>I noticed two strange things,<br>
First, when I used Distributed for training, I modified the code. After a while, the application reported an error when RUNNING because of my modification.<br>
Second, I added print to the code, and after a while, the program that was already running will start to print accordingly</p>

wanchaol: <p><a class="mention" href="/u/ckx506772099">@ckx506772099</a> Sorry for the late reply! Please see <a href="https://discuss.pytorch.org/t/ddp-is-affected-by-code-modification/148046" class="inline-onebox">DDP is affected by code modification</a>, this is a python multiprocessing module behavior instead of DDP.</p>