AssertionError: min nan should be less than max nan

jiacheng1gujiaxin: <p>Thank you. After I merged conv and batchnorm. I have solved this problem, but I have encountered this problem in training - aware quantification. Do you have any suggestions?</p>
<blockquote>
<p>File “/home/g/anaconda3/lib/python3.7/site-packages/torch/quantization/observer.py”, line 165, in _calculate_qparams<br>
zero_point = qmin - round(min_val / scale)<br>
ValueError: cannot convert float NaN to integer</p>
</blockquote>
<p>Is there a problem with my data?</p>
<p>Need to add exception capture?</p>
<p>It was later found that Nan existed in conv2d.weight.<br>
And that’s happened in the process of training.</p>

ptrblck: <p>If your weights got a NaN value, this might be due to a NaN input or a faulty weight update caused by e.g. a high learning rate.<br>
Did you observe the loss during training?<br>
If some weights are exploding, you would usually see a NaN loss.</p>

chihyu: <p>I have the similar problem when model use the training - aware quantification and the training loss is not nan. How to solve it?<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “model-code/resnet34/train_age_quan.py”, line 1130, in forward<br>
x = self.ConvBNReLU1(x)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 243, in forward<br>
return self.activation_post_process(F.relu(ConvBn2d._forward(self, input)))<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 95, in _forward<br>
conv = self._conv_forward(input, self.weight_fake_quant(scaled_weight))<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 81, in forward<br>
self.scale, self.zero_point = self.calculate_qparams()<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 76, in calculate_qparams<br>
return self.activation_post_process.calculate_qparams()<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 481, in calculate_qparams<br>
return self._calculate_per_channel_qparams(self.min_vals, self.max_vals)<br>
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 150, in _calculate_per_channel_qparams<br>
), “min {} should be less than max {}”.format(min_vals[i], max_vals[i])<br>
AssertionError: min nan should be less than max nan</p>

jerryzh168: <aside class="quote no-group" data-username="chihyu" data-post="4" data-topic="58995">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/4bbf92/48.png" class="avatar"> chihyu:</div>
<blockquote>
<p>File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in <strong>call</strong></p>
</blockquote>
</aside>
<p>can you check if values in weights/activations contains nan?</p>

haviernick: <aside class="quote no-group" data-username="jiacheng1gujiaxin" data-post="1" data-topic="58995">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f17d59/48.png" class="avatar"> jiacheng1gujiaxin:</div>
<blockquote>
<p>ValueError: cannot convert float NaN to integer</p>
</blockquote>
</aside>
<p>You can avoid this with a mask method. Note first that in python NaN is defined as the number which is not equal to itself:</p>
<pre><code class="lang-auto">&gt;float('nan') == float('nan')      
False
</code></pre>
<p>It might be worth avoiding use of np.NaN altogether. NaN literally means “not a number”, and it cannot be converted to an integer.  In general, Python prefers raising an exception to returning NaN, so things like sqrt(-1) and log(0.0) will generally raise instead of returning NaN. However, you may get this value back from some other library.  From v0.24, you actually can. Pandas introduces Nullable Integer Data Types which allows integers to coexist with NaNs.  Also, even at the lastest versions of <a href="http://net-informations.com/ds/pd/default.htm" rel="noopener nofollow ugc">pandas</a> if the column is object type you would have to convert into float first, something like:</p>
<p><code>df['column_name'].astype(np.float).astype("Int32")</code></p>
<p>NB: You have to go through numpy float first and then to nullable Int32, for some reason.</p>