RuntimeError: value cannot be converted to type float without overflow: (-2.42042e-22,3.95285e-06)

username: amittiwari
cooked: <p>I am training speech to text model on OpenNMT-py.<br>
I used MFCC algo at preprocess level. But unable to start training.<br>
python3 train.py -model_type audio -enc_rnn_size 1024 -dec_rnn_size 1024 -audio_enc_pooling 1,1,1,2,2,2 -dropout 0.1 -enc_layers 6 -dec_layers 4 -rnn_type LSTM -data data/speech/demofiles-vctk-mfcc/demo -save_model models/exp-vctk-mfcc/demo-model-vctk-mfcc -global_attention mlp -batch_size 6  -optim sgd -max_grad_norm 100 -decay_method noam -train_steps 10000 -encoder_type brnn -decoder_type rnn -bridge -window_size 0.025 -image_channel_size 1<br>
After running this at time of data loading,<br>
[2019-10-29 14:42:49,730 INFO]  * tgt vocab size = 9386<br>
[2019-10-29 14:42:49,732 INFO] Building model…<br>
/home/amit/.local/lib/python3.5/site-packages/torch/nn/modules/rnn.py:51: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.1 and num_layers=1<br>
“num_layers={}”.format(dropout, num_layers))<br>
[2019-10-29 14:42:50,667 INFO] NMTModel(<br>
(encoder): AudioEncoder(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(W): Linear(in_features=1024, out_features=1024, bias=False)<br>
(batchnorm_0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(rnn_0): LSTM(26, 512, dropout=0.1, bidirectional=True)<br>
(pool_0): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(rnn_1): LSTM(1024, 512, dropout=0.1, bidirectional=True)<br>
(pool_1): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(batchnorm_1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(rnn_2): LSTM(1024, 512, dropout=0.1, bidirectional=True)<br>
(pool_2): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)<br>
(batchnorm_2): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(rnn_3): LSTM(1024, 512, dropout=0.1, bidirectional=True)<br>
(pool_3): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(batchnorm_3): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(rnn_4): LSTM(1024, 512, dropout=0.1, bidirectional=True)<br>
(pool_4): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(batchnorm_4): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(rnn_5): LSTM(1024, 512, dropout=0.1, bidirectional=True)<br>
(pool_5): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(batchnorm_5): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(decoder): InputFeedRNNDecoder(<br>
(embeddings): Embeddings(<br>
(make_embedding): Sequential(<br>
(emb_luts): Elementwise(<br>
(0): Embedding(9386, 500, padding_idx=1)<br>
)<br>
)<br>
)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(rnn): StackedLSTM(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(layers): ModuleList(<br>
(0): LSTMCell(1524, 1024)<br>
(1): LSTMCell(1024, 1024)<br>
(2): LSTMCell(1024, 1024)<br>
(3): LSTMCell(1024, 1024)<br>
)<br>
)<br>
(attn): GlobalAttention(<br>
(linear_context): Linear(in_features=1024, out_features=1024, bias=False)<br>
(linear_query): Linear(in_features=1024, out_features=1024, bias=True)<br>
(v): Linear(in_features=1024, out_features=1, bias=False)<br>
(linear_out): Linear(in_features=2048, out_features=1024, bias=True)<br>
)<br>
)<br>
(generator): Sequential(<br>
(0): Linear(in_features=1024, out_features=9386, bias=True)<br>
(1): Cast()<br>
(2): LogSoftmax()<br>
)<br>
)<br>
[2019-10-29 14:42:50,668 INFO] encoder: 34770944<br>
[2019-10-29 14:42:50,668 INFO] decoder: 54146226<br>
[2019-10-29 14:42:50,668 INFO] * number of parameters: 88917170<br>
[2019-10-29 14:42:50,669 INFO] Starting training on CPU, could be very slow<br>
[2019-10-29 14:42:50,669 INFO] Start training loop and validate every 10000 steps…<br>
[2019-10-29 14:42:50,669 INFO] Loading dataset from data/speech/demofiles-vctk-mfcc/demo.train.0.pt<br>
[2019-10-29 14:42:50,923 INFO] number of examples: 5000<br>
Traceback (most recent call last):<br>
File “train.py”, line 200, in <br>
main(opt)<br>
File “train.py”, line 88, in main<br>
single_main(opt, -1)<br>
File “/home/amit/Desktop/amit/OpenNMT-py/onmt/train_single.py”, line 143, in main<br>
valid_steps=opt.valid_steps)<br>
File “/home/amit/Desktop/amit/OpenNMT-py/onmt/trainer.py”, line 243, in train<br>
report_stats)<br>
File “/home/amit/Desktop/amit/OpenNMT-py/onmt/trainer.py”, line 392, in _gradient_accumulation<br>
self.optim.step()<br>
File “/home/amit/Desktop/amit/OpenNMT-py/onmt/utils/optimizers.py”, line 360, in step<br>
self.<em>optimizer.step()<br>
File “/home/amit/.local/lib/python3.5/site-packages/torch/optim/sgd.py”, line 106, in step<br>
p.data.add</em>(-group[‘lr’], d_p)<br>
RuntimeError: value cannot be converted to type float without overflow: (-2.42042e-22,3.95285e-06)</p>

username: albanD
cooked: <p>Hi,</p>
<p>Have you seen this warning in your code</p>
<pre><code class="lang-auto">/home/amit/.local/lib/python3.5/site-packages/torch/nn/modules/rnn.py:51: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.1 and num_layers=1
</code></pre>
<p>?</p>
<p>Also what is the value of your learning rate?</p>

username: amittiwari
cooked: <p>Hi,<br>
it was default as set in OpenNMT-py.<br>
lr=1</p>

username: amittiwari
cooked: <p>I have increased the learning rate to 10.<br>
Still not working.</p>

username: albanD
cooked: <p>This warning does not talk about the learning rate. It is a problem with the use of dropout for an rnn with a single layer.</p>

username: amittiwari
cooked: <p>I have tried it. But same result.<br>
When I am using decay method “rsqrt” it works fine.<br>
but “noam” as decay method it is not working.</p>

username: albanD
cooked: <p>I’m not sure to understand what these arguments mean. IT depends on your code.</p>
<p>Yo can try to <a href="https://pytorch.org/docs/stable/torch.html#torch.set_flush_denormal">flush denormals</a> to avoid such large small numbers. It might help if you<br>
re doing compute on the CPU.</p>

username: amittiwari
cooked: <p>This problem occurred on GPU .</p>

username: albanD
cooked: <p><a class="mention" href="/u/ngimel">@ngimel</a> do you know where this error comes from?</p>

username: amittiwari
cooked: <p>error comes while calculating loss and step size of gradient.</p>