Repository: awslabs/gluonts

Issue #501: NaN gradient in TransformerEstimator when use GPU but works fine for CPU
Created: 2019-12-08T17:52:35Z
i found that loss function defined in hybrid_forward for TransformerTrainingNetwork(overall mean) is different from DeepARTrainingNetwork(mean on axis=1), then attempt to modify TransformerTrainingNetwork's loss to  DeepARTrainingNetwork's format(include observed_values filter), but when loss sum larger then some values(exact value not tested, usually 1000+), some model params gradient become NaN after first batch loss backward (thus before any params update by Trainer) when using GPU, and with the same model config only change to training using CPU all works fine even for many epochs, so,

is overall mean use in TransformerTrainingNetwork by design to reduce loss sum?
what's the possible reason for works fine in CPU but encounter NaN gradient with GPU?

ps. there is no NaN in loss vector before NaN first appearance in params gradient.
envs: CentOS 7.5, python 3.6.7, mxnet-cu90 1.4.1,  gluonts commit fbcb244, GTX 1080 Ti, CUDA Version 9.0.176, cuDNN 7.0.5
params log in training:

with GPU

INFO:root:Start model training
INFO:root:Epoch[0] Learning rate is 0.001
  0%|                                                                                          | 0/2 [00:00<?, ?it/s]
===loss sum 1141.0615 ,loss nan count 0
transformertrainingnetwork0_None_distr_mu_weight , data abs mean: 0.1807779 ,grad abs mean: 360.64877
transformertrainingnetwork0_None_distr_mu_bias , data abs mean: 0.0 ,grad abs mean: 479.75717
transformertrainingnetwork0_None_distr_sigma_weight , data abs mean: 0.24338286 ,grad abs mean: 227.50531
transformertrainingnetwork0_None_distr_sigma_bias , data abs mean: 0.0 ,grad abs mean: 300.70776
transformertrainingnetwork0_None_distr_nu_weight , data abs mean: 0.20589632 ,grad abs mean: 3.034824
transformertrainingnetwork0_None_distr_nu_bias , data abs mean: 0.0 ,grad abs mean: 4.0935373
enc_inputlayer0_dense0_weight , data abs mean: 0.12418315 ,grad abs mean: nan
enc_inputlayer0_dense0_bias , data abs mean: 0.0 ,grad abs mean: nan
enc_layernormalization0_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: nan
enc_layernormalization0_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 2.521532e+32
enc_multiheadselfattention_dense0_weight , data abs mean: 0.14938776 ,grad abs mean: 8.457253e+31
enc_multiheadselfattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 1.90768e+32
enc_multiheadselfattention_dense1_weight , data abs mean: 0.10618979 ,grad abs mean: 1.0929538e+32
enc_multiheadselfattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 1.3961101e+32
enc_layernormalization1_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 6315757000000.0
enc_layernormalization1_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 1845129.0
enc_transformerfeedforward_dense0_weight , data abs mean: 0.09692495 ,grad abs mean: 437317.97
enc_transformerfeedforward_dense0_bias , data abs mean: 0.0 ,grad abs mean: 550232.75
enc_transformerfeedforward_dense1_weight , data abs mean: 0.09759854 ,grad abs mean: 1098986.4
enc_transformerfeedforward_dense1_bias , data abs mean: 0.0 ,grad abs mean: 1496465.0
enc_layernormalization2_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 1794.3195
enc_layernormalization2_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 427.01648
dec_inputlayer0_dense0_weight , data abs mean: 0.12752555 ,grad abs mean: nan
dec_inputlayer0_dense0_bias , data abs mean: 0.0 ,grad abs mean: nan
dec_layernormalization0_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 1.6512583e+25
dec_layernormalization0_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 3472032300000.0
dec_multiheadselfattention_dense0_weight , data abs mean: 0.15343781 ,grad abs mean: 1429269300000.0
dec_multiheadselfattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 2170051500000.0
dec_multiheadselfattention_dense1_weight , data abs mean: 0.108351976 ,grad abs mean: 1272869300000.0
dec_multiheadselfattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 1647051900000.0
dec_layernormalization1_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 169149.33
dec_layernormalization1_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 349.0154
dec_multiheadattention_dense0_weight , data abs mean: 0.15222535 ,grad abs mean: 232.39099
dec_multiheadattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 380.73752
dec_multiheadattention_dense1_weight , data abs mean: 0.15506381 ,grad abs mean: 11.908104
dec_multiheadattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 14.781871
dec_multiheadattention_dense2_weight , data abs mean: 0.15391286 ,grad abs mean: 7.889603
dec_multiheadattention_dense2_bias , data abs mean: 0.0 ,grad abs mean: 1.2464356e-05
dec_multiheadattention_dense3_weight , data abs mean: 0.15240093 ,grad abs mean: 390.4511
dec_multiheadattention_dense3_bias , data abs mean: 0.0 ,grad abs mean: 501.0399
dec_layernormalization2_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 67.898125
dec_layernormalization2_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 84.413895
dec_transformerfeedforward_dense0_weight , data abs mean: 0.097284675 ,grad abs mean: 21.935583
dec_transformerfeedforward_dense0_bias , data abs mean: 0.0 ,grad abs mean: 28.14898
dec_transformerfeedforward_dense1_weight , data abs mean: 0.097281046 ,grad abs mean: 65.9311
dec_transformerfeedforward_dense1_bias , data abs mean: 0.0 ,grad abs mean: 88.86635
dec_layernormalization3_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 96.66262
dec_layernormalization3_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 128.00034
transformertrainingnetwork0_featureembedder0_cat_0_embedding_weight , data abs mean: 0.21088068 ,grad abs mean: nan
INFO:root:Number of parameters in TransformerTrainingNetwork: 34231



with CPU

INFO:root:Start model training
INFO:root:Epoch[0] Learning rate is 0.001
  0%|                                                                                                   | 0/2 [00:00<?, ?it/s]
===loss sum 1397.0464 ,loss nan count 0
transformertrainingnetwork0_None_distr_mu_weight , data abs mean: 0.21154597 ,grad abs mean: 143.13747
transformertrainingnetwork0_None_distr_mu_bias , data abs mean: 0.0 ,grad abs mean: 185.55713
transformertrainingnetwork0_None_distr_sigma_weight , data abs mean: 0.24524659 ,grad abs mean: 207.92241
transformertrainingnetwork0_None_distr_sigma_bias , data abs mean: 0.0 ,grad abs mean: 265.96445
transformertrainingnetwork0_None_distr_nu_weight , data abs mean: 0.20836934 ,grad abs mean: 20.236267
transformertrainingnetwork0_None_distr_nu_bias , data abs mean: 0.0 ,grad abs mean: 25.69025
enc_inputlayer0_dense0_weight , data abs mean: 0.12418315 ,grad abs mean: 12.238937
enc_inputlayer0_dense0_bias , data abs mean: 0.0 ,grad abs mean: 18.019035
enc_layernormalization0_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 5.648037
enc_layernormalization0_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 8.609286
enc_multiheadselfattention_dense0_weight , data abs mean: 0.14983651 ,grad abs mean: 8.037011
enc_multiheadselfattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 14.096378
enc_multiheadselfattention_dense1_weight , data abs mean: 0.107939176 ,grad abs mean: 2.7715132
enc_multiheadselfattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 3.6767232
enc_layernormalization1_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 12.474259
enc_layernormalization1_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 16.955927
enc_transformerfeedforward_dense0_weight , data abs mean: 0.095692255 ,grad abs mean: 5.047846
enc_transformerfeedforward_dense0_bias , data abs mean: 0.0 ,grad abs mean: 6.8885784
enc_transformerfeedforward_dense1_weight , data abs mean: 0.09778515 ,grad abs mean: 14.155687
enc_transformerfeedforward_dense1_bias , data abs mean: 0.0 ,grad abs mean: 19.242792
enc_layernormalization2_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 20.647558
enc_layernormalization2_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 27.836872
dec_inputlayer0_dense0_weight , data abs mean: 0.12385981 ,grad abs mean: 14.3610735
dec_inputlayer0_dense0_bias , data abs mean: 0.0 ,grad abs mean: 20.458885
dec_layernormalization0_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 8.040732
dec_layernormalization0_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 11.1711645
dec_multiheadselfattention_dense0_weight , data abs mean: 0.15260002 ,grad abs mean: 9.06193
dec_multiheadselfattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 17.194529
dec_multiheadselfattention_dense1_weight , data abs mean: 0.106750794 ,grad abs mean: 5.6618304
dec_multiheadselfattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 7.0167923
dec_layernormalization1_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 24.55144
dec_layernormalization1_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 26.383638
dec_multiheadattention_dense0_weight , data abs mean: 0.15347582 ,grad abs mean: 21.468824
dec_multiheadattention_dense0_bias , data abs mean: 0.0 ,grad abs mean: 26.453135
dec_multiheadattention_dense1_weight , data abs mean: 0.1517558 ,grad abs mean: 0.54713905
dec_multiheadattention_dense1_bias , data abs mean: 0.0 ,grad abs mean: 0.7014004
dec_multiheadattention_dense2_weight , data abs mean: 0.15080683 ,grad abs mean: 0.54961073
dec_multiheadattention_dense2_bias , data abs mean: 0.0 ,grad abs mean: 4.3104407e-08
dec_multiheadattention_dense3_weight , data abs mean: 0.15244319 ,grad abs mean: 19.834389
dec_multiheadattention_dense3_bias , data abs mean: 0.0 ,grad abs mean: 27.47301
dec_layernormalization2_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 30.298561
dec_layernormalization2_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 39.08081
dec_transformerfeedforward_dense0_weight , data abs mean: 0.09609707 ,grad abs mean: 10.228908
dec_transformerfeedforward_dense0_bias , data abs mean: 0.0 ,grad abs mean: 12.891011
dec_transformerfeedforward_dense1_weight , data abs mean: 0.09515645 ,grad abs mean: 30.626188
dec_transformerfeedforward_dense1_bias , data abs mean: 0.0 ,grad abs mean: 40.005424
dec_layernormalization3_layernorm0_gamma , data abs mean: 1.0 ,grad abs mean: 50.41512
dec_layernormalization3_layernorm0_beta , data abs mean: 0.0 ,grad abs mean: 64.15854
transformertrainingnetwork0_featureembedder0_cat_0_embedding_weight , data abs mean: 0.21088068 ,grad abs mean: 33.846794
INFO:root:Number of parameters in TransformerTrainingNetwork: 34231

Duplicate issue #950: Getting Nans when training MQCNN on GPU
Created: 2020-07-24T15:47:09Z
Description
FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu18.04

RUN apt-get update --fix-missing && \
    apt-get install -y --no-install-recommends \
        wget bzip2 ca-certificates curl git build-essential \
        cmake python3.7 python3.7-dev python3.7-distutils pkg-config && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3.7 get-pip.py

# We're emulating nvidia-smi to be able to build GPU container on devices without GPU
RUN echo '#!/bin/bash\necho Building container with GPU support' > /usr/bin/nvidia-smi && \
    chmod +x /usr/bin/nvidia-smi

ADD . /gluonts

RUN pip3.7 install mxnet-cu92>=1.6
RUN pip3.7 install /gluonts[shell]

ENTRYPOINT ["python3.7", "-m", "gluonts.shell"]



If I set hybridize=False it works again.
This does not happen in our internal MXNet version/container (see MQCNN).
This does not happen with older MXNet versions (I think 1.4)
This does happen also for newer Cuda versions (I think 10.?)
Reducing the initial learning rate only works for ridiculously low learning rates, and even then it's not reliable

Error message or code output
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/gluonts/shell/__main__.py", line 154, in <module>
    cli(prog_name=__package__)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/gluonts/shell/__main__.py", line 131, in train_command
    train.run_train_and_test(env, forecaster_type_by_name(forecaster))
  File "/usr/local/lib/python3.7/dist-packages/gluonts/shell/train.py", line 85, in run_train_and_test
    forecaster, env.datasets["train"], env.datasets.get("validation")
  File "/usr/local/lib/python3.7/dist-packages/gluonts/shell/train.py", line 100, in run_train
    training_data=train_dataset, validation_data=validation_dataset
  File "/usr/local/lib/python3.7/dist-packages/gluonts/model/estimator.py", line 271, in train
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/gluonts/model/estimator.py", line 244, in train_model
    validation_iter=validation_data_loader,
  File "/usr/local/lib/python3.7/dist-packages/gluonts/mx/trainer/_base.py", line 373, in __call__
    "Got NaN in first epoch. Try reducing initial learning rate."

Environment

see Dockerfile

(Add as much information about your environment as possible, e.g. dependencies versions.)

Closing comment (2020-08-03T14:58:11Z): Possibly related to: #501