Repository: awslabs/gluonts

Issue #2264: TemporalFusionTransformer fails on GPU if non-default num_outputs is provided
Created: 2022-09-01T09:47:44Z
Description
TemporalFusionTransformerEstimator crashes when training on GPU with num_outputs != 3 (any non-default value).
To Reproduce
import mxnet as mx
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.model.tft import TemporalFusionTransformerEstimator
from gluonts.mx.trainer import Trainer

dataset = get_dataset("m3_other")
ctx = mx.context.gpu()
model = TemporalFusionTransformerEstimator(
    trainer=Trainer(ctx=ctx),
    freq=dataset.metadata.freq,
    prediction_length=dataset.metadata.prediction_length,
    num_outputs=9,
)
model.train(dataset.train)
If I change ctx = mx.context.cpu() or set num_outputs=3, the problem disappears.
Error message or code output
Traceback (most recent call last):
  File "/local/home/shchuro/workspace/autogluon/shchuro/train_gluonts.py", line 14, in <module>
    model.train(dataset.train)
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/gluonts/mx/model/estimator.py", line 194, in train
    return self.train_model(
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/gluonts/mx/model/estimator.py", line 169, in train_model
    self.trainer(
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/gluonts/mx/trainer/_base.py", line 436, in __call__
    epoch_loss = loop(
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/gluonts/mx/trainer/_base.py", line 334, in loop
    _ = net(*batch.values())
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/mxnet/gluon/block.py", line 825, in __call__
    out = self.forward(*args)
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/mxnet/gluon/block.py", line 1492, in forward
    return self._call_cached_op(x, *args)
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/mxnet/gluon/block.py", line 1233, in _call_cached_op
    out = self._cached_op(*cargs)
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/mxnet/_ctypes/ndarray.py", line 148, in __call__
    check_call(_LIB.MXInvokeCachedOpEx(
  File "/home/shchuro/miniconda3/envs/ag/lib/python3.9/site-packages/mxnet/base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "../3rdparty/tvm/nnvm/src/core/graph.cc", line 109
MXNetError: Check failed: it != node2index_.end(): control dep not found in graph

Environment

Operating system: AL2
Python version: 3.9.13
GluonTS version: 0.9.6
MXNet version: 1.9.1-cu112
CUDA Version: 11.7
GPU: Tesla V100

Duplicate issue #2316: mqcnn on gpu: control dep not found in graph
Created: 2022-09-20T12:16:03Z
Training mqcnn models on a gpu is consistently failing for me with the error
MXNetError: Check failed: it != node2index_.end(): control dep not found in graph

Complete error message:
$ python examples/benchmark_m4.py 
evaluating gluonts.model.seq2seq._mq_dnn_estimator.MQCNNEstimator(add_age_feature=False, add_time_feature=True, batch_size=32, cardinality=[414], channels_seq=None, context_length=None, decoder_mlp_dim_seq=None, dilation_seq=None, distr_output=None, embedding_dimension=None, enable_decoder_dynamic_feature=True, enable_encoder_dynamic_feature=True, freq="H", is_iqf=True, kernel_size_seq=None, max_ts_len=None, num_forking=None, prediction_length=48, quantiles=None, scaling=None, scaling_decoder_dynamic_feature=False, seed=None, trainer=gluonts.mx.trainer._base.Trainer(add_default_callbacks=True, batch_size=None, callbacks=None, clip_gradient=10.0, ctx=mxnet.context.Context("gpu", 0), epochs=100, hybridize=True, init="xavier", learning_rate=0.001, learning_rate_decay_factor=0.5, minimum_learning_rate=5e-05, num_batches_per_epoch=50, patience=10, weight_decay=1e-08), use_feat_dynamic_real=False, use_feat_static_cat=True, use_past_feat_dynamic_real=False, use_residual=True) on TrainDatasets(metadata=MetaData(freq='H', target=None, feat_static_cat=[CategoricalFeatureInfo(name='feat_static_cat', cardinality='414')], feat_static_real=[], feat_dynamic_real=[], feat_dynamic_cat=[], prediction_length=48), train=DatasetCollection(datasets=[Map(data=JsonLinesFile(path=PosixPath('/home/matus/.mxnet/gluon-ts/datasets/m4_hourly/train/data.json')))], interleave=False), test=DatasetCollection(datasets=[Map(data=JsonLinesFile(path=PosixPath('/home/matus/.mxnet/gluon-ts/datasets/m4_hourly/test/data.json')))], interleave=False))
[14:02:48] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8201 != compiled-against version 8101.  Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
  0%|                                                                                                                                                                   | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "../3rdparty/tvm/nnvm/src/core/graph.cc", line 109
MXNetError: Check failed: it != node2index_.end(): control dep not found in graph
Traceback (most recent call last):
  File "examples/benchmark_m4.py", line 104, in <module>
    sub_df = df[
  File "/home/matus/miniconda3/envs/eng-gpu/lib/python3.8/site-packages/pandas/core/frame.py", line 3511, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/matus/miniconda3/envs/eng-gpu/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 5796, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/matus/miniconda3/envs/eng-gpu/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 5856, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['dataset', 'estimator', 'RMSE', 'mean_wQuantileLoss', 'MASE', 'sMAPE',\n       'OWA', 'MSIS'],\n      dtype='object')] are in the [columns]"

This output comes from a simplified version of benchmark_m4.py:
diff --git a/examples/benchmark_m4.py b/examples/benchmark_m4.py
index 84f442b..1844099 100644
--- a/examples/benchmark_m4.py
+++ b/examples/benchmark_m4.py
@@ -14,6 +14,10 @@
 """
 This example shows how to fit a model and evaluate its predictions.
 """
+import warnings
+warnings.simplefilter(action='once', category=FutureWarning)
+#warnings.simplefilter(action='ignore', category=FutureWarning)
+
 import pprint
 from functools import partial
 
@@ -28,22 +32,22 @@ from gluonts.mx.trainer import Trainer
 
 datasets = [
     "m4_hourly",
-    "m4_daily",
-    "m4_weekly",
-    "m4_monthly",
-    "m4_quarterly",
-    "m4_yearly",
+    # "m4_daily",
+    # "m4_weekly",
+    # "m4_monthly",
+    # "m4_quarterly",
+    # "m4_yearly",
 ]
 
 epochs = 100
 num_batches_per_epoch = 50
 estimators = [
     MQCNNEstimator,
-    DeepAREstimator,
-    partial(
-        DeepAREstimator,
-        distr_output=PiecewiseLinearOutput(8),
-    ),
+    # DeepAREstimator,
+    # partial(
+    #     DeepAREstimator,
+    #     distr_output=PiecewiseLinearOutput(8),
+    # ),
 ]
 
 
@@ -60,6 +64,7 @@ def evaluate(dataset_name, estimator):
         trainer=Trainer(
             epochs=epochs,
             num_batches_per_epoch=num_batches_per_epoch,
+            ctx="gpu",
         ),
     )
Details about my setup:
$ pip list | egrep "gluon|mxnet"
gluonts               0.10.6
mxnet-cu112           1.9.1
$ nvidia-smi 
[...]
| NVIDIA-SMI 470.82.00    Driver Version: 470.82.00    CUDA Version: 11.4     |
[...]
$ lsb_release -d
Description:    Ubuntu 20.04.3 LTS

Closing comment (2022-09-25T06:20:42Z): @mharvan potentially related to #2264: a workaround there is to set hybridize=False when constructing the Trainer