Repository: jdb78/pytorch-forecasting

Issue:
RAM memory getting filled on training
- PyTorch-Forecasting version: 0.8.2
- PyTorch version: 1.7.1
- Python version: 3.7.9
- Operating System: Linux Ubuntu 18.04

### Expected behavior

Following the **N-beats tutorial** on [the docs](https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/ar.html), but with my own dataset.

It's a **large dataset** (+1.5M rows), but I've trimmed it to 100k for dev purposes.

### Actual behavior

**RAM gets filled while training epochs**, even when using `gpus = 1`, leading to a Kernel crash. 

I wonder if, **under the hood, either Pytorch Lightning of Pytorch forecasting** make the classic mistake of `loss += loss`, keeping all the parameters in memory, instead of the adviced `loss += loss.item()`.


Discussion:
To narrow down the problem, do you you have the same issue when running with `log_interval=-1`?
Hi,

I am a bit of a newcomer to Pytorch, so can you specify what **method** is this a parameter of? `.fit(log_interval=-1)`?

I've done some googling, but can't seem to find it.

Thanks in advance.
________________________________
From: Jan Beitner <notifications@github.com>
Sent: Sunday, January 24, 2021 6:22:07 PM
To: jdb78/pytorch-forecasting <pytorch-forecasting@noreply.github.com>
Cc: Pablo <pablonavaber@hotmail.com>; Author <author@noreply.github.com>
Subject: Re: [jdb78/pytorch-forecasting] RAM memory getting filled on training (#286)


To narrow down the problem, do you you have the same issue when running with log_interval=-1?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/jdb78/pytorch-forecasting/issues/286#issuecomment-766397896>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHEB2L3RE6ZUCBSDQKRPAWDS3RJL7ANCNFSM4WQTOH7A>.

It is a NBeats __init__ parameter. So use it in the `from_dataset` method
Amazingly, haven't had that problem in a new run of the same notebook today. 

Used `log_interval = -1` when building the net: should it be responsible?

I have one more question: is **Nbeats forecasting** (image below) all the prediction steps at the same time (**multi-step prediction**), or is it doing a **single-step prediction** at every point, and then collecting them for the plot?

![imagen](https://user-images.githubusercontent.com/29891887/105994304-c7091c00-60a7-11eb-8461-397cd209a63e.png)

It is a multi-step architecture.

Question: If you are using `log_interval > 0`, can you reproduce the memory issue? If yes, this is a bug we need to fix.

Resolving patch:
diff --git a/pytorch_forecasting/models/nbeats/__init__.py b/pytorch_forecasting/models/nbeats/__init__.py
--- a/pytorch_forecasting/models/nbeats/__init__.py
+++ b/pytorch_forecasting/models/nbeats/__init__.py
@@ -293,7 +293,7 @@ def to_prediction(y):
             return self.transform_output(dict(prediction=y[[idx]], target_scale=x["target_scale"][[idx]]))[0]
 
         # plot target vs prediction
-        ax[0].plot(time, torch.cat([x["encoder_target"][idx], x["decoder_target"][idx]]).cpu(), label="target")
+        ax[0].plot(time, torch.cat([x["encoder_target"][idx], x["decoder_target"][idx]]).detach().cpu(), label="target")
         ax[0].plot(
             time,
             torch.cat(
@@ -321,11 +321,17 @@ def to_prediction(y):
                 continue
             if title == "trend":
                 ax[1].plot(
-                    time, to_prediction(output[title]).cpu(), label=title.capitalize(), c=next(prop_cycle)["color"]
+                    time,
+                    to_prediction(output[title]).detach().cpu(),
+                    label=title.capitalize(),
+                    c=next(prop_cycle)["color"],
                 )
             else:
                 ax2.plot(
-                    time, to_prediction(output[title]).cpu(), label=title.capitalize(), c=next(prop_cycle)["color"]
+                    time,
+                    to_prediction(output[title]).detach().cpu(),
+                    label=title.capitalize(),
+                    c=next(prop_cycle)["color"],
                 )
         ax[1].set_xlabel("Time")
         ax[1].set_ylabel("Decomposition")
diff --git a/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py b/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py
--- a/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py
+++ b/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py
@@ -739,7 +739,7 @@ def plot_interpretation(self, interpretation: Dict[str, torch.Tensor]) -> Dict[s
 
         # attention
         fig, ax = plt.subplots()
-        attention = interpretation["attention"].cpu()
+        attention = interpretation["attention"].detach().cpu()
         attention = attention / attention.sum(-1).unsqueeze(-1)
         ax.plot(
             np.arange(-self.hparams.max_encoder_length, attention.size(0) - self.hparams.max_encoder_length), attention
@@ -761,13 +761,13 @@ def make_selection_plot(title, values, labels):
             return fig
 
         figs["static_variables"] = make_selection_plot(
-            "Static variables importance", interpretation["static_variables"].cpu(), self.static_variables
+            "Static variables importance", interpretation["static_variables"].detach().cpu(), self.static_variables
         )
         figs["encoder_variables"] = make_selection_plot(
-            "Encoder variables importance", interpretation["encoder_variables"].cpu(), self.encoder_variables
+            "Encoder variables importance", interpretation["encoder_variables"].detach().cpu(), self.encoder_variables
         )
         figs["decoder_variables"] = make_selection_plot(
-            "Decoder variables importance", interpretation["decoder_variables"].cpu(), self.decoder_variables
+            "Decoder variables importance", interpretation["decoder_variables"].detach().cpu(), self.decoder_variables
         )
 
         return figs
@@ -779,7 +779,7 @@ def log_interpretation(self, outputs):
         # extract interpretations
         interpretation = {
             # use padded_stack because decoder length histogram can be of different length
-            name: padded_stack([x["interpretation"][name] for x in outputs], side="right", value=0).sum(0)
+            name: padded_stack([x["interpretation"][name].detach() for x in outputs], side="right", value=0).sum(0)
             for name in outputs[0]["interpretation"].keys()
         }
         # normalize attention with length histogram squared to account for: 1. zeros in attention and
@@ -811,7 +811,12 @@ def log_interpretation(self, outputs):
         # log lengths of encoder/decoder
         for type in ["encoder", "decoder"]:
             fig, ax = plt.subplots()
-            lengths = padded_stack([out["interpretation"][f"{type}_length_histogram"] for out in outputs]).sum(0).cpu()
+            lengths = (
+                padded_stack([out["interpretation"][f"{type}_length_histogram"] for out in outputs])
+                .sum(0)
+                .detach()
+                .cpu()
+            )
             if type == "decoder":
                 start = 1
             else:
@@ -832,5 +837,5 @@ def log_embeddings(self):
         for name, emb in self.input_embeddings.items():
             labels = self.hparams.embedding_labels[name]
             self.logger.experiment.add_embedding(
-                emb.weight.data.cpu(), metadata=labels, tag=name, global_step=self.global_step
+                emb.weight.data.detach().cpu(), metadata=labels, tag=name, global_step=self.global_step
             )


Test patch: