Repository: Lightning-AI/pytorch-lightning

Issue:
Logging dominating the training time
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

After migrating a simple 1-layer LSTM sequence classifiactino model to lightning, we found that it was ~7 times slower than the original non-lightning version. we are using tensorboard logger
- after examination, we found that the training time was dominated by calls to log_metrics (in training_loop.py)
- setting the options `row_log_interval=1000` and  `log_save_interval=1000` increased the speed by ~6 times

It looks like under certain circumstances, the default logging frequency is too high, causing drastic increases in training times. It was not intuitive to realize that this was the cause of the slowdown: took several hours of investigation.

### To Reproduce

Steps to reproduce the behavior:

1. create a 1-layer LSTM classifier model. 100 neurons for input and hidden dimensions
    - use a tensorboard logger
2. train using batches of 32 sequences of ~50 tokens or so
   - note the speed
4. add the options to trainer `row_log_interval=1000` and  `log_save_interval=1000`
5. notice that the speed increases by 6 times

#### Code sample

### Expected behavior

default logging frequency for training should be low enough to cover common scenarios such as 1-layer lstm, without causing unexpected slowdowns, which are hard to diagnose
- those people who realize they wish to increase logging frequency will know what they want to do, and find the information fairly easily
- those who just want something up and running fairly quickly wont need to run profiling to figure out why their script is very slow

### Environment

Note that in the instructions it says `python collect_env.py`, but the actual command is `python collect_env_details.py`

```
* CUDA:
        - GPU:
                - Tesla T4
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.18.1
        - pyTorch_debug:     False
        - pyTorch_version:   1.4.0
        - pytorch-lightning: 0.7.6
        - tensorboard:       2.2.1
        - tqdm:              4.45.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                -
        - processor:         x86_64
        - python:            3.7.4
        - version:           #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020
```



Discussion:
Hi! thanks for your contribution!, great first issue!
makes sense! should we change the default to 1000?

@PyTorchLightning/core-contributors 
@williamFalcon I'd prefer not to change these. If you change this to 1000 and people are new and only have small datasets you won't have proper logging while people usually have larger datasets then 100 batches. If they don't see any logging, they may feel things are broken and don't look for options.

I think 100 is a sensible default (and for me it's not slowing down training)
I remember using a schedule for logging before (in Keras I think?), so it started with a small value and gradually (linearly or exponentially) ramped up to bigger intervals.

Maybe some form of schedule can make all users happy 😁
Yeah, I've found that in the small lightweight framework I wrote for myself in my personal life, that an exponential schedule works well for me. (I make it plateau at some point; actually, my personal-life one is based on elapsed time, rather than batches. so I'll log after eg 0.01 seconds, then 0.02 seconds, then 0.05 seconds, then 0.1 seconds, and so on, then after the 30 second log, continue logging every 30 seconds).
btw, there is the parity benchmark to verify the speed...
maybe we can improve that parity benchmark to capture the case where @hpasapp observed 7x slower training times :)
We already have one model with LSTM in parity... What other model would you add? 
Is it using a tensorboard logger?
What if auto-change it if we detect a recurrent model in a submodule? any RNN (maybe transformer?)
@williamFalcon I don't think this is a good idea. Automatically changing defaults only leads to confusion
i meant more as a warning not an actual change
we could do this, but probably this won't work if users write their own RNN cells :)
I'm not sure I understand why one would change the settings specifically for RNNs. Are you saying that LSTM models run *faster* than simple linear classifiers? My own experience is that LSTM models run fairly slowly, because each time-step has to run in-order, cannot be parallelized.

My own hypothesis for why this issue hasnt arisen previously is one or both of two possibilities:
1. the tests are not running using a tensorboard logger. tensorboard logger is perhaps not very speedy?
1. many people are relying on `pl` internal profiler, and the internal profiler doesnt include timings for `pl` internals. For example, in this case, when I run the `pl` profiler, it reports:

```
Action                  |  Mean duration (s)    |  Total time (s)
-----------------------------------------------------------------
on_train_start          |  0.00033947           |  0.00033947
on_epoch_start          |  0.00020974           |  0.00020974
get_train_batch         |  0.0025013            |  0.99803
on_batch_start          |  1.5067e-05           |  0.0059965
model_forward           |  0.00089081           |  0.35454
model_backward          |  0.0014778            |  0.58816
on_after_backward       |  2.6948e-06           |  0.0010725
optimizer_step          |  0.0006639            |  0.26423
on_batch_end            |  0.00023203           |  0.092348
on_epoch_end            |  1.5031e-05           |  1.5031e-05
on_train_end            |  0.00032059           |  0.00032059
```

ie, the slowest thing here is `get_train_batch`, taking ~1 second. And total time here is ~2 seconds, if you add up all the numbers. However, the actual total epoch time is ~20 seconds.

If one instruments `pl` training_loop's inner loop over batches, over one epoch, one finds that, for this example, forward-backward were taking ~1.5 seconds, creating batches was taking ~1 second, and logging to tensorboard was taking ~16+1 seconds:

```
fetch data 0.95
run training batch 1.48
update learning rates 0.00
val 0.00
checkpoint 0.75
save logs 1.26
log metrics 16.29
```

are you using truncated backprop?
if so, then training step is being called much more frequently than just per batch... and thus you would log much more frequently.

imagine a batch with 10000 time steps, then you would love every 10 steps of that batch.

So, really, the warning should come when tbptt is used AND we detect a low logging freq
why would BPTT mean training step is called more frequently than per batch? Even if one was doing seq2seq, during training, one can use teacher-forcing maximum likelihood, feeding in the entire sequence in a single call to nn.LSTM forward:

![Screen Shot 2020-06-01 at 10 55 27 AM](https://user-images.githubusercontent.com/20174596/83421644-6c270400-a3f6-11ea-93ed-62d2a55bb04f.png)

But in my case, the output is not even a sequence, since it's a classifier. So, we just send the entire input sequence through the LSTM, in a single call to `forward`, and then pass the final hidden state (or similar) through a Linear classifier. The entire batch runs in a single `forward` call.
(By the way, here is how I measured the timings:
- instrumentation of training_loop.py tight loop over batches: https://gist.github.com/hpasapp/2035c483214cd4b1490f29acabd7679d
- instrumention class: https://gist.github.com/hpasapp/8b18c3fd1240eab42aa01f6fea508e4b
   (just paste this at the top of training_loop.py; basically accumulates timings per-state, then we dump at end of batch)

Resolving patch:
diff --git a/pytorch_lightning/trainer/__init__.py b/pytorch_lightning/trainer/__init__.py
--- a/pytorch_lightning/trainer/__init__.py
+++ b/pytorch_lightning/trainer/__init__.py
@@ -806,7 +806,7 @@ def on_train_end(self, trainer, pl_module):
 Example::
 
     # default used by the Trainer
-    trainer = Trainer(row_log_interval=10)
+    trainer = Trainer(row_log_interval=50)
 
 use_amp:
 
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -106,7 +106,7 @@ def __init__(
         test_percent_check: float = 1.0,
         val_check_interval: float = 1.0,
         log_save_interval: int = 100,
-        row_log_interval: int = 10,
+        row_log_interval: int = 50,
         distributed_backend: Optional[str] = None,
         precision: int = 32,
         print_nan_grads: bool = False,  # backward compatible, todo: remove in v0.9.0


Test patch: