Logging dominating the training time
🐛 Bug
After migrating a simple 1-layer LSTM sequence classifiactino model to lightning, we found that it was ~7 times slower than the original non-lightning version. we are using tensorboard logger

after examination, we found that the training time was dominated by calls to log_metrics (in training_loop.py)
setting the options row_log_interval=1000 and  log_save_interval=1000 increased the speed by ~6 times

It looks like under certain circumstances, the default logging frequency is too high, causing drastic increases in training times. It was not intuitive to realize that this was the cause of the slowdown: took several hours of investigation.
To Reproduce
Steps to reproduce the behavior:

create a 1-layer LSTM classifier model. 100 neurons for input and hidden dimensions

use a tensorboard logger


train using batches of 32 sequences of ~50 tokens or so

note the speed


add the options to trainer row_log_interval=1000 and  log_save_interval=1000
notice that the speed increases by 6 times

Code sample
Expected behavior
default logging frequency for training should be low enough to cover common scenarios such as 1-layer lstm, without causing unexpected slowdowns, which are hard to diagnose

those people who realize they wish to increase logging frequency will know what they want to do, and find the information fairly easily
those who just want something up and running fairly quickly wont need to run profiling to figure out why their script is very slow

Environment
Note that in the instructions it says python collect_env.py, but the actual command is python collect_env_details.py
* CUDA:
        - GPU:
                - Tesla T4
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.18.1
        - pyTorch_debug:     False
        - pyTorch_version:   1.4.0
        - pytorch-lightning: 0.7.6
        - tensorboard:       2.2.1
        - tqdm:              4.45.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                -
        - processor:         x86_64
        - python:            3.7.4
        - version:           #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020