question: Can the learning rate find by dp using one more gpu be used in ddp?
context: When using pytorch_lightning.tuner.lr_finder.lr_find, ddp have some error. So i change to dp using 4 gpus. Can the learning rate find by dp used by ddp? They have same gpu numbers.
answer: I think the answer is no.
  DP doesn't change your effective batch size while DDP does (in your case with one node and 4GPUs, the effective batch size is 4 times bigger with DDP). You can find more info about the effective batch size in the "multi-GPU" section of Lightning's documentation here.
  As a consequence of this, you should probably increase your learning rate. Rule of thumbs is to increase it linearly (so by 4) but there is more than just doing that. Have a look at that paper: https://arxiv.org/pdf/1706.02677.pdf