How to hold 'validation_step' until training one epoch finished
Hi.
It seems like validation_step runs simultaneously at the end of training.
As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur.
How can i fix it?
Thanks in advance!
validation_step runs simultaneously at the end of training.

Validation is considered part of the fitting procedure but it never runs concurrent to training.

As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur. How can i fix it?

It's definitely caused by a bug either on your end or ours. Can you try and reproduce it?
You can adapt https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py to do it