question: How to hold 'validation_step' until training one epoch finished
context: Hi.
  It seems like validation_step runs simultaneously at the end of training.
  As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur.
  How can i fix it?
  Thanks in advance!
answer: validation_step runs simultaneously at the end of training.
  
  Validation is considered part of the fitting procedure but it never runs concurrent to training.
  
  As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur. How can i fix it?
  
  It's definitely caused by a bug either on your end or ours. Can you try and reproduce it?
  You can adapt https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py to do it