question: Gradient Clipping with mix precision in case of NaN loss
context: Greetings. I am getting NaN val loss Cannot log infinite or NaN value to attribute training/val_loss/ with cnnlstm network.I am thinking to use gradient clipping.
  But the doc say gradient clipping should not be used with mixed precision.
  If using mixed precision, the gradient_clip_val does not need to be changed as the gradients are unscaled before applying the clipping function.
  
  Further, i am doing regression, i  dont know what value of gradient clipping should i use?
  Further i checked trainer doc and find that
  track_grad_norm
  (Union[int, float, str]) – -1 no tracking. Otherwise tracks that p-norm. May be set to ‘inf’ infinity-norm. If using Automatic Mixed Precision (AMP), the gradients will be unscaled before logging them.
  
  Can you explain what this mean 'If using Automatic Mixed Precision (AMP), the gradients will be unscaled before logging them'
answer: But the doc say gradient clipping should not be used with mixed precision.
  
  You totally can, that's saying that any scaling applied by 16bit precision training will be undone before clipping the gradients.
  Which means you do not need to worry about changing the gradient clipping value with vs without precision=16
  
  i dont know what value of gradient clipping should i use?
  
  Nobody does :P
  Try some experiments and find out!
  
  'If using Automatic Mixed Precision (AMP), the gradients will be unscaled before logging them'
  
  Same thing as I explained above. It's just a technical detail, you do not need to worry about it