self.manual_backward() vs. loss.backward() when optimizing manually
According to the manual_backward() documentation, it takes care of scaling when using mixed precision. In that case, is it correct to assume one can simply and safely use loss.backward() during manual optimization if not using mixed precision?
hey @MGheini
It's not just precision but a common hook to support all other strategies like deepspeed/ddp and certain hooks like on_after_backward are called too. So manual_backward is suggested to make sure no-code change is required for eg in case any of the strategies is updated by the user.