Cleanup more things, only load grad_p if necessary