Add a few more checks and comments related to ref temp removal