Only move things when we have to.