Only update the path when we need to, and only add/remove once with just what we need.