Only clean GEM_ENV/GEM_HOME when actually needed.

See #2915 for more information.