Refactored #569

Removed _without_ method and ! because I dislike them

The cache is now cleared when Gem::Specification::reset is called for
safety.  This is not called in normal operation.

For safety, ensure the file referenced exists before using the cache.  I
may be too paranoid about this, though.