Handle Gem spec file in a more clean way

Check loaded_specs again, but fall back to manual detected if it isn't
populated. Allow "missing" gemspec files, especially when working with
Gems installed from github by Bundler.