Don't let the gemspec load every lib dependency