Only rescue LoadError for linguist

Currently this also catches a LoadError for rugged, a dependency Linguist
added but didn't add to its gemspec.

https://github.com/github/linguist/pull/1600
https://github.com/github/linguist/pull/1601