Dependencies: specify only in .gemspec

Deduplicate all the things