I'll refer you to http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/ for the background about keeping them in the gemspec.

The groups are more to ensure things like `pry` and `yard` are not picked up by an upstream user's app.

I'm also pretty sure `thor` isn't used directly :-/