Hm, try specifying each ext in the gemspec.