Use gem_dir instead of duplicating it

See comments by @voxik in #744