Remove gem path heuristic

This failed to account for gemrcs.  Oh well.
Remove defensive copies for performance reasons