Cache sorted gem path

This provides a tiny performance improvement