You can get the gems by mining the stones on the way of shrines. Also, you can find them in chests that randomly appear on the map. Or you can buy them with real money.