You can either find them at random when you kill an enemy or you can upgrade your character to get a certain amount of them (depending on which stat you choose to upgrade). Gems are also given as a bonus every time you gain a level.