Many think it is the diamond. But the truth is that there is a market created by De Beers in South Africa, they only release a certain amount each year, thus making them valuable. If they were to release all the diamonds in their stock, they would be worthless.
Emeralds are far rarer.
If you ever take the time to look at emeralds you will see that they are always flawed with air spots, pieces of other rock and minerals and generally cloudy. Most emeralds are found in South America, and the saddest thing is that the drug lords make more money off of cocaine then they do the emeralds, so they lock off the emerald mines with a guard and send their workers to produce drugs instead. This is another contributing factor to the rareness of emeralds.