You must have the expertise to identify gems, distinguish between natural and synthetic stones, and be prepared to do a lot of legwork. Some people get a thrill from the hunt and greatly enjoy this method. Your investment concerns need not be restricted to cut stones. Rough gems, mineral specimens and finished jewelry all hold potential for investment. Selling the gems for a profit is the second consideration.
Jewelry stores are a likely source, as are auction houses and online auctions.
Also note that none of these are as simple as calling your broker. You must be prepared to find your own buyers. If you have weak sales skills, you should not invest in gems. Another point to consider is that lower priced gems receive a higher markup than expensive ones. Low priced gems are frequently marked up three to five times.