Learn about the hobby before you jump in. Especially learn as much as you can about the grading of coins. This will save you more trouble than just about anything else. There's a difference between circulated and uncirculated and fine and extra-fine grades, but sometimes only an expert can truly know the difference. It's best to start out with something fairly accessible and not ultra-rare.
Pennies are a good choice for many collectors and the latest boom in coin collecting.
To an untrained eye, a coin may appear to be in better shape than it really is. If you have doubts or want to make absolutely certain that the coins you purchase are really graded properly, consider purchasing coins sold through a reputable dealer or grading house. In other words, look for significant dates and mintages of coins that have and will retain highest value. In any case, always buy the highest dollar coin you can. Assuming you get a fair price for it, you will never regret the purchase.