Experts often warn against closing a credit card, especially your oldest one, since it can have a negative impact on your credit score.