Yes, you should be worried because if your social security card is lost or stolen, you are at a greater risk for identity theft.