It's when you can't stop thinking about being apart that you should worry. Being with someone shouldn't be a continuous struggle of hoping for the other person to change. If you can't imagine growing older with them as they are right now, it's probably time to throw in the towel.