That's just the nature of a quick bread or a sweet bread. It's okay and you didn't do anything wrong.