There is a possibility that your battery will get damaged faster and won't last long if you keep on over changing the battery. But doing it a couple of times by mistake is fine and your battery performance shouldn't be affected.