Hot-swapping, or replacing the Raspberry Pi's pins while powered on, is generally a bad idea. It's always safer to remove it from the power supply before switching.