No, it is not necessary to remove the holder when charging the pocket charger. The device can be easily charged by plugging the power supply into the USB port.