Connecting the switch to both doesn't mean you are shorting the charger, you are just offering a parallel current path. If the charger is still turned on then it will still be running and sending the power around in a circle. Since the device consumes energy to do its job it will be just wasting power and if you are doing it for any length of time you should turn it off.