When staking through Ledger, you have full control over your assets even if you delegate them. You keep control of your private keys secured by your hardware device.