It’s not a big deal if your Ledger is stolen unless the thief also has your pin and/or secret seed. This is why it is important to keep both your pin and secret seed hidden in a secure place. If your Ledger is stolen you can purchase another one and restore you wallet. You can use your secret seed to restore it.