Yes, legally if you never returned the key (including it being lost in the mail), he can charge you for the cost to change the locks, since he would have incurred that cost due to your negligence (carelessness) in not getting it to him. He would have to sue you to get the money if you choose to not pay him voluntarily; and to win, he'd have to prove (such as by his testimony) that you never sent it back to him, while you could provide testimony that you did mail it and the letter was never returned as undeliverable, so he should have received it. (If he received the mail, then lost the key later, after having gotten it back from you, that is his responsibility, not yours.) The court would then decide who's story is more credible. You have two advantages in this scenario: first, he may not sue for the money (and you are right: he cannot legally withhold your W2), since it may not be economically worthwhile for him to take legally action over the cost of the locks; and second, if he sues you, the burden of proof is on him, which means he must be more  credible/believale than you--if all things are even, the defendant (the person being sued; i.e. you) wins.