Do key validation only when key is provided

`key_id` is optional, the previous code didn't handle the case of `key_id` not specified. This fixes a regression introduced by #4544.