Scope the keys to the handshake.

The rest of the connection doesn't need them.