Now that empty keys are dropped, we can always add ssl_* keys

If the connection isn't over SSL, ssl_* keys will be empty and dropped
anyway.