Don't start agent when connected using SSH.
Only load private keys.