Don't forget the pam config for pam_ssh_agent_auth