If I am not wrong, when using encrypted private keys, SSH asks for the encryption password in order to be able to decrypt and use the key. That won't work when connecting programmatically, as the user can't be prompted for the pass. When one adds the key to the `ssh-agent`, though, there is no need to ask for the password, as the agent will take care of handling the authentication. That's why (I think) encrypted keys fallback to the agent by default.