circle ci's push_doc.sh fails at git push step

I am investigating with the recent failures when deploying the build doc from circle ci.

- https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/43060/workflows/59a6e5eb-7bfd-40a4-92b1-a3b255dc028a/jobs/224165

I did an interactive run on circle ci and reproduced the various steps of the `push_doc.sh` script until the failing git push which I ran in verbose mode:

```
[...]
circleci@a8e707caff89:~/scikit-learn.github.io$ export GIT_SSH_COMMAND="ssh -v"
circleci@a8e707caff89:~/scikit-learn.github.io$ git push
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to github.com [140.82.114.4] port 22.
debug1: Connection established.
debug1: identity file /home/circleci/.ssh/id_rsa type -1
debug1: identity file /home/circleci/.ssh/id_rsa-cert type -1
debug1: identity file /home/circleci/.ssh/id_dsa type -1
debug1: identity file /home/circleci/.ssh/id_dsa-cert type -1
debug1: identity file /home/circleci/.ssh/id_ecdsa type -1
debug1: identity file /home/circleci/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/circleci/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/circleci/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/circleci/.ssh/id_ed25519 type -1
debug1: identity file /home/circleci/.ssh/id_ed25519-cert type -1
debug1: identity file /home/circleci/.ssh/id_ed25519_sk type -1
debug1: identity file /home/circleci/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/circleci/.ssh/id_xmss type -1
debug1: identity file /home/circleci/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version babeld-881dd265
debug1: no match: babeld-881dd265
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM
debug1: Host 'github.com' is known and matches the ECDSA host key.
debug1: Found key in /home/circleci/.ssh/known_hosts:2
Warning: Permanently added the ECDSA host key for IP address '140.82.114.4' to the list of known hosts.
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key:  ED25519 SHA256:Lw0UZUyJVFwZPU+QkMs9j2M6o9aNlbMDqKDwEzhM0z0 agent
debug1: Will attempt key:  RSA SHA256:RCJ0Fk3zyjQVSTDC5iLRY7t8qQjk3W1ddKhqJyRyQIc agent
debug1: Will attempt key: /home/circleci/.ssh/id_rsa 
debug1: Will attempt key: /home/circleci/.ssh/id_dsa 
debug1: Will attempt key: /home/circleci/.ssh/id_ecdsa 
debug1: Will attempt key: /home/circleci/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/circleci/.ssh/id_ed25519 
debug1: Will attempt key: /home/circleci/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/circleci/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key:  ED25519 SHA256:Lw0UZUyJVFwZPU+QkMs9j2M6o9aNlbMDqKDwEzhM0z0 agent
debug1: Server accepts key:  ED25519 SHA256:Lw0UZUyJVFwZPU+QkMs9j2M6o9aNlbMDqKDwEzhM0z0 agent
debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.114.4]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_ALL = en_US.UTF-8
debug1: Sending command: git-receive-pack 'scikit-learn/scikit-learn.github.io.git'
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
ERROR: The key you are authenticating with has been marked as read only.
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 2280, received 2400 bytes, in 0.1 seconds
Bytes per second: sent 23717.6, received 24965.8
debug1: Exit status 1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

- `SHA256:Lw0UZUyJVFwZPU+QkMs9j2M6o9aNlbMDqKDwEzhM0z0` is our deploy key for the `scikit-learn/scikit-learn` repo which is useless to push to `scikit-learn/scikit-learn.github.io`. This key has a fingerprint that matches the `/home/circleci/.ssh/id_rsa ` identity file in the circle ci session; 
- `SHA256:RCJ0Fk3zyjQVSTDC5iLRY7t8qQjk3W1ddKhqJyRyQIc` is the key of our @sklearn-ci account that has push rights to `scikit-learn/scikit-learn.github.io` and which is configured in the [circle ci settings](https://app.circleci.com/settings/project/github/scikit-learn/scikit-learn/ssh) the `github.com` hostname.

So the problem seems that the first key is accepted by the github.com server but it has no push rights to `scikit-learn/scikit-learn.github.io.git` and then git fails without attempting to use the second key.

Note that there are no other identity files under `/home/circleci/.ssh/`, in particular, no file that matches `SHA256:RCJ0Fk3zyjQVSTDC5iLRY7t8qQjk3W1ddKhqJyRyQIc` which must be loaded to the ssh-agent by circle ci from another location.

COMMENT:
I manually removed the deploy key (SHA256:Lw0UZUyJVFwZPU+QkMs9j2M6o9aNlbMDqKDwEzhM0z0) from the ssh-agent and make the `git push` and it works in this case.

From what I understand, we ssh with the deploy key first and don't retry with the right SSH key (the sklearn-ci key). I assume that removing the deploy key from CircleCI will solve the problem. Alternatively, we can remove the key from the agent if the deploy key is useful.

@thomasjpfan Do you know if we use the deploy key for anything else?

Labels: Build / CI