Keyholder

Keyholder is a set of scripts that allow a group of users to use an SSH key without sharing the private key with the members of the group. This is accomplished by running a locked-down instance of ssh-agent, and running ssh-agent-proxy in front of it. The proxy allows trusted users to list the ssh-agent's identities and to send the agent signing requests. Access to a given key is restricted to members of a given Unix group and will be rejected when the user is not a member.
== Overview ==
Administration of Keyholder is done via a shell script, keyholder located under /usr/local/sbin which is typically not in a normal user PATH.
 keyholder -- Manage shared SSH agent
 
 keyholder status
 Lists service status and the fingerprints of all identities
 currently represented by the agent
 
 keyholder add KEY
 Add a private key identity to the agent
 
 keyholder arm
 Add all keys in /etc/keyholder.d
 
 keyholder disarm
 Deletes all identities from the agent
 
 keyholder start/stop/restart
 Start / stop / restart the keyholder service
== Arming keys ==
Keys are stored under /srv/private/modules/secret/secrets/keyholder/ in the Puppet#Private puppet repo.
We use Keyholder for SCAP deploys. Keyholder starts automatically, but a user with root has to load both the mwdeploy and deploy-service identities. This is by design. If the agent is unarmed, an Icinga check will issue the following alert:
PROBLEM - Keyholder SSH agent on tin is CRITICAL: Keyholder is not armed. Run `keyholder arm` to arm it.
To arm the agent, log in to deploy1003, and run keyholder arm. The agent will automatically attempt to load the secret deployment key, which is protected with a passphrase. The passphrases for production are stored in Pwstore, for WMCS OpenStack projects they are usually stored on the project-specific puppetmaster.
=== Production passphrases ===
Identity 	Realm 	Location
mwdeploy 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-service 	production 	pwstore/pw.git/deployment-key-passphrase
eventlogging 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-ci-docroot 	production 	pwstore/pw.git/deployment-key-passphrase
phab-deploy 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-debmonitor 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-librenms 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-puppetboard 	production 	pwstore/pw.git/deployment-key-passphrase
dumpsdeploy 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-rancid 	production 	pwstore/pw.git/network-monitoring-keys-passphrase
analytics-deploy 	production 	pwstore/pw.git/deployment-key-passphrase
deploy_airflow 	production 	pwstore/pw.git/deployment-key-passphrase
zuul 	production 	pwstore/pw.git/deployment-key-passphrase
rancid 	production 	pwstore/pw.git/network-monitoring-keys-passphrase
netbox 	production 	pwstore/pw.git/deployment-key-passphrase
cumin_master 	production 	pwstore/pw.git/cumin-master-key-passphrase
authdns_acmechief 	production 	pwstore/pw.git/authdns-acmechief-key-passphrase
metamonitor 	production 	pwstore/pw.git/metamonitor-key-passphrase
deploy-hiddenparma 	production 	pwstore/pw.git/deployment-key-passphrase
homer 	production 	pwstore/pw.git/homer-key-passphrase
deploy-homer 	production 	pwstore/pw.git/deployment-key-passphrase
deploy-ci-docroot 	production 	pwstore/pw.git/deployment-key-passphrase
apache2modsec 	production 	pwstore/pw.git/apache2modsec-key-passphrase
cumin_openstack_master	production for WMCS	pwstore/pw.git/cumin-openstack-master-key-passphrase
cloud_cumin_master 	production for WMCS	pwstore/pw.git/cloud-cumin-master-key-passphrase
=== WMCS projects passphrases ===
Identity 	Realm 	Location
cumin 	integration 	integration-puppetserver-01.integration.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/cumin.passphrase
cumin 	tools 	'cumin ssh key password' in tools-puppetserver-01.tools.eqiad1.wikimedia.cloud:/srv/git/labs/private/toolforge-secrets.txt
cumin 	beta cluster	deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
mwdeploy 	beta cluster	deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/mwdeploy_rsa.passphrase
deploy-service 	beta cluster	deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/servicedeploy_rsa.passphrase
eventlogging 	beta cluster	passphrase is: 'eventlogging'
dumpsdeploy 	beta cluster	passphrase is (without the quote marks): 'some boring passphrase'
analytics-deploy 	beta cluster	deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:/srv/git/labs/private/files/ssh/tin/analytics_deploy.passphrase
scap 	beta cluster	deployment-puppetserver-1:/srv/git/labs/private/files/ssh/tin/scap.passphrase
wmcs_openstack_instance_puppet_user	cloudinfra 	cloudinfra-internal-puppetmaster-02:/var/lib/git/labs/private/passwords.txt
== Hints ==
You can verify that the agent is armed by running /usr/local/sbin/keyholder status.
SCAP deployers do not have access to the private key or the passphrase, so in case the deployment server is rebooted, SCAP deployments will be blocked until a root arms the agent.
A deployer can then use the proxy to connect to a host:
SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -oIdentitiesOnly=yes -oIdentityFile=/etc/keyholder.d/<somekey> <somehost>
== Generating a key for a new identity or update an existing one ==
Use the following commands on puppetmaster1001 (so it never touches the network or other host before being setup into the private repo):
 puppetmaster1001# cd /root
 puppetmaster1001# ssh-keygen -t ed25519 -b 256 -C "/etc/keyholder.d/<identity>" -f <identity> # check keys look fine
 puppetmaster1001# mv <identity> <identity>.pub /srv/private/modules/secret/secrets/keyholder/
 puppetmaster1001# cd /srv/private/modules/secret/secrets/keyholder/; git add <identity> <identity>.pub; git commit
 
 application_host# run-puppet-agent
 
 deployment_host# run-puppet-agent # note there are 2 deployment hosts, one for each dc as of this writing
 deployment_host# keyholder restart # you can just add, I did a restart because I was updating one
 deployment_host# keyholder arm
If using ED25519 keys is not possible because incompatibilities (e.g. network equipment or access to PSUs), it is possible to failback to generate RSA keys:
 puppetmaster1001# ssh-keygen -t rsa -b 4096 -C "/etc/keyholder.d/<identity>" -f <identity>
==See also==
*Scap3 documentation for ssh access https://doc.wikimedia.org/scap/scap3/ssh-access.html
*Scap3 keyholder key documentation Scap/Scap3 keyholder key