Fundraising/techops/docs/ssl client authentication

In WMF Fundraising we protect some internal-use websites (e.g. civicrm, dash, superset, grafana) with SSL client certificate authentication. Client Certificates are issued by Fundraising Tech Operations. Each user gets their own certificate to install on their web browser. This page documents steps to get the certificate working on your system.
= Requesting A Certificate =
We issue SSL client certificates during onboarding, as part of civicrm account setup. As with other access requests, start with a task in Phabricator. C-level approval is required, and can be quoted in the task. (See Fundraising Access Request for full details.) For issues with civicrm account setup please follow-up on the Phabricator ticket. For many users we send a password by text message (see below) to your "cell phone of record" so please make sure your contact information is up to date at the beginning of the access request. Wikimedia staff should be on the Wikimedia Contact List, and fundraising contractors should be listed on the Fundraising Contact List
= Certificate Expiration and Renewal =
We issue SSL client certs with a maximum of one-year expiration. FR Tech Operations monitors expiration, and will contact you regarding renewal shortly before your existing cert expires.
= Retrieve Your Certificate =
== Download Via Email + SMS ==
In most cases FR Tech Operations will send your certificate by email and a certificate password separately by text message to your "cell phone of record." Use your email application to save the client_cert.p12 file to your Downloads directory.
== Download Via Shell Account (for users with shell access) ==
If you have shell access to the fundraising cluster, you may receive your certificate and password in your home directory on frpm1002 or frdev1002. In your remote home directory you'll find two files:
 client_cert.p12 -- the new PKS12 certificate to import to your browser
 client_cert.pw -- text file containing a passphrase
Use the scp command or a graphical scp/sftp tool such as FileZilla. For example, to use scp on OSX/Linux to fetch from frdev1002, the command would be:
 mycomputer:~> scp frdev1002:client_cert.* ~/Downloads/
= Installing the Certificate on Your System =
== Chrome/Chromium/Safari on OSX ==
=== Part 1: macOS Keychain Install ===
# Retrieve your certificate
# Open up Keychain Access application
# If we renewed your certificate, delete any old one(s) first
## Click login in the upper left
## Click My Certificates on the lower left
## Look for certificates matching your name
## Click on a cert to check that it is "Issued by: FundraisingCA"
## If so remove the certificate using the delete key.
# Select File at the top and then Import Items
# Click Downloads, select client_cert.p12, and click Open
# When prompted Enter the password for "client_cert.p12" enter the <U>certificate password</u> sent by FR Tech Operations.This is the only time you need the certificate password. Everywhere else what's needed is your Mac login password.
#Right click/control-click on the certificate, select Get Info
##Expand Trust menu
##On the When using this certificate menu, select Always Trust
##Close Get Info box
#(On older OSX releases, there's an option to Trust Always, if it's there select it.)
# Click Ok
# Quit Keychain Access
=== Part 2: Browser Setup ===
# Close Chrome/Chromium/Safari if running
# Open up Chrome/Chromium/Safari and go to https://civicrm.wikimedia.org
# At the Select a certificate prompt, choose your new cert and click OkIf more than one certificate is listed, choose the one that is your name followed by "(FundraisingCA)"
# At the prompt for "login" keychain password enter your Mac login password and select Always Allow
# You may be prompted again for other system components using the keychain, use your Mac login password and click Always Allow if the option is available.
=== Part 3: Why Doesn't It Work?! ===
# If everything else went as described but you get a certificate error, see Help below.
== Firefox on OSX/Linux/Windows ==
# Retrieve your certificate
# Open up Firefox
# Click on the "hamburger" menu on the right side.
# Click on Settings.
# Under Privacy & Security, scroll to Certificates
# Click on View Certificates...
# Select the Your Certificates tab
# If you have a master password for Firefox this prompt may appear, here enter the Firefox password
# If we renewed your certificate, delete any certificates matching your name under the Wikimedia Foundation heading
# Click Import
# Click Downloads, select client_cert.p12, and click Open
# At this prompt, enter the certificate password provided by FR Tech Operations
# Once you have successfully imported your certificate, click OK
# Go to https://civicrm.wikimedia.org
# When this window appears, make sure the certificate matching your name is selected and click OK
== Google Chromium on Linux ==
# Retrieve your certificate
# Open up Chromium
# Go to Settings, scroll to the bottom and click Advanced
# Click on Manage certificates
# Select Your Certificates
# If we renewed your certificate, look for the heading org-Wikimedia Foundation, expand it and delete any certificate matching your name 
# Click Import
# Click Downloads and select client_cert.p12, and click Open.
# At Enter your certificate password enter the certificate password provided by FR Tech Operations
# Click OK
# Go to https://civicrm.wikimedia.org
# At Select a certificate prompt, make sure the certificate matching your name is selected and click OK
= Post-Install Cleanup =
Be sure to delete the files client_cert.p12 and, if present, client_cert.pw. You don't need these once the install is done.
= Help! =
If you see "An authorized SSL client certificate is required to access this server." when attempting to use Civicrm or superset, your SSL client certificate is not working properly.
There are few reasons this can happen. 
If your certificate was previously working, the failure usually happens if the certificate selection is interrupted for some reason. This failed selection result will remain cached in the browser for some period of time which unfortunately varies depending on OS/browser.
=== Clear Browser Cache ===
The quickest way to clear the browser cache is to fully quit the browser ensuring all windows are closed. This should drop the cache and prompt you to select the certificate when you restart and connect to superset/civicrm. Take extra care that you don't accidentally cancel, hit ESC, or navigate away from the certificate selection since those are the main causes we have seen.
===Reboot Machine===
There are occasional cases where the cache won't clear with a restart of the browser but will with a full machine reboot. Keep in mind to save any work before rebooting your machine.
=== Verify Certificate Loaded with Permissions ===
If that doesn't work, it is best to verify that the certificate is properly loaded in the Keychain. That would be using steps similar to the certificate install but instead of deleting the certificate, just verifying that it is loaded correctly and that the trust settings are appropriately set to Always Allow.
If none of these steps resolve your issue, please contact Fundraising Tech Operations by email fr-tech-ops@wikimedia.org or IRC, or by text message if it is an emergency.
== Repeated 'macOS X wants to make changes' Prompt ==
Safari has been known to get confused, see https://discussions.apple.com/thread/4010007 but be aware that the suggested remedy causes macOS to clear passwords stored in Keychain.
== macOS Keeps Asking for the Login Keychain Password ==
See: https://support.apple.com/en-gb/HT201609 (wayback page)
== System Time Is Incorrect ==
If your system clock has drifted really far from reality, you may see certificate-related errors. For macOS see 
https://support.apple.com/en-gb/HT203413
== Safari Appears to Stall After Entering https://civicrm.wikimedia.org ==
Sometimes password prompts for Safari or com.apple.WebKit.Networking show up in surprising places on the screen. If you don't see a prompt try restarting Safari. Remember to click Always Allow when you have the option, and this shouldn't crop up as frequently.