HTTPS/testing

Testing puppet changes to ssl terminators (read and sanity check before blindly following this, maybe something's changed since this was written)
Note: puppet agent --disable didn't actually disable puppet on two of the 12 hosts last time we did this, no idea why.
# verify that /usr/local/dsh/node_groups/ssl is in sync with /home/w/conf/pybal/*/https on fenari.
# take one ssl server out of pool:
#* we want to take one out from tampa since it will have minimal impact on traffic
#* go to /home/wikipedia/common/docroot/noc/pybal/pmtpa, edit the https file
#* change one of them from 'enabled' : True to 'enabled' : False
#* wait a minute for the changes to propogate
#* check that there is no traffic going to the host; ssh on and do netstat -tanvp (t is tcp, p gives pid and program name, dunno if we care about that but whatever) and check that there is no traffic from outside our subnets
# turn off puppet on all ssl hosts, this is esams *and* tampa *and* eqiad
#* on fenari dsh -g ssl -cM 'puppet agent --disable'
#* check one of them to be sure it really won't run (ssh over and try puppet agent --onetime --verbose --no-daemonize and make sure it whines)
# on the depooled host re-enable puppet for testing
#* puppet agent --enable
# on the depooled host run puppet to get the changes over, see how nginx is
#* puppetd --test
#* check that nginx is running, check the logs: /var/log/nginx/error.log and /var/log/nginx/access.log and expect nothing much interesting in them
#* check one of the nginx conf files for sanity: /etc/nginx/sites-enabled/wikipedia (it cn be compared to another host)
#* restart nginx on the host
# test some urls on the server
#* you have to run from the server itself
#* you have to have the service ip in the url (so it's intercepted by the listener on the lo port)
#* probably need to install curl on there (don't forget to remove after done)
#* curl -k -I -v -H "Host: en.wikipedia.org" https://208.80.152.201/wiki/
#* curl -k -I -v -H "Host: en.wikipedia.org" https://208.80.152.201/wiki/Main_Page
#* (address stolen from looking at VIP which it listens to in /etc/nginx/sites-enabled/wikipedia
# re-enable and run puppet on a host in a cluster with some traffic, reload nginx, see how things are
# re-enable and run puppet and reload nginx on a couple more boxes, say the rest in $cluster-with-some-traffic one at a time
# If it all looks right enable puppet everywhere
#* on fenari dsh -g ssl -cM 'puppet agent --enable'
#* make sure puppet has run everywhere...(could do via dsh)
#* restart nginx on all hosts dsh -g ssl -M -w '/etc/init.d/nginx reload; sleep 60'
-------
This is what a successful test looks like:
root@ssl4:~# curl -k -I -v -H 'Host: en.wikipedia.org' https://208.80.152.201/wiki/
* About to connect() to 208.80.152.201 port 443 (#0)
* Trying 208.80.152.201... connected
* successfully set certificate verify locations:
* CAfile: none
 CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
* start date: 2012-10-22 00:00:00 GMT
* expire date: 2016-01-20 12:00:00 GMT
* common name: *.wikipedia.org (does not match '208.80.152.201')
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* SSL certificate verify ok.
> HEAD /wiki/ HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: en.wikipedia.org
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: nginx/1.1.19
Server: nginx/1.1.19
< Date: Thu, 09 May 2013 06:06:10 GMT
Date: Thu, 09 May 2013 06:06:10 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Connection: keep-alive
Connection: keep-alive
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Cache-Control: s-maxage=1200, must-revalidate, max-age=0
Cache-Control: s-maxage=1200, must-revalidate, max-age=0
< Vary: Accept-Encoding,X-Forwarded-Proto,Cookie
Vary: Accept-Encoding,X-Forwarded-Proto,Cookie
< X-Vary-Options: Accept-Encoding;list-contains=gzip,X-Forwarded-Proto,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
X-Vary-Options: Accept-Encoding;list-contains=gzip,X-Forwarded-Proto,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
< Last-Modified: Thu, 09 May 2013 06:06:10 GMT
Last-Modified: Thu, 09 May 2013 06:06:10 GMT
< Location: https://en.wikipedia.org/wiki/Main_Page
Location: https://en.wikipedia.org/wiki/Main_Page
< X-Cache: MISS from sq76.wikimedia.org
X-Cache: MISS from sq76.wikimedia.org
< X-Cache-Lookup: HIT from sq76.wikimedia.org:3128
X-Cache-Lookup: HIT from sq76.wikimedia.org:3128
< X-Cache: MISS from sq66.wikimedia.org
X-Cache: MISS from sq66.wikimedia.org
< X-Cache-Lookup: MISS from sq66.wikimedia.org:80
X-Cache-Lookup: MISS from sq66.wikimedia.org:80
< Via: 1.1 sq76.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq66.wikimedia.org:80 (squid/2.7.STABLE9)
Via: 1.1 sq76.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq66.wikimedia.org:80 (squid/2.7.STABLE9)
* no chunk, no close, no size. Assume close to signal end
< 
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
root@ssl4:~# curl -k -I -v -H 'Host: en.wikipedia.org' https://208.80.152.201/wiki/Main_Page
* About to connect() to 208.80.152.201 port 443 (#0)
* Trying 208.80.152.201... connected
* successfully set certificate verify locations:
* CAfile: none
 CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
* start date: 2012-10-22 00:00:00 GMT
* expire date: 2016-01-20 12:00:00 GMT
* common name: *.wikipedia.org (does not match '208.80.152.201')
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* SSL certificate verify ok.
> HEAD /wiki/Main_Page HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: en.wikipedia.org
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.1.19
Server: nginx/1.1.19
< Date: Thu, 09 May 2013 06:09:53 GMT
Date: Thu, 09 May 2013 06:09:53 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Content-Length: 64909
Content-Length: 64909
< Connection: keep-alive
Connection: keep-alive
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Cache-Control: s-maxage=2678400, must-revalidate, max-age=0
Cache-Control: s-maxage=2678400, must-revalidate, max-age=0
< Content-Language: en
Content-Language: en
< Vary: Accept-Encoding,Cookie
Vary: Accept-Encoding,Cookie
< X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
< Last-Modified: Thu, 09 May 2013 05:50:28 GMT
Last-Modified: Thu, 09 May 2013 05:50:28 GMT
< X-Cache: MISS from sq75.wikimedia.org
X-Cache: MISS from sq75.wikimedia.org
< X-Cache-Lookup: MISS from sq75.wikimedia.org:3128
X-Cache-Lookup: MISS from sq75.wikimedia.org:3128
< Age: 1164
Age: 1164
< X-Cache: HIT from sq75.wikimedia.org
X-Cache: HIT from sq75.wikimedia.org
< X-Cache-Lookup: HIT from sq75.wikimedia.org:80
X-Cache-Lookup: HIT from sq75.wikimedia.org:80
< Via: 1.1 sq75.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq75.wikimedia.org:80 (squid/2.7.STABLE9)
Via: 1.1 sq75.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq75.wikimedia.org:80 (squid/2.7.STABLE9)
< 
* Connection #0 to host 208.80.152.201 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):