Package: https-everywhere-checker
HTTPS Everywhere Rule Checker=============================Author: Ondrej Mikle, CZ.NIC (ondrej.mikle \|at\_sign\| nic.cz)Installation and requirements-----------------------------::pip install https-everywhere-checkerConfiguration-------------Copy ``checker.config.sample`` to ``checker.config`` and change the``rulesdir`` under ``[rulesets]`` to point to a directory with the XMLfiles of HTTPS Everywhere rules (usually the``src/chrome/content/rules`` of locally checked out git tree of HTTPSEverywhere).Running-------Once you have modified the config, run:::check-https-rules checker.configOutput will be written to selected log file, infos/warnings/errorscontain the useful information.Features---------  Attempts to follow Firefox behavior as closely as possible (includingrewriting HTTP redirects according to rules; well except forJavascript and meta-redirects)-  IDN domain support-  Currently two metrics on "distance" of two resources implemented, oneis purely string-based, the other tries to measure "similarity of theshape of DOM tree"-  Multi-threaded scanner-  Support for various "platforms" (e.g. CAcert), i.e. sets of CAcertificate sets which can be switched during following of redirects-  set of used CA certificates can be statically restricted to one CAcertificate set (see ``static_ca_path`` in config file)What errors in rulesets can be detected----------------------------------------  big difference in HTML page structure-  error in ruleset - declared target that no rule rewrites, bad regexps(usually capture groups are wrong), incomplete FQDNs, non-existentdomains-  HTTP 200 in original page, while rewritten page returns 4xx/5xx-  cycle detection in redirects-  transvalid certificates (incomplete chains)-  other invalid certificate detection (self-signed, expired, CNmismatch...)False positives and shortcomings---------------------------------  Some pages deliberately have different HTTP and HTTPS page, some forexample redirect to different page under https-  URLs to scan are naively guessed from target hosts, having test setof URLs in a ruleset would improve it (better coverage)Known bugs----------CURL+NSS can't handle hosts with SNI sharing same IP address~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~PyCURL and NSS incorrectly handle the case when two FQDNs have identicalIP address, use Server Name Indication and try to resume TLS sessionwith the same session ID. Even turning off SSL session cache via setting``pycurl.SSL_SESSIONID_CACHE`` to zero won't help (it's ignored bylibcurl/pycurl for some reason). PyCURL+NSS fail to see that serverdidn't acknowledge SNI in response (see RFC 4366 reference below), thus'Host' header in HTTP and SNI seen by server are different, thus HTTP404.This one issue was especially insidious bug, many thanks to Pavel Janíkfor helping hunt this bug down.Testcase^^^^^^^^See ``curl_test_nss/curl_testcase_nss_sni.py`` script that demonstratesthe bug.Technical details^^^^^^^^^^^^^^^^^PyCURL sends TLS handshake with SNI for the first host. This works.Connection is then closed, but PyCURL+NSS remembers the SSL session ID.It will attempt to use the same session ID when later connecting tosecond host on the same IP.However, the server won't acknowledge what client requested with newSNI, because client attempts to resume during TLS handshake using theincorrect session ID. Thus the session is "resumed" to the first host'sSNI.Side observation: When validation is turned off in PyCURL+NSS, it alsoturns off session resume as a side effect (the code is in curl's nss.c).Workaround^^^^^^^^^^Set config to use SSLv3 instead of default TLSv1 (option ``ssl_version``under ``http`` section).Normative reference^^^^^^^^^^^^^^^^^^^See last four paragraphs of `RFC 4366, section3.1 <https://tools.ietf.org/html/rfc4366#section-3.1>`__. Contrast with`RFC 6066 section 3 <https://tools.ietf.org/html/rfc6066#section-3>`__,last two paragraphs. In TLS 1.2 the logic is reversed - server must notresume such connection and must go through full handshake again.At most 9 capture groups in rule supported~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~This is a workaround for ambiguous rewrites in rules such as:::<rule from="^http://(www\.)?01\.org/" to="https://$101.org/" />The ``$101`` would actually mean 101-st group, so we assume that only first digit after ``$``denotes the group (which is how it seems to work in javascript).May not work under Windows~~~~~~~~~~~~~~~~~~~~~~~~~~According to `PyCURLdocumentation <http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCAPATH>`__,using CAPATH may not work under Windows. I'd guess it's due to openssl's``c_rehash`` utility that creates symlinks to PEM certificates.Hypothetically it could work if the symlinks were replaced by regularfiles with identical names, but haven't tried.Threading bugs and workarounds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~There are some race conditions with Python threads and OpenSSL/GnuTLSthat cause about due to SIGPIPE or SIGSEGV. While libcurl code seems tohave implemented the necessary callbacks, there's a bug somewhere :-)Workaround: set ``fetch_in_subprocess`` under ``http`` section in configto true when using multiple threads for fetching. Using subprocess is onby default.You might have to set PYTHONPATH if working dir is different from codedir with python scripts.If underlying SSL library is NSS, threading looks fine.As a side effect, the CURL+NSS SNI bug does not happen with subprocesses(SSL session ID cache is not kept among process invocations).If pure-threaded version starts eating too much memory (like 1 GB in aminute), turn on the ``fetch_in_subprocess`` option metioned above. Somecombinations of CURL and SSL library versions do that. Spawning separatesubprocesses prevents any caches building up and eating too much memory.Using subprocess hypothetically might cause a deadlock due toinsufficient buffer size when exchanging data through stdin/stdout incase of a large HTML page, but hasn't happened for any of the rules(I've tried to run them on the complete batch of rulesets contained inHTTPS Everywhere Nov 2 2012 commitc343f230a49d960dba90424799c3bacc2325fc94). Though in case deadlockhappens, increase buffer size in ``subprocess.Popen`` invocation in``http_client.py``.Generic bugs/quirks of SSL libraries~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Each of the three possible libraries (OpenSSL, GnuTLS, NSS) hasdifferent set of quirks. GnuTLS seems to be the most strict oneregarding relevant RFCs and will not for instance tolerate certificatechain in wrong order or forgive server not sending ``close_notify``alert.Thus it's entirely possible that while a server chain and SSL/TLShandshake seems OK when using one lib, it may break with the other.Transvalid certificates (transitive closure of root and intermediate certs)---------------------------------------------------------------------------The ``platform_certs/FF_transvalid.tar.bz2`` attempts to simulate commonbrowser behavior of caching intermediate certs. The directory containsFF's builtin certs and all intermediate certs that validate from FF'sbuiltin certs (a transitive closure).The certs above are in a tarball (need to be unpacked and c\_rehash'dfor use).The script is in ``certs_transitive_closure/build_closure.sh`` and israther crude, definitely needs some double-checking of sanity (seecomments inside the script).Quick outline of the script's algorithm:1. IntermediateSet\_0 := {trusted builtin certs from clean install ofFirefox}2. Certs that have basic constraints CA=true or are X509 version 1 areexported from some DB like SSL Observatory3. Iterate over all exported certs, add new unique certificates not yetcontained in IntermediateSet\_n validate against latestIntermediateSet\_n, forming IntermediateSet\_{n+1}4. n += 15. If any certs were added in step 3, goto 3, else endLast IntermediateSet is the closure.History-------0.1.0 (unreleased)++++++++++++++++++* First release on PyPI.