Repository: randombit/botan
Pull request #2006 (2019-06-29 11:01:33)

Documentation improvements


Patch:
diff --git a/doc/api_ref/credentials_manager.rst b/doc/api_ref/credentials_manager.rst
index 1f1963f82..8f78970db 100644
--- a/doc/api_ref/credentials_manager.rst
+++ b/doc/api_ref/credentials_manager.rst
@@ -3,21 +3,30 @@ Credentials Manager
 ==================================================
 
 A ``Credentials_Manager`` is a way to abstract how the application
-stores credentials in a way that is usable by protocol
-implementations. Currently the main user is the :doc:`tls`
-implementation.
+stores credentials. The main user is the :doc:`tls` implementation.
 
 .. cpp:class:: Credentials_Manager
 
-   .. cpp:function:: std::vector<X509_Certificate> \
+   .. cpp:function:: std::vector<Certificate_Store*> \
          trusted_certificate_authorities( \
          const std::string& type, \
          const std::string& context)
 
-      Return the list of trusted certificate authorities.
+      Return the list of certificate stores, each of which is assumed
+      to contain (only) trusted certificate authorities. The
+      ``Credentials_Manager`` retains ownership of the
+      Certificate_Store pointers.
+
+      .. note::
+
+         It would have been a better API to return a vector of
+         ``shared_ptr`` here.  This may change in a future major release.
 
       When *type* is "tls-client", *context* will be the hostname of
-      the server, or empty if the hostname is not known.
+      the server, or empty if the hostname is not known. This allows
+      using a different set of certificate stores in different contexts,
+      for example using the system certificate store unless contacting
+      one particular server which uses a cert issued by an internal CA.
 
       When *type* is "tls-server", the *context* will again be the
       hostname of the server, or empty if the client did not send a
@@ -39,6 +48,11 @@ implementation.
       ``acceptable_CAs`` parameter gives a list of CAs the peer trusts.
       This may be empty.
 
+      .. warning::
+         If this function returns a certificate that is not one of the
+         types given in ``cert_key_types`` confusing handshake
+         failures will result.
+
    .. cpp:function:: std::vector<X509_Certificate> cert_chain( \
          const std::vector<std::string>& cert_key_types, \
          const std::string& type, \
@@ -79,6 +93,14 @@ SRP Authentication
 ``Credentials_Manager`` contains the hooks used by TLS clients and
 servers for SRP authentication.
 
+.. note::
+
+   Support for TLS-SRP is deprecated, and will be removed in a future
+   major release. When that occurs these APIs will be removed. Prefer
+   instead performing a standard TLS handshake, then perform a PAKE
+   authentication inside of (and cryptographically bound to) the TLS
+   channel.
+
 .. cpp:function:: bool attempt_srp(const std::string& type, \
                                    const std::string& context)
 
@@ -108,8 +130,7 @@ servers for SRP authentication.
 Preshared Keys
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-TLS and some other protocols support the use of pre shared keys for
-authentication.
+TLS supports the use of pre shared keys for authentication.
 
 .. cpp:function:: SymmetricKey psk(const std::string& type, \
                                    const std::string& context, \
diff --git a/doc/api_ref/ffi.rst b/doc/api_ref/ffi.rst
index 84f729837..4c9472dd2 100644
--- a/doc/api_ref/ffi.rst
+++ b/doc/api_ref/ffi.rst
@@ -535,15 +535,15 @@ Multiple Precision Integers
 
 .. cpp:function:: int botan_mp_add(botan_mp_t result, botan_mp_t x, botan_mp_t y)
 
-   Add two ``botan_mp_t``s and store the output in ``result``.
+   Add two ``botan_mp_t`` and store the output in ``result``.
 
 .. cpp:function:: int botan_mp_sub(botan_mp_t result, botan_mp_t x, botan_mp_t y)
 
-   Subtract two ``botan_mp_t``s and store the output in ``result``.
+   Subtract two ``botan_mp_t`` and store the output in ``result``.
 
 .. cpp:function:: int botan_mp_mul(botan_mp_t result, botan_mp_t x, botan_mp_t y)
 
-   Multiply two ``botan_mp_t``s and store the output in ``result``.
+   Multiply two ``botan_mp_t`` and store the output in ``result``.
 
 .. cpp:function:: int botan_mp_div(botan_mp_t quotient, botan_mp_t remainder, \
                            botan_mp_t x, botan_mp_t y)
@@ -677,24 +677,43 @@ Public Key Creation, Import and Export
 
 .. cpp:function:: int botan_privkey_create_rsa(botan_privkey_t* key, botan_rng_t rng, size_t n_bits)
 
-.. cpp:function:: int botan_privkey_create_ecdsa(botan_privkey_t* key, botan_rng_t rng, const char* params)
+   Create an RSA key of the given size
 
-.. cpp:function:: int botan_privkey_create_ecdh(botan_privkey_t* key, botan_rng_t rng, const char* params)
+.. cpp:function:: int botan_privkey_create_ecdsa(botan_privkey_t* key, botan_rng_t rng, const char* curve)
+
+   Create a ECDSA key of using a named curve
+
+.. cpp:function:: int botan_privkey_create_ecdh(botan_privkey_t* key, botan_rng_t rng, const char* curve)
+
+   Create a ECDH key of using a named curve
 
 .. cpp:function:: int botan_privkey_create_mceliece(botan_privkey_t* key, botan_rng_t rng, size_t n, size_t t)
 
+   Create a McEliece key using the specified parameters. See
+   :ref:`mceliece` for details on choosing parameters.
+
 .. cpp:function:: int botan_privkey_create_dh(botan_privkey_t* key, botan_rng_t rng, const char* params)
 
+   Create a finite field Diffie-Hellman key using the specified named group, for example
+   "modp/ietf/3072".
+
 .. cpp:function:: int botan_privkey_load(botan_privkey_t* key, botan_rng_t rng, \
                                  const uint8_t bits[], size_t len, \
                                  const char* password)
 
+   Load a private key. If the key is encrypted, ``password`` will be
+   used to attempt decryption.
+
 .. cpp:function:: int botan_privkey_destroy(botan_privkey_t key)
 
+   Destroy the object.
+
 .. cpp:function:: int botan_privkey_export(botan_privkey_t key, \
                                    uint8_t out[], size_t* out_len, \
                                    uint32_t flags)
 
+   Export a public key. If flags is 1 then PEM format is used.
+
 .. cpp:function:: int botan_privkey_export_encrypted(botan_privkey_t key, \
                                              uint8_t out[], size_t* out_len, \
                                              botan_rng_t rng, \
@@ -732,14 +751,21 @@ Public Key Creation, Import and Export
    Encrypt a private key. The PBKDF function runs for the specified number of iterations.
    At least 100,000 is recommended.
 
+.. cpp:function:: int botan_privkey_export_pubkey(botan_pubkey_t* out, botan_privkey_t in)
+
+.. cpp:function:: int botan_privkey_get_field(botan_mp_t output, \
+                                      botan_privkey_t key, \
+                                      const char* field_name)
+
+    Read an algorithm specific field from the private key object, placing it into output.
+    For example "p" or "q" for RSA keys, or "x" for DSA keys or ECC keys.
+
 .. cpp:type:: opaque* botan_pubkey_t
 
    An opaque data type for a public key. Don't mess with it.
 
 .. cpp:function:: int botan_pubkey_load(botan_pubkey_t* key, const uint8_t bits[], size_t len)
 
-.. cpp:function:: int botan_privkey_export_pubkey(botan_pubkey_t* out, botan_privkey_t in)
-
 .. cpp:function:: int botan_pubkey_export(botan_pubkey_t key, uint8_t out[], size_t* out_len, uint32_t flags)
 
 .. cpp:function:: int botan_pubkey_algo_name(botan_pubkey_t key, char out[], size_t* out_len)
@@ -758,14 +784,6 @@ Public Key Creation, Import and Export
     Read an algorithm specific field from the public key object, placing it into output.
     For example "n" or "e" for RSA keys or "p", "q", "g", and "y" for DSA keys.
 
-.. cpp:function:: int botan_privkey_get_field(botan_mp_t output, \
-                                      botan_privkey_t key, \
-                                      const char* field_name)
-
-    Read an algorithm specific field from the private key object, placing it into output.
-    For example "p" or "q" for RSA keys, or "x" for DSA keys or ECC keys.
-
-
 RSA specific functions
 ----------------------------------------
 
diff --git a/doc/api_ref/pubkey.rst b/doc/api_ref/pubkey.rst
index 6f867103f..a8b93b457 100644
--- a/doc/api_ref/pubkey.rst
+++ b/doc/api_ref/pubkey.rst
@@ -41,6 +41,17 @@ call
   A constructor that creates a new random RSA private key with a modulus
   of length *bits*.
 
+  RSA key generation is relatively slow, and can take an unpredictable
+  amount of time. Generating a 2048 bit RSA key might take 5 to 10
+  seconds on a slow machine like a Raspberry Pi 2. Even on a fast
+  desktop it might take up to half a second. In a GUI blocking for
+  that long can be a problem. The usual approach is to perform key
+  generation in a new thread, with a animated modal UI element so the
+  user knows the application is still alive. If you wish to provide a
+  progress estimate things get a bit complicated but some library
+  users documented their approach in
+  `a blog post <https://medium.com/nexenio/indicating-progress-of-rsa-key-pair-generation-the-practical-approach-a049ba829dbe>`_.
+
 Algorithms based on the discrete-logarithm problem use what is called a
 *group*; a group can safely be used with many keys, and for some operations,
 like key agreement, the two keys *must* use the same group.  There are
@@ -673,6 +684,28 @@ and a ECDSA signature using EMSA1 with SHA-256. Subsequently the computed signat
     }
 
 
+Ed25519 Variants
+^^^^^^^^^^^^^^^^^^
+
+Most signature schemes in Botan follow a hash-then-sign paradigm. That is, the
+entire message is digested to a fixed length representative using a collision
+resistant hash function, and then the digest is signed. Ed25519 instead signs
+the message directly. This is beneficial, in that the Ed25519 design should
+remain secure even in the (extremely unlikely) event that a collision attack on
+SHA-512 is found. However it means the entire message must be buffered in
+memory, which can be a problem for many applications which might need to sign
+large inputs. To use this variety of Ed25519, use a padding name of "Pure".
+
+Ed25519ph (pre-hashed) instead hashes the message with SHA-512 and then signs
+the digest plus a special prefix specified in RFC 8032. To use it, specify
+padding name "Ed25519ph".
+
+Another variant of pre-hashing is used by GnuPG. There the message is digested
+with any hash function, then the digest is signed. To use it, specify any valid
+hash function. Even if SHA-512 is used, this variant is not compatible with
+Ed25519ph.
+
+For best interop with other systems, prefer "Ed25519ph".
 
 Key Agreement
 ---------------------------------
@@ -763,6 +796,8 @@ applies the key derivation function KDF2(SHA-256) with 256 bit output length to
      }
 
 
+.. _mceliece:
+
 McEliece
 --------------------------
 
@@ -843,8 +878,8 @@ eXtended Merkle Signature Scheme (XMSS)
 
 Botan implements the single tree version of the eXtended Merkle Signature
 Scheme (XMSS) using Winternitz One Time Signatures+ (WOTS+). The implementation
-is based on RFC 8391 "XMSS: eXtended Merkle Signature Scheme", available at
-https://datatracker.ietf.org/doc/rfc8391/.
+is based on `RFC 8391 "XMSS: eXtended Merkle Signature Scheme"
+<https://tools.ietf.org/html/rfc8391>`_.
 
 XMSS uses the Botan interfaces for public key cryptography.
 The following algorithms are implemented:
diff --git a/doc/api_ref/python.rst b/doc/api_ref/python.rst
index 5022d9bf8..35a77684c 100644
--- a/doc/api_ref/python.rst
+++ b/doc/api_ref/python.rst
@@ -14,6 +14,7 @@ The Python binding is based on the `ffi` module of botan and the
 Starting in 2.8, the class names were renamed to match Python standard
 conventions. However aliases are defined which allow older code to
 continue to work; the older names are mentioned as "previously X".
+These aliases will be removed in a future major release.
 
 Versioning
 ----------------------------------------
@@ -40,15 +41,15 @@ Random Number Generators
 
      Previously ``rng``
 
-     Type 'user' also allowed (userspace HKDF RNG seeded from system
+     Type 'user' also allowed (userspace HMAC_DRBG seeded from system
      rng). The system RNG is very cheap to create, as just a single file
      handle or CSP handle is kept open, from first use until shutdown,
      no matter how many 'system' rng instances are created. Thus it is
-     easy to use the RNG in a one-off way, with `botan.rng().get(32)`.
+     easy to use the RNG in a one-off way, with `botan.RandomNumberGenerator().get(32)`.
 
    .. py:method:: get(length)
 
-      Return some bits
+      Return some bytes
 
    .. py:method:: reseed(bits = 256)
 
@@ -62,8 +63,6 @@ Random Number Generators
 
       Add some unpredictable seed data to the RNG
 
-
-
 Hash Functions
 ----------------------------------------
 
@@ -71,7 +70,7 @@ Hash Functions
 
     Previously ``hash_function``
 
-    Algo is a string (eg 'SHA-1', 'SHA-384', 'Skein-512')
+    The ``algo`` param is a string (eg 'SHA-1', 'SHA-384', 'BLAKE2b')
 
     .. py:method:: algo_name()
 
@@ -196,6 +195,7 @@ Ciphers
 
 Bcrypt
 ----------------------------------------
+
 .. py:function:: bcrypt(passwd, rng, work_factor = 10)
 
    Provided the password and an RNG object, returns a bcrypt string
@@ -239,6 +239,7 @@ Scrypt
 
 KDF
 ----------------------------------------
+
 .. py:function:: kdf(algo, secret, out_len, salt)
 
    Performs a key derviation function (such as "HKDF(SHA-384)") over
@@ -252,6 +253,73 @@ Public Key
 
   Previously ``public_key``
 
+  .. py:classmethod:: load(val)
+
+     Load a public key. The value should be a PEM or DER blob.
+
+  .. py:classmethod:: load_rsa(n, e)
+
+     Load an RSA public key giving the modulus and public exponent
+     as integers.
+
+  .. py:classmethod:: load_dsa(p, q, g, y)
+
+     Load an DSA public key giving the parameters and public value
+     as integers.
+
+  .. py:classmethod:: load_dh(p, g, y)
+
+     Load an Diffie-Hellman public key giving the parameters and
+     public value as integers.
+
+  .. py:classmethod:: load_elgamal(p, q, g, y)
+
+     Load an ElGamal public key giving the parameters and
+     public value as integers.
+
+  .. py:classmethod:: load_ecdsa(curve, pub_x, pub_y)
+
+     Load an ECDSA public key giving the curve as a string
+     (like "secp256r1") and the public point as a pair of
+     integers giving the affine coordinates.
+
+  .. py:classmethod:: load_ecdh(curve, pub_x, pub_y)
+
+     Load an ECDH public key giving the curve as a string
+     (like "secp256r1") and the public point as a pair of
+     integers giving the affine coordinates.
+
+  .. py:classmethod:: load_sm2(curve, pub_x, pub_y)
+
+     Load a SM2 public key giving the curve as a string (like
+     "sm2p256v1") and the public point as a pair of integers giving
+     the affine coordinates.
+
+  .. py:method:: check_key(rng_obj, strong=True):
+
+     Test the key for consistency. If ``strong`` is ``True`` then
+     more expensive tests are performed.
+
+  .. py:method:: export(pem=False)
+
+     Exports the public key using the usual X.509 SPKI representation.
+     If ``pem`` is True, the result is a PEM encoded string. Otherwise
+     it is a binary DER value.
+
+  .. py:method:: to_der()
+
+     Like ``self.export(False)``
+
+  .. py:method:: to_pem()
+
+     Like ``self.export(True)``
+
+  .. py:method:: get_field(field_name)
+
+     Return an integer field related to the public key. The valid field names
+     vary depending on the algorithm. For example RSA public modulus can be
+     extracted with ``rsa_key.get_field("n")``.
+
   .. py:method:: fingerprint(hash = 'SHA-256')
 
      Returns a hash of the public key
@@ -265,41 +333,101 @@ Public Key
      Returns the estimated strength of this key against known attacks
      (NFS, Pollard's rho, etc)
 
-  .. py:method:: encoding(pem=False)
-
-     Returns the encoding of the key, PEM if set otherwise DER
-
 Private Key
 ----------------------------------------
 
 .. py:class:: PrivateKey
 
-    Previously ``private_key``
+  Previously ``private_key``
+
+  .. py:classmethod:: create(algo, param, rng)
+
+     Creates a new private key. The parameter type/value depends on
+     the algorithm. For "rsa" is is the size of the key in bits.
+     For "ecdsa" and "ecdh" it is a group name (for instance
+     "secp256r1"). For "ecdh" there is also a special case for group
+     "curve25519" (which is actually a completely distinct key type
+     with a non-standard encoding).
+
+  .. py:classmethod:: load(val, passphrase="")
+
+     Return a private key (DER or PEM formats accepted)
+
+  .. py:classmethod:: load_rsa(p, q, e)
+
+     Return a private RSA key
+
+  .. py:classmethod:: load_dsa(p, q, g, x)
+
+     Return a private DSA key
+
+  .. py:classmethod:: load_dh(p, g, x)
+
+     Return a private DH key
+
+  .. py:classmethod:: load_elgamal(p, q, g, x)
+
+     Return a private ElGamal key
+
+  .. py:classmethod:: load_ecdsa(curve, x)
+
+     Return a private ECDSA key
+
+  .. py:classmethod:: load_ecdh(curve, x)
+
+     Return a private ECDH key
+
+  .. py:classmethod:: load_sm2(curve, x)
+
+     Return a private SM2 key
+
+  .. py:method:: get_public_key()
+
+     Return a public_key object
+
+  .. py:method:: to_pem()
+
+     Return the PEM encoded private key (unencrypted)
+
+  .. py:method:: to_der()
+
+     Return the PEM encoded private key (unencrypted)
+
+  .. py:method:: check_key(rng_obj, strong=True):
+
+     Test the key for consistency. If ``strong`` is ``True`` then
+     more expensive tests are performed.
+
+  .. py:method:: algo_name()
+
+     Returns the algorithm name
 
-    .. py:classmethod:: create(algo, param, rng)
+  .. py:method:: export(pem=False)
 
-       Creates a new private key. The parameter type/value depends on
-       the algorithm. For "rsa" is is the size of the key in bits.
-       For "ecdsa" and "ecdh" it is a group name (for instance
-       "secp256r1"). For "ecdh" there is also a special case for group
-       "curve25519" (which is actually a completely distinct key type
-       with a non-standard encoding).
+     Exports the private key in PKCS8 format. If ``pem`` is True, the
+     result is a PEM encoded string. Otherwise it is a binary DER
+     value. The key will not be encrypted.
 
-    .. py:classmethod:: load(val, passphrase="")
+  .. py:method:: export_encrypted(passphrase, rng, pem=False, msec=300, cipher=None, pbkdf=None)
 
-       Load a private key (DER or PEM formats accepted)
+     Exports the private key in PKCS8 format, encrypted using the
+     provided passphrase. If ``pem`` is True, the result is a PEM
+     encoded string. Otherwise it is a binary DER value.
 
-    .. py:method:: get_public_key()
+  .. py:method:: to_der()
 
-       Return a public_key object
+     Like ``self.export(False)``
 
-    .. py:method:: to_pem()
+  .. py:method:: to_pem()
 
-       Return the PEM encoded private key (unencrypted)
+     Like ``self.export(True)``
 
-    .. py:method:: to_der()
+  .. py:method:: get_field(field_name)
 
-       Return the PEM encoded private key (unencrypted)
+     Return an integer field related to the public key. The valid field names
+     vary depending on the algorithm. For example first RSA secret prime can be
+     extracted with ``rsa_key.get_field("p")``. This function can also be
+     used to extract the public parameters.
 
 Public Key Operations
 ----------------------------------------
diff --git a/doc/api_ref/tls.rst b/doc/api_ref/tls.rst
index f78d59e90..35df6a676 100644
--- a/doc/api_ref/tls.rst
+++ b/doc/api_ref/tls.rst
@@ -409,15 +409,21 @@ The full code for a TLS client using BSD sockets is in `src/cli/tls_client.cpp`
      * TLS client authentication is disabled. See src/lib/tls/credentials_manager.h.
      */
     class Client_Credentials : public Botan::Credentials_Manager
-    {
+       {
        public:
+          Client_Credentials()
+             {
+             // Here we base trust on the system managed trusted CA list
+             m_stores.push_back(new Botan::System_Certificate_Store);
+             }
+
           std::vector<Botan::Certificate_Store*> trusted_certificate_authorities(
              const std::string& type,
              const std::string& context) override
              {
-             // return a list of certificates of CAs we trust for tls server certificates,
-             // e.g., all the certificates in the local directory "cas"
-             return { new Botan::Certificate_Store_In_Memory("cas") };
+             // return a list of certificates of CAs we trust for tls server certificates
+             // ownership of the pointers remains with Credentials_Manager
+             return m_stores;
              }
 
           std::vector<Botan::X509_Certificate> cert_chain(
@@ -439,6 +445,9 @@ The full code for a TLS client using BSD sockets is in `src/cli/tls_client.cpp`
              // associated with the leaf certificate here
              return nullptr;
              }
+
+       private:
+           std::vector<Botan::Certificate_Store*> m_stores;
     };
 
     int main()
diff --git a/doc/api_ref/x509.rst b/doc/api_ref/x509.rst
index fac9ae838..95f43d9d7 100644
--- a/doc/api_ref/x509.rst
+++ b/doc/api_ref/x509.rst
@@ -116,7 +116,7 @@ The major certificate format in use today is X.509v3, used for instance in the
       extension. Also will return true if the extended key usage extension is
       not used in the current certificate.
 
-  .. cpp:function: std::vector<OID> extended_key_usage() const
+  .. cpp:function:: std::vector<OID> extended_key_usage() const
 
       Return the list of extended key usages. May be empty.
 
diff --git a/doc/dev_ref/contents.rst b/doc/dev_ref/contents.rst
index 05adaa331..6505762e1 100644
--- a/doc/dev_ref/contents.rst
+++ b/doc/dev_ref/contents.rst
@@ -9,6 +9,8 @@ contributions to the library
 
    contributing
    configure
+   test_framework
+   continuous_integration
    fuzzing
    release_process
    todo
diff --git a/doc/dev_ref/continuous_integration.rst b/doc/dev_ref/continuous_integration.rst
new file mode 100644
index 000000000..504435c75
--- /dev/null
+++ b/doc/dev_ref/continuous_integration.rst
@@ -0,0 +1,85 @@
+Continuous Integration and Automated Testing
+===============================================
+
+CI Build Script
+----------------
+
+The Travis and AppVeyor builds are orchestrated using a script
+``src/scripts/ci_build.py``. This allows one to easily reproduce the
+build steps of CI on a local machine.
+
+A seperate repo https://github.com/randombit/botan-ci-tools holds
+binaries which are used by the CI.
+
+Travis CI
+-----------
+
+https://travis-ci.org/randombit/botan
+
+This is the primary CI, and tests the Linux, macOS, and iOS builds. Among other
+things it runs tests using valgrind, cross compilation to different
+architectures (currently ARM, PowerPC and MIPS), MinGW build, and the a build
+that produces the coverage report.
+
+The Travis configurations is in ``src/scripts/ci/travis.yml``, which executes a
+setup script ``src/scripts/ci/setup_travis.sh`` to install needed packages.
+Then ``src/scripts/ci_build.py`` is invoked.
+
+AppVeyor
+----------
+
+https://ci.appveyor.com/project/randombit/botan
+
+Runs a build/test cycle using MSVC on Windows. Like Travis it uses
+``src/scripts/ci_build.py``. The AppVeyor setup script is in
+``src/scripts/ci/setup_appveyor.bat``
+
+The AppVeyor build uses ``sccache`` as a compiler cache. Since that is not
+available in the AppVeyor images it takes a precompiled copy checked into the
+``botan-ci-tools`` repo.
+
+Kullo CI
+----------
+
+This was the initial CI system and tests Linux, macOS, Windows, and Android
+builds. Notably this is currently the only CI system Botan uses which has an
+Android build enabled. It does not use ``ci_build.py``. This system is
+maintained by @webmaster128
+
+LGTM
+---------
+
+https://lgtm.com/projects/g/randombit/botan/
+
+An automated linter that is integrated with Github. It automatically checks each
+incoming PR. It also supports custom queries/alerts, which likely would be useful.
+
+Coverity
+---------
+
+https://scan.coverity.com/projects/624
+
+An automated source code scanner. Use of Coverity scanner is rate-limited,
+sometimes it is very slow to produce a new report, and occasionally the service
+goes offline for days or weeks at a time. New reports are kicked off manually by
+rebasing branch ``coverity_scan`` against the most recent master and force
+pushing it.
+
+Sonar
+-------
+
+https://sonarcloud.io/dashboard?id=botan
+
+Sonar scanner is another software quality scanner. Unfortunately a recent update
+of their scanner caused it to take over an hour to produce a report which caused
+Travis CI timeouts, so it has been disabled. It should be re-enabled to run on
+demand in the same way Coverity is.
+
+OSS-Fuzz
+----------
+
+https://github.com/google/oss-fuzz/
+
+OSS-Fuzz is a distributed fuzzer run by Google. Every night, each library fuzzer
+in ``src/fuzzer`` is built and run on many machines with any findings reported
+by email.
diff --git a/doc/dev_ref/fuzzing.rst b/doc/dev_ref/fuzzing.rst
index 519bae4e1..46e60fb53 100644
--- a/doc/dev_ref/fuzzing.rst
+++ b/doc/dev_ref/fuzzing.rst
@@ -88,4 +88,4 @@ have the signature:
 
 ``void fuzz(const uint8_t in[], size_t len)``
 
-After adding your fuzzer, rerun `./configure.py` and build.
+After adding your fuzzer, rerun ``./configure.py`` and build.
diff --git a/doc/dev_ref/mistakes.rst b/doc/dev_ref/mistakes.rst
index 03b2c7905..6ea9ea927 100644
--- a/doc/dev_ref/mistakes.rst
+++ b/doc/dev_ref/mistakes.rst
@@ -1,6 +1,6 @@
 
-Mistakes
-===========
+Mistakes Were Made
+===================
 
 These are mistakes made early on in the project's history which are difficult to
 fix now, but mentioned in the hope they may serve as an example for others.
diff --git a/doc/dev_ref/reading_list.rst b/doc/dev_ref/reading_list.rst
index b39046803..1b27d05d6 100644
--- a/doc/dev_ref/reading_list.rst
+++ b/doc/dev_ref/reading_list.rst
@@ -21,8 +21,8 @@ Implementation Techniques
   for aes_ssse3.
 
 * "Elliptic curves and their implementation" Langley
-   http://www.imperialviolet.org/2010/12/04/ecc.html
-   Describes sparse representations for ECC math
+  http://www.imperialviolet.org/2010/12/04/ecc.html
+  Describes sparse representations for ECC math
 
 Random Number Generation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -50,7 +50,7 @@ Public Key Side Channels
   http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.1028&rep=rep1&type=pdf
 
 * "Resistance against Differential Power Analysis for Elliptic Curve Cryptosystems"
-   Coron http://www.jscoron.fr/publications/dpaecc.pdf
+  Coron http://www.jscoron.fr/publications/dpaecc.pdf
 
 * "Further Results and Considerations on Side Channel Attacks on RSA"
   Klima, Rosa https://eprint.iacr.org/2002/071
diff --git a/doc/dev_ref/release_process.rst b/doc/dev_ref/release_process.rst
index d1418777f..405801c81 100644
--- a/doc/dev_ref/release_process.rst
+++ b/doc/dev_ref/release_process.rst
@@ -1,6 +1,10 @@
 Release Process and Checklist
 ========================================
 
+Releases are done quarterly, normally on the first non-holiday Monday
+of January, April, July and October. A feature freeze goes into effect
+starting 9 days before the release.
+
 .. highlight:: shell
 
 .. note::
@@ -88,6 +92,9 @@ Don't forget to also push tags::
 Build The Windows Installer
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+.. note::
+   We haven't distributed Windows binaries for some time.
+
 On Windows, run ``configure.py`` to setup a build::
 
  $ python ./configure.py --cc=msvc --cpu=$ARCH --distribution-info=unmodified
diff --git a/doc/roadmap.rst b/doc/roadmap.rst
index c5a25c3df..ab73e0e87 100644
--- a/doc/roadmap.rst
+++ b/doc/roadmap.rst
@@ -6,7 +6,7 @@ Near Term Plans
 ----------------------------------------
 
 Here is an outline for the development plans over the next 12-18 months, as of
-April 2019.
+June 2019.
 
 TLS Hardening/Testing
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -20,49 +20,35 @@ Expose TLS at FFI layer
 
 Exposing TLS to C would allow for many new applications to make use of Botan.
 
-Post-Quantum CCA2 Encryption
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+TLS v1.3
+^^^^^^^^^^^^^^^
 
-Which scheme(s) to implement is open to question. HRSS is one possiblity.
-But providing at least one PQ-secure CCA2 encryption scheme would be very
-useful for TLS, PGP, and other protocols.
+A complete implementation of TLS v1.3 is planned. DTLS v1.3 may or may not be
+supported as well.
 
-Password Authenticated Key Exchanges
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Adding support for modern PAKE algorithms (such as SPAKE2+ or OPAQUE),
-including encrypted database backed storage for verifiers.
-
-Elliptic Curve Pairings
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-These are useful in many interesting protocols. Initially BN curves are the main
-target (particularly BN-256 for compatibility with Go's bn256 module) but likely
-we'll also want BLS curves.
+Botan 3.x
+----------------------------------------
 
-And possibly some functionality built on top of pairings, such as identity based
-encryption.
+Botan 3 is currently planned for release in mid 2020. Botan 2 will remain
+supported for several years, to allow time for applications to switch over.
 
-ASN.1 Redesign
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This version will adopt C++17 and use new std types such as string_view,
+optional, and any, along with adopting memory span and guarded integer
+types. All deprecated features/APIs of 2.x (which notably includes TLS v1.0/v1.1
+support) will be removed. Beyond explicitly deprecated functionality, there
+should be no breaking API changes in the transition to 3.x
 
-Design and build a new ASN.1 encoding/decoding library which is easier to use
-and more efficient (fewer memory copies, etc). For at least 2.x this new ASN.1
-code will exist in parallel with the existing library to avoid breaking applications.
+Features currently targetted for Botan 3 include
 
-Longer View (Future Major Release)
-----------------------------------------
+* New post-quantum algorithms: especially a CCA2 secure encryption scheme and a
+  lattice-based signature scheme are of interest.
 
-Eventually (currently estimated for mid 2020), Botan 3.x will be released.
-This schedule allows some substantial time with Botan 2.x and 3.x supported
-simultaneously, to allow for application switch over.
+* Password Authenticated Key Exchanges: one or more modern PAKEs
+  (such as SPAKE2+ or OPAQUE) to replace SRP.
 
-This version will adopt C++17 and use new std types such as string_view,
-optional, and any, along with adopting memory span and guarded integer
-types.
+* Elliptic Curve Pairings: useful in many interesting protocols.
+  BN-256 and BLS12-381 seem the most likely.
 
-In this future 3.x release, all deprecated features/APIs of 2.x (which notably
-includes TLS v1.0/v1.1 support) will be removed. Beyond explicitly deprecated
-functionality, there should be no breaking API changes in the transition to 3.x
+* New ASN.1 library
 
-Botan 3.x will likely be the first version to support TLS 1.3.
+Some of these features may end being backported to Botan 2 as well.
diff --git a/readme.rst b/readme.rst
index f406cede9..b302daa8b 100644
--- a/readme.rst
+++ b/readme.rst
@@ -7,11 +7,13 @@ permissive `Simplified BSD <https://botan.randombit.net/license.txt>`_ license.
 Botan's goal is to be the best option for cryptography in C++ by offering the
 tools necessary to implement a range of practical systems, such as TLS protocol,
 X.509 certificates, modern AEAD ciphers, PKCS#11 and TPM hardware support,
-password hashing, and post quantum crypto schemes.
-See the `documentation <https://botan.randombit.net/manual>`_ for more information.
-
-A Python binding is included, and several other `language bindings
+password hashing, and post quantum crypto schemes. A Python binding is included,
+and several other `language bindings
 <https://github.com/randombit/botan/wiki/Language-Bindings>`_ are available.
+It is used in many `open source and commercial products <https://github.com/randombit/botan/wiki/Users>`_.
+
+See the `documentation <https://botan.randombit.net/handbook>`_ for more
+information about included features.
 
 Development is coordinated on `GitHub <https://github.com/randombit/botan>`_
 and contributions are welcome. If you need help, please open an issue on


Test patch:
diff --git a/doc/dev_ref/test_framework.rst b/doc/dev_ref/test_framework.rst
new file mode 100644
index 000000000..241c51bd1
--- /dev/null
+++ b/doc/dev_ref/test_framework.rst
@@ -0,0 +1,314 @@
+Test Framework
+================
+
+Botan uses a custom-built test framework. Some portions of it are
+quite similar to assertion-based test frameworks such as Catch or
+Gtest, but it also includes many features which are well suited for
+testing cryptographic algorithms.
+
+The intent is that the test framework and the test suite evolve
+symbiotically; as a general rule of thumb if a new function would make
+the implementation of just two distinct tests simpler, it is worth
+adding to the framework on the assumption it will prove useful again.
+Feel free to propose changes to the test system.
+
+When writing a new test, there are three key classes that are used,
+namely ``Test``, ``Test::Result``, and ``Text_Based_Test``. A ``Test``
+(or ``Test_Based_Test``) runs and returns one or more ``Test::Result``.
+
+Namespaces in Test
+-------------------
+
+The test code lives in a distinct namespace (``Botan_Tests``) and all
+code in the tests which calls into the library should use the
+namespace prefix ``Botan::`` rather than a ``using namespace``
+declaration. This makes it easier to see where the test is actually
+invoking the library, and makes it easier to reuse test code for
+applications.
+
+Test Data
+-----------
+
+The test framework is heavily data driven. As of this writing, there
+is about 1 Mib of test code and 17 MiB of test data. For most (though
+certainly not all) tests, it is better to add a data file representing
+the input and outputs, and run the tests over it. Data driven tests
+make adding or editing tests easier, for example by writing scripts
+which produce new test data and output it in the expected format.
+
+Test
+--------
+
+.. cpp:class:: Test
+
+  .. cpp:function:: virtual std::vector<Test::Result> run() = 0
+
+     This is the key function of a ``Test``: it executes and returns a
+     list of results. Almost all other functions on ``Test`` are
+     static functions which just serve as helper functions for ``run``.
+
+  .. cpp:function:: static std::string read_data_file(const std::string& path)
+
+     Return the contents of a data file and return it as a string.
+
+  .. cpp:function:: static std::vector<uint8_t> read_binary_data_file(const std::string& path)
+
+     Return the contents of a data file and return it as a vector of
+     bytes.
+
+  .. cpp:function:: static std::string data_file(const std::string& what)
+
+     An alternative to ``read_data_file`` and ``read_binary_file``,
+     use only as a last result, typically for library APIs which
+     themselves accept a filename rather than a data blob.
+
+  .. cpp:function:: static bool run_long_tests() const
+
+     Returns true if the user gave option ``--run-long-tests``. Use
+     this to gate particularly time-intensive tests.
+
+  .. cpp:function:: static Botan::RandomNumberGenerator& rng()
+
+     Returns a reference to a fast, not cryptographically secure
+     random number generator. It is deterministicly seeded with the
+     seed logged by the test runner, so it is possible to reproduce
+     results in "random" tests.
+
+Tests are registered using the macro ``BOTAN_REGISTER_TEST`` which
+takes 2 arguments: the name of the test and the name of the test class.
+For example given a ``Test`` instance named ``MyTest``, use::
+
+  BOTAN_REGISTER_TEST("mytest", MyTest);
+
+All test names should contain only lowercase letters, numbers, and
+underscore.
+
+Test::Result
+-------------
+
+.. cpp:class:: Test::Result
+
+    A ``Test::Result`` records one or more tests on a particular topic
+    (say "AES-128/CBC" or "ASN.1 date parsing"). Most of the test functions
+    return true or false if the test was successful or not; this allows
+    performing conditional blocks as a result of earlier tests::
+
+      if(result.test_eq("first value", produced, expected))
+         {
+         // further tests that rely on the initial test being correct
+         }
+
+    Only the most commonly used functions on ``Test::Result`` are documented here,
+    see the header ``tests.h`` for more.
+
+    .. cpp:function:: Test::Result(const std::string& who)
+
+       Create a test report on a particular topic. This will be displayed in the
+       test results.
+
+    .. cpp:function:: bool test_success()
+
+       Report a test that was successful.
+
+    .. cpp:function:: bool test_success(const std::string& note)
+
+       Report a test that was successful, including some comment.
+
+    .. cpp:function:: bool test_failure(const std::string& err)
+
+       Report a test failure of some kind. The error string will be logged.
+
+    .. cpp:function:: bool test_failure(const std::string& what, const std::string& error)
+
+       Report a test failure of some kind, with a description of what failed and
+       what the error was.
+
+    .. cpp:function:: void test_failure(const std::string& what, const uint8_t buf[], size_t buf_len)
+
+       Report a test failure due to some particular input, which is provided as
+       arguments. Normally this is only used if the test was using some
+       randomized input which unexpectedly failed, since if the input is
+       hardcoded or from a file it is easier to just reference the test number.
+
+    .. cpp:function:: bool test_eq(const std::string& what, const std::string& produced, const std::string& expected)
+
+       Compare to strings for equality.
+
+    .. cpp:function:: bool test_ne(const std::string& what, const std::string& produced, const std::string& expected)
+
+       Compare to strings for non-equality.
+
+    .. cpp:function:: bool test_eq(const char* producer, const std::string& what, \
+                                   const uint8_t produced[], size_t produced_len, \
+                                   const uint8_t expected[], size_t expected_len)
+
+       Compare two arrays for equality.
+
+    .. cpp:function:: bool test_ne(const char* producer, const std::string& what, \
+                                   const uint8_t produced[], size_t produced_len, \
+                                   const uint8_t expected[], size_t expected_len)
+
+       Compare two arrays for non-equality.
+
+    .. cpp:function:: bool test_eq(const std::string& producer, const std::string& what, \
+                                   const std::vector<uint8_t>& produced, \
+                                   const std::vector<uint8_t>& expected)
+
+       Compare two vectors for equality.
+
+    .. cpp:function:: bool test_ne(const std::string& producer, const std::string& what, \
+                                   const std::vector<uint8_t>& produced, \
+                                   const std::vector<uint8_t>& expected)
+
+       Compare two vectors for non-equality.
+
+    .. cpp:function:: bool confirm(const std::string& what, bool expr)
+
+       Test that some expression evaluates to ``true``.
+
+    .. cpp:function:: template<typename T> bool test_not_null(const std::string& what, T* ptr)
+
+       Verify that the pointer is not null.
+
+    .. cpp:function:: bool test_lt(const std::string& what, size_t produced, size_t expected)
+
+       Test that ``produced`` < ``expected``.
+
+    .. cpp:function:: bool test_lte(const std::string& what, size_t produced, size_t expected)
+
+       Test that ``produced`` <= ``expected``.
+
+    .. cpp:function:: bool test_gt(const std::string& what, size_t produced, size_t expected)
+
+       Test that ``produced`` > ``expected``.
+
+    .. cpp:function:: bool test_gte(const std::string& what, size_t produced, size_t expected)
+
+       Test that ``produced`` >= ``expected``.
+
+    .. cpp:function:: bool test_throws(const std::string& what, std::function<void ()> fn)
+
+       Call a function and verify it throws an exception of some kind.
+
+    .. cpp:function:: bool test_throws(const std::string& what, const std::string& expected, std::function<void ()> fn)
+
+       Call a function and verify it throws an exception of some kind
+       and that the exception message exactly equals ``expected``.
+
+Text_Based_Test
+-----------------
+
+A ``Text_Based_Text`` runs tests that are produced from a text file
+with a particular format which looks somewhat like an INI-file::
+
+  # Comments begin with # and continue to end of line
+  [Header]
+  # Test 1
+  Key1 = Value1
+  Key2 = Value2
+
+  # Test 2
+  Key1 = Value1
+  Key2 = Value2
+
+.. cpp:class:: VarMap
+
+  An object of this type is passed to each invocation of the text-based test.
+  It is used to access the test variables. All access takes a key, which is
+  one of the strings which was passed to the constructor of ``Text_Based_Text``.
+  Accesses are either required (``get_req_foo``), in which case an exception is
+  throwing if the key is not set, or optional (``get_opt_foo``) in which case
+  the test provides a default value which is returned if the key was not set
+  for this particular instance of the test.
+
+  .. cpp:function:: std::vector<uint8_t> get_req_bin(const std::string& key) const
+
+     Return a required binary string. The input is assumed to be hex encoded.
+
+  .. cpp:function:: std::vector<uint8_t> get_opt_bin(const std::string& key) const
+
+     Return an optional binary string. The input is assumed to be hex encoded.
+
+  .. cpp:function:: std::vector<std::vector<uint8_t>> get_req_bin_list(const std::string& key) const
+
+  .. cpp:function:: Botan::BigInt get_req_bn(const std::string& key) const
+
+     Return a required BigInt. The input can be decimal or (with "0x" prefix) hex encoded.
+
+  .. cpp:function:: Botan::BigInt get_opt_bn(const std::string& key, const Botan::BigInt& def_value) const
+
+     Return an optional BigInt. The input can be decimal or (with "0x" prefix) hex encoded.
+
+  .. cpp:function:: std::string get_req_str(const std::string& key) const
+
+     Return a required text string.
+
+  .. cpp:function:: std::string get_opt_str(const std::string& key, const std::string& def_value) const
+
+     Return an optional text string.
+
+  .. cpp:function:: size_t get_req_sz(const std::string& key) const
+
+     Return a required integer. The input should be decimal.
+
+  .. cpp:function:: size_t get_opt_sz(const std::string& key, const size_t def_value) const
+
+     Return an optional integer. The input should be decimal.
+
+.. cpp:class:: Text_Based_Test : public Test
+
+  .. cpp:function:: Text_Based_Test(const std::string& input_file, \
+                    const std::string& required_keys, \
+                    const std::string& optional_keys = "")
+
+     This constructor is
+
+     .. note::
+        The final element of required_keys is the "output key", that is
+        the key which signifies the boundary between one test and the next.
+        When this key is seen, ``run_one_test`` will be invoked. In the
+        test input file, this key must always appear least for any particular
+        test. All the other keys may appear in any order.
+
+  .. cpp:function:: Test::Result run_one_test(const std::string& header, \
+                    const VarMap& vars)
+
+     Runs a single test and returns the result of it. The ``header``
+     parameter gives the value (if any) set in a ``[Header]`` block.
+     This can be useful to distinguish several types of tests within a
+     single file, for example "[Valid]" and "[Invalid]".
+
+  .. cpp:function:: bool clear_between_callbacks() const
+
+     By default this function returns ``false``. If it returns
+     ``true``, then when processing the data in the file, variables
+     are not cleared between tests. This can be useful when several
+     tests all use some common parameters.
+
+Test Runner
+-------------
+
+If you are simply writing a new test there should be no need to modify
+the runner, however it can be useful to be aware of its abilities.
+
+The runner can run tests concurrently across many cores. By default single
+threaded execution is used, but you can use ``--test-threads`` option to
+specify the number of threads to use. If you use ``--test-threads=0`` then
+the runner will probe the number of active CPUs and use that (but limited
+to at most 16). If you want to run across many cores on a large machine,
+explicitly specify a thread count. The speedup is close to linear.
+
+The RNG used in the tests is deterministic, and the seed is logged for each
+execution. You can cause the random sequence to repeat using ``--drbg-seed``
+option.
+
+.. note::
+   Currently the RNG is seeded just once at the start of execution. So you
+   must run the exact same sequence of tests as the original test run in
+   order to get reproducible results.
+
+If you are trying to track down a bug that happens only occasionally, two very
+useful options are ``--test-runs`` and ``--abort-on-first-fail``. The first
+takes an integer and runs the specified test cases that many times. The second
+causes abort to be called on the very first failed test. This is sometimes
+useful when tracing a memory corruption bug.