Repository: randombit/botan
Pull request #3651 (2023-07-31 08:15:18)

TLS Anvil Server Tests in Nightly CI
(Refers to #3444)
This adds [TLS Anvil](https://github.com/tls-attacker/TLS-Anvil) server tests to Botan's nightly CI. Unfortunately, a current bug of the origin TLS-Anvil repo required me to create a [fork](https://github.com/FAlbertDev/TLS-Anvil) and work around it (I already created an issue for the TLS-Anvil repo). Until the bug is fixed, the CI script uses my fork. 
The job's workflow is the following:
1) Build Botan and TLS-Anvil
2) Run the `script src/scripts/ci/ci_tlsanvil_test.py` to setup and start a botan tls_server and the TLS-Anvil process. The TLS-Anvil process tests the server by playing a client  (repeatedly) connecting to the server. TLS-Anvil creates a TestSuiteResults folder for storing the results with additional test information.
3) The TestSuiteResults folder and the TLS-Anvil log files are uploaded as an artifact, so we can analyze them if necessary.
4) The script `src/scripts/ci/ci_tlsanvil_check.py` checks if the tests are successful, i.e., if the results in TestSuiteResults meet our expectations. Currently, some tests fail. I will look into these together with @reneme after his holidays. Until then, I allowed the respective tests to fail.

An example workflow execution with failing tests (without deactivating the currently failing tests) is [here](https://github.com/FAlbertDev/botan/actions/runs/5690338681/job/15423485761). When allowing the tests to fail, the workflow looks like [this](https://github.com/FAlbertDev/botan/actions/runs/5692102195/job/15428489867). Since the tests take around 1.5h, they are only executed nightly. 
I also plan to add the TLS-Anvil client tests in a future PR. 

coveralls: 
[![Coverage Status](https://coveralls.io/builds/61669732/badge)](https://coveralls.io/builds/61669732)

coverage: 91.696% (+0.006%) from 91.69% when pulling **85f2adde4bfb50daf41ed5715207657def67d426 on FAlbertDev:ci/tls-anvil** into **3c7229cfee09d3355b7217590e65522059ad246a on randombit:master**.

FAlbertDev: Thanks for your review, @randombit! I addressed your suggestions in ba484ec :)
FAlbertDev: My current state for this PR: 
I'm currently working on the usage of the `tls_http_server` instead of `tls_server` to speed up the tests by parallelization. Some test results are currently inconsistent between both CLI modules (probably because of a small issue with the tls_http_server). I'll investigate this and will probably open another PR addressing this issue after understanding the problem better.
Also, I need to implement your suggestions, @reneme, and redirect the checkout of TLS-Anvil to the origin repo, which is fixed now.
randombit: LGTM. Only remaining comment from me would be please squash before merging.

Patch:
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
index 307787575..75400f4b1 100644
--- a/.github/workflows/nightly.yml
+++ b/.github/workflows/nightly.yml
@@ -1,5 +1,6 @@
 
 # (C) 2023 Jack Lloyd
+# (C) 2023 Fabian Albert, Rohde & Schwarz Cybersecurity
 #
 # Botan is released under the Simplified BSD License (see license.txt)
 
@@ -54,3 +55,35 @@ jobs:
 
       - name: Valgrind Checks
         run: python3 ./src/scripts/ci_build.py --cc=gcc --make-tool=make valgrind-full
+
+  tls_anvil_server_test:
+    name: "TLS-Anvil - Botan Server Test"
+
+    runs-on: ubuntu-22.04
+
+    steps:
+      - name: Fetch Botan Repository
+        uses: actions/checkout@v3
+
+      - name: Setup Build Agent
+        uses: ./.github/actions/setup-build-agent
+        with:
+          target: tlsanvil
+          cache-key: linux-x86_64-tlsanvil
+
+      - name: Build and Test Botan Server with TLS-Anvil
+        run: >
+          python3 ./src/scripts/ci/ci_tlsanvil_test.py
+          --botan-dir .
+          --test-target server
+          --parallel $(nproc)
+
+      - uses: actions/upload-artifact@v3
+        with:
+          name: tls-anvil-server-test-results
+          path: |
+            ./TestSuiteResults/
+            ./logs/
+
+      - name: Check TLS-Anvil Test Results
+        run: python3 ./src/scripts/ci/ci_tlsanvil_check.py --verbose ./TestSuiteResults
diff --git a/src/scripts/ci/ci_tlsanvil_check.py b/src/scripts/ci/ci_tlsanvil_check.py
new file mode 100644
index 000000000..7aeb7c7ac
--- /dev/null
+++ b/src/scripts/ci/ci_tlsanvil_check.py
@@ -0,0 +1,200 @@
+# Parses a TLS-Anvil results directory. Returns 0 iff all results are expected.
+#
+# (C) 2023 Jack Lloyd
+# (C) 2023 Fabian Albert, Rohde & Schwarz Cybersecurity
+#
+# Botan is released under the Simplified BSD License (see license.txt)
+import sys
+import argparse
+import os
+import json
+import logging
+
+
+result_level = {
+    "STRICTLY_SUCCEEDED": 0,
+    "CONCEPTUALLY_SUCCEEDED": 1,
+    "PARTIALLY_FAILED": 2,
+    "FULLY_FAILED": 3,
+}
+
+
+def expected_result_for(method_id: str):
+    """ Get the expected result for a given test id """
+    # TODO: Analyze failing tests and document if/why they are allowed to fail
+    allowed_to_conceptually_succeed = {
+        "both.tls13.rfc8446.RecordProtocol.sendEncryptedAppRecordWithNoNonZeroOctet",
+        "server.tls13.rfc8446.PreSharedKey.isLastButDuplicatedExtension",
+        "server.tls12.rfc7919.FfDheShare.abortsWhenGroupsDontOverlap",
+        "server.tls12.rfc5246.TLSRecordProtocol.sendNotDefinedRecordTypesWithCCSAndFinished",
+        "both.tls13.rfc8446.RecordProtocol.sendEncryptedHandshakeRecordWithNoNonZeroOctet"
+    }
+
+    allowed_to_partially_fail = {
+        "server.tls12.statemachine.StateMachine.earlyChangeCipherSpec",
+        "server.tls12.rfc7568.DoNotUseSSLVersion30.sendClientHelloVersion0300RecordVersion"
+    }
+
+    allowed_to_fully_fail = {
+        "both.tls13.rfc8446.KeyUpdate.respondsWithValidKeyUpdate",
+        "server.tls13.rfc8446.ClientHello.invalidLegacyVersion_ssl3",
+        "server.tls13.rfc8446.ClientHello.invalidLegacyVersion_ssl30",
+        "server.tls13.rfc8446.RecordLayer.zeroLengthRecord_Finished",
+        "server.tls13.rfc8446.KeyShare.abortsWhenSharedSecretIsZero",
+        "server.tls12.rfc8422.TLSExtensionForECC.rejectsInvalidCurvePoints",
+        "server.tls12.rfc5246.ClientHello.leaveOutExtensions",
+        "server.tls12.rfc5246.E1CompatibilityWithTLS10_11andSSL30.acceptAnyRecordVersionNumber",
+        "both.tls13.rfc8446.KeyUpdate.appDataUnderNewKeysSucceeds"
+    }
+
+    if method_id in allowed_to_fully_fail:
+        return result_level["FULLY_FAILED"]
+
+    if method_id in allowed_to_partially_fail:
+        return result_level["PARTIALLY_FAILED"]
+
+    if method_id in allowed_to_conceptually_succeed:
+        return result_level["CONCEPTUALLY_SUCCEEDED"]
+
+    return result_level["STRICTLY_SUCCEEDED"]
+
+
+def test_result_valid(method_id: str, result: str):
+    """
+    Return True iff the result is valid for the method.
+    """
+    if result == "DISABLED":
+        return True
+
+    expected_res = expected_result_for(method_id)
+    if result_level[result] < expected_res:
+        logging.warning("Warning: Test result better than expected for '%s'. Consider tighten the expectation.", method_id)
+
+    return result_level[result] <= expected_result_for(method_id)
+
+
+def failing_test_info(json_data, method_id) -> str:
+    """ Print debug information about a failing test """
+    info_str = ""
+    try:
+        method_class, method_name = method_id.rsplit('.', 1)
+        info = [f"Error: {method_id} - Unexpected result '{json_data['Result']}'"]
+        info += [""]
+        info += [f"Class Name: 'de.rub.nds.tlstest.suite.tests.{method_class}'"]
+        info += [f"Method Name: '{method_name}'"]
+        info += [""]
+        if json_data['TestMethod']['RFC'] is not None:
+            info += [ f"RFC {json_data['TestMethod']['RFC']['number']}, Section {json_data['TestMethod']['RFC']['Section']}:"]
+        else:
+            info += ["Custom Test Case:"]
+        info += [f"{json_data['TestMethod']['Description']}"]
+        info += [""]
+
+        info += [f"Result: {json_data['Result']} (expected {list(result_level.keys())[list(result_level.values()).index(expected_result_for(method_id))]})"]
+        if json_data['DisabledReason'] is not None:
+            info += [f"Disabled Reason: {json_data['DisabledReason']}"]
+
+
+        additional_res_info = list({state["AdditionalResultInformation"] for state in json_data['States'] if state["AdditionalResultInformation"] != ""})
+        additional_test_info = list({state["AdditionalTestInformation"] for state in json_data['States'] if state["AdditionalTestInformation"] != ""})
+        state_result = [{state["Result"] for state in json_data['States']}]
+
+        if len(state_result) > 1 or len(additional_res_info) > 1 or len(additional_test_info) > 1:
+            info += ["Different results for different states. See test results artifact for more information."]
+
+        if len(additional_res_info) == 1:
+            info += ["", f"Additional Result Info: {additional_res_info[0]}"]
+
+        if len(additional_test_info) == 1:
+            info += ["", f"Additional Test Info: {additional_test_info[0]}"]
+        info += [""]
+
+        info_str = "\n".join(info)
+
+        # Color in red
+        info_str = "\n".join([f"\033[0;31m{line}\033[0m" for line in info_str.split("\n")])
+
+        # In GitHub Actions logging group
+        info_str = f"::group::{info_str}\n::endgroup::"
+
+    except KeyError:
+        logging.warning("Cannot process test info.")
+        info_str = ""
+
+    return info_str
+
+
+def process_results_container(results_container_path: str):
+    """
+    Given a path, process the respective results container .json file.
+    Returns True, iff the results of the container are expected.
+    """
+    success = False
+    with open(results_container_path, "r", encoding="utf-8") as results_container_file:
+        try:
+            json_data = json.load(results_container_file)
+            method_id = ".".join(
+                [
+                    json_data["TestMethod"]["ClassName"],
+                    json_data["TestMethod"]["MethodName"],
+                ]
+            ).removeprefix("de.rub.nds.tlstest.suite.tests.")
+            result = json_data["Result"]
+            is_valid = test_result_valid(method_id, result)
+            if is_valid:
+                logging.debug("%s: '%s' -> ok", method_id, result)
+                success = True
+            else:
+                # Print a GitHub logging group in red
+                logging.error(failing_test_info(json_data, method_id))
+
+        except KeyError:
+            logging.error("Json file '%s' has missing entries.", results_container_path)
+
+    return success
+
+
+def main(args=None):
+    """Parse args and check all result container files"""
+    if args is None:
+        args = sys.argv[1:]
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--verbose", action="store_true", default=False)
+    parser.add_argument("results-dir", help="directory of TLS-Anvil test results")
+
+    args = vars(parser.parse_args(args))
+
+    logging.basicConfig(
+        level=(logging.DEBUG if args["verbose"] else logging.INFO),
+        format="%(message)s",
+    )
+
+    results_dir = args["results-dir"]
+
+    if not os.access(results_dir, os.X_OK):
+        raise FileNotFoundError("Unable to read TLS-Anvil results dir")
+
+    failed_methods_count = 0
+    total_methods_count = 0
+    for root, _, files in os.walk(results_dir):
+        for file in files:
+            if file == "_containerResult.json":
+                abs_path = os.path.abspath(os.path.join(root, file))
+                total_methods_count += 1
+                if not process_results_container(abs_path):
+                    failed_methods_count += 1
+
+    logging.info(
+        "(%i/%i) test methods successful.",
+        total_methods_count - failed_methods_count,
+        total_methods_count,
+    )
+    total_success = failed_methods_count == 0
+    logging.info("Total result: %s", "Success." if total_success else "Failed.")
+
+    return int(not total_success)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/src/scripts/ci/setup_gh_actions.sh b/src/scripts/ci/setup_gh_actions.sh
index ead15db6a..dea492242 100755
--- a/src/scripts/ci/setup_gh_actions.sh
+++ b/src/scripts/ci/setup_gh_actions.sh
@@ -23,7 +23,7 @@ if type -p "apt-get"; then
     if [ "$TARGET" = "valgrind" ] || [ "$TARGET" = "valgrind-full" ]; then
         sudo apt-get -qq install valgrind
 
-    elif [ "$TARGET" = "shared" ] || [ "$TARGET" = "examples" ] ; then
+    elif [ "$TARGET" = "shared" ] || [ "$TARGET" = "examples" ] || [ "$TARGET" = "tlsanvil" ] ; then
         sudo apt-get -qq install libboost-dev
 
     elif [ "$TARGET" = "clang" ]; then


Test patch:
diff --git a/src/scripts/ci/ci_tlsanvil_test.py b/src/scripts/ci/ci_tlsanvil_test.py
new file mode 100644
index 000000000..c633b32d4
--- /dev/null
+++ b/src/scripts/ci/ci_tlsanvil_test.py
@@ -0,0 +1,139 @@
+# Script to run inside the CI container to test the botan
+# TLS client/server with TLS-Anvil
+#
+# (C) 2023 Jack Lloyd
+# (C) 2023 Fabian Albert, Rohde & Schwarz Cybersecurity
+#
+# Botan is released under the Simplified BSD License (see license.txt)
+import sys
+import argparse
+import os
+import subprocess
+
+
+class Config:
+    """ Hardcoded configurations for this CI script """
+    tls_anvil_docker_image = "ghcr.io/tls-attacker/tlsanvil"
+    tls_anvil_version_tag = "v1.2.2"
+    key_and_cert_storage_path = "/tmp/"
+    test_suite_results_dest = "."
+    test_suite_results_dir_name = "TestSuiteResults"
+    tmp_key_file_name = "tmp_rsa_key.pem"
+    tmp_cert_file_name = "tmp_rsa_cert.pem"
+    server_dest_ip = "127.0.0.1"
+    server_dest_port = 4433
+    botan_server_log = "./logs/botan_server.log"
+    botan_config_args = ["--compiler-cache=ccache", "--build-targets=static,cli",
+                          "--without-documentation", "--with-boost"]
+
+def group_output(group_title: str, func):
+    """
+    Wraps a function to be called within a GitHub actions group, so that
+    the console output is expandable.
+
+    Returns the wrapped function
+    """
+    def wrapped_func(*args, **kwargs):
+        print(f"::group::{group_title}", flush=True)
+        ret = func(*args, **kwargs)
+        print("\n::endgroup::", flush=True)
+        return ret
+    return wrapped_func
+
+
+def create_cert_and_key(botan_dir_path):
+    """
+    Create a X.509 certificate and associated RSA key at Config.key_and_cert_storage_path
+    using Botan's CLI.
+
+    Returns: (<cert path>, <key path>)
+    """
+
+    key_path = os.path.join(Config.key_and_cert_storage_path, Config.tmp_key_file_name)
+    cert_path = os.path.join(Config.key_and_cert_storage_path, Config.tmp_cert_file_name)
+
+    with open(key_path, 'w', encoding='utf-8') as keyfile:
+        subprocess.run([botan_dir_path, "keygen", "--algo=RSA", "--params=2048"], stdout=keyfile, check=True)
+
+    with open(cert_path, 'w', encoding='utf-8') as certfile:
+        subprocess.run([botan_dir_path, "gen_self_signed", key_path, "localhost"], stdout=certfile, check=True)
+
+    return (cert_path, key_path)
+
+
+def build_botan(botan_dir: str, parallel_jobs: int) -> str:
+    """
+    Configure and build botan.
+
+    Returns the botan executable path
+    """
+    group_output("Configure Botan", subprocess.run)(["python3", "./configure.py"] + Config.botan_config_args, check=True, cwd=botan_dir)
+    group_output("Build Botan with Make", subprocess.run)(["make", f"-j{parallel_jobs}"], check=True, cwd=botan_dir)
+
+    return os.path.join(botan_dir, "botan")
+
+
+def server_test(botan_dir_path: str, parallel: int):
+    """ Test the Botan TLS server """
+    cert_path, key_path = create_cert_and_key(botan_dir_path)
+    docker_img = f"{Config.tls_anvil_docker_image}:{Config.tls_anvil_version_tag}"
+
+    group_output("Pull TLS-Anvil image", subprocess.run)(["docker", "pull", docker_img], check=True)
+
+    tls_anvil_cmd = [
+        "docker", "run",
+        "--network", "host",
+        "-v", f"{Config.test_suite_results_dest}:/output",
+        docker_img,
+        "-strength", "1",
+        "-parallelHandshakes", str(parallel),
+        "-disableTcpDump",
+        "-outputFolder", os.path.join(Config.test_suite_results_dest, Config.test_suite_results_dir_name),
+        "-connectionTimeout", "5000",
+        "server", "-connect", f"{Config.server_dest_ip}:{Config.server_dest_port}"
+    ]
+
+    botan_server_cmd = [
+        botan_dir_path, "tls_http_server", cert_path, key_path, f"--port={Config.server_dest_port}"
+    ]
+
+    os.makedirs(os.path.dirname(Config.botan_server_log), exist_ok=True)
+
+    # Run Botan and test is with TLS-Anvil
+    with open(Config.botan_server_log, 'w', encoding='utf-8') as server_log_file:
+        botan_server_process = subprocess.Popen(botan_server_cmd, stdout=server_log_file, stderr=server_log_file)
+        subprocess.run(tls_anvil_cmd, check=True)
+        botan_server_process.kill()
+
+
+def client_test(botan_dir_path: str, parallel: int):
+    """ Test the Botan TLS server """
+    raise NotImplementedError("Client tests not yet implemented")
+
+
+def main(args=None):
+    if args is None:
+        args = sys.argv[1:]
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--botan-dir", help="Botan base directory", required=True)
+    parser.add_argument("--test-target", help="The TLS side to test", choices=['client', 'server'], required=True)
+    parser.add_argument("--parallel", help="The number of parallel handshakes", type=int, default=1)
+
+    args = vars(parser.parse_args(args))
+
+    if not os.path.isdir(args["botan_dir"]):
+        raise FileNotFoundError(f"Unable to find '{args['botan_dir']}'")
+
+    botan_exe_path = build_botan(args["botan_dir"], args["parallel"])
+
+    if args["test_target"] == "server":
+        server_test(botan_exe_path, args["parallel"])
+    elif args["test_target"] == "client":
+        client_test(botan_exe_path, args["parallel"])
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())