Repository: randombit/botan (cpp)

Issue:
Automated Tests for GCC's Stack Scrubbing
After introducing basic support for GCC's stack scrubbing (#4882), I think it would be beneficial to have an automated test that verifies that the scrubbing actually works. Being a compiler feature, this doesn't produce any observable side effect within the programming language abstractions. At least as long as we don't venture into undefined-behavior territory.

I used the opportunity to play with GDB's python API and came up with the prototype below. Essentially, this script sets a breakpoint at `SHA_256::compress_digest` (which got annotated for stack scrubbing in #4882). Whenever this breakpoint is hit, the script takes note of the function's entire stack frame and lets the program continue until after `SHA_256::compress_digest` has returned. It then breaks again and checks that said stack frame is filled with zeros.

The good news: this works quite well (albeit making a few assumptions). The bad news: it is awfully slow when applied to the full unit test harness. Not just because it only works in debug mode, but also since it does this check for _every_ invocation of `SHA_256::compress_digest`. And that doesn't even include any other algorithm that is annotated like that.

Anyway I think its a good prototype towards ensuring that the stack scrubbing feature actually delivers what it promised.

Setup:

```bash
# First configure and build the library
./configure.py --build-targets=static,cli \
               --minimized-build          \
               --enable-modules=sha2_32   \
               --cc-bin=g++-14            \
               --enable-stack-scrubbing   \
               --debug-mode # debug mode is essential!
make cli

# Then run the CLI under gdb supplying the python script listed below
gdb -x strubtest.py --batch --args ./botan hash readme.rst
```

This should not produce any warnings. In contrast to a build that was configured without the `--enable-stack-scrubbing` switch. Then you should see:

```
[...]
Error: Botan::SHA_256::compress_digest didn't get its stack frame scrubbed after usage
Error: Botan::SHA_256::compress_digest didn't get its stack frame scrubbed after usage
Error: Botan::SHA_256::compress_digest didn't get its stack frame scrubbed after usage
[...]
```

Here's the python script:

```python
import gdb

def stack_pointer(frame):
    # Currently this will likely work on x86_64 only.
    # TODO: find a better way to do that!
    return frame.read_register('rsp')

def base_pointer(frame):
    # Currently this will likely work on x86_64 only.
    # TODO: find a better way to do that!
    return frame.read_register('rbp')

class PostStrubLocation(gdb.FinishBreakpoint):
    def __init__(self, caller_frame, target_frame, function_name):
        super(PostStrubLocation, self).__init__(caller_frame, True)
        self.function_name = function_name
        self.payload_stack_memory = (stack_pointer(target_frame),
                                     base_pointer(target_frame) - stack_pointer(target_frame))
        assert not self.is_stackframe_scrubbed()

    def stackframe_memory(self):
        return gdb.selected_inferior().read_memory(*self.payload_stack_memory)

    def is_stackframe_scrubbed(self):
        return all(b'\x00' == b for b in self.stackframe_memory())

    def stop(self):
        if not self.is_stackframe_scrubbed():
            print(f"Error: {self.function_name} didn't get its stack frame scrubbed after usage")

class StrubTarget(gdb.Breakpoint):
    def __init__(self, function_name):
        super(StrubTarget, self).__init__(function_name)
        self.function_name = function_name

    def stop(self):
        target_frame = gdb.newest_frame()
        caller_frame = gdb.newest_frame().older()

        assert target_frame.name() == self.function_name
        
        # __attribute__( strub("internal") ) creates a "virtual wrapper" around
        # the annotated function. This wrapper has the same symbol name as the
        # actual target function. To tell them apart, we simply _assume_ that
        # the wrapper is always at the lower address in the binary. This may
        # change in the future or differ across compiler versions!
        if len(self.locations) > 1:
            wrapper_address = min(loc.address for loc in self.locations)
            if wrapper_address == target_frame.pc():
                return False

        # Set a temporary breakpoint after the "virtual wrapper" (caller_frame)
        # has returned. This is just after the stack scrubbing was performed.
        PostStrubLocation(caller_frame, target_frame, self.function_name)

        # Don't stop for interactive inspection at this location
        return False

# Main function
def main():
    StrubTarget("Botan::SHA_256::compress_digest")
    gdb.execute("run")

# Register the main function to be called when the script is executed
gdb.execute("python main()")
```

Pull request:
Test: GCC's Stack Scrubbing
This introduces a new 'strubbing' CI job that builds the library with `--enable-stack-scrubbing --debug-mode` and then uses the gdb extension outlined in #4922 to exercise the currently annotated functions in SHA-256. Unfortunately these tests must be new CI jobs because the library has to be built in debug mode for this to work properly.

I think it makes sense to have this in the main CI while the annotation is being rolled out to more functions. After this is stable, I'd suggest to push the job into the nightly.

The gist of the new CI job is this:

```
python3 /home/runner/work/botan/botan/source/src/scripts/test_strubbed_symbols.py --botan-cli /home/runner/work/botan/botan/build/botan
  Checking Botan::version_string... fail (expected)
  Checking Botan::SHA_256::compress_digest... ok
  Checking Botan::SHA_256::compress_digest_x86... ok
  Checking Botan::SHA_256::compress_digest_x86_avx2... ok
  Checking Botan::SHA_256::compress_digest_x86_simd... ok
```

Currently, this works on x86_64 only. But it should be fairly simple to extend to other architectures, most notably for armv8 I guess. Essentially, the `stack_pointer()` and `base_pointer()` functions in the gdb extension would need to become aware of the architecture and the test script has to select its test cases accordingly.

Future work: it would be cool if we could somehow check that all strubbed symbols are covered. It seems that one can fetch those out of the binary, but at this point I'm not sure whether this is reliable:

```
nm libbotan-3.a | grep strub.0 | c++filt
```

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0ef6254ad..37161d38b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -256,6 +256,9 @@ jobs:
           - target: no_pcurves
             compiler: gcc
             host_os: ubuntu-24.04
+          - target: strubbing
+            compiler: gcc-14
+            host_os: ubuntu-24.04
 
     runs-on: ${{ matrix.host_os }}
 
diff --git a/src/scripts/ci/gha_linux_packages.py b/src/scripts/ci/gha_linux_packages.py
index 4d3cfe44f..4f33fc804 100755
--- a/src/scripts/ci/gha_linux_packages.py
+++ b/src/scripts/ci/gha_linux_packages.py
@@ -115,6 +115,9 @@ def gha_linux_packages(target, compiler):
         packages.append('lcov')
         packages.append('python3-coverage')
 
+    if target in ['strubbing']:
+        packages.append('gdb')
+
     if target in ['coverage', 'sanitizer', 'clang-tidy']:
         packages.append('softhsm2')
         packages.append('libtspi-dev')     # TPM 1 development library [TODO(Botan4) remove this]
diff --git a/src/scripts/ci_build.py b/src/scripts/ci_build.py
index 8fecd860b..5f723ac57 100755
--- a/src/scripts/ci_build.py
+++ b/src/scripts/ci_build.py
@@ -72,6 +72,7 @@ def known_targets():
         'sde',
         'shared',
         'static',
+        'strubbing',
         'valgrind',
         'valgrind-full',
         'valgrind-ct',
@@ -111,7 +112,7 @@ class LoggingGroup:
 def build_targets(target, target_os):
     if target in ['shared', 'minimized', 'examples', 'limbo'] or target.startswith('policy-'):
         yield 'shared'
-    elif target in ['static', 'fuzzers', 'cross-arm32-baremetal', 'emscripten']:
+    elif target in ['static', 'fuzzers', 'cross-arm32-baremetal', 'emscripten', 'strubbing']:
         yield 'static'
     elif target_os in ['windows']:
         yield 'shared'
@@ -124,7 +125,7 @@ def build_targets(target, target_os):
     if target not in ['examples', 'limbo']:
         yield 'cli'
 
-    if target not in ['examples', 'limbo', 'hybrid-tls13-interop-test']:
+    if target not in ['examples', 'limbo', 'hybrid-tls13-interop-test', 'strubbing']:
         yield 'tests'
 
     if target in ['coverage']:
@@ -267,6 +268,12 @@ def determine_flags(target, target_os, target_cpu, target_cc, cc_bin, ccache,
     if target in ['coverage', 'valgrind', 'valgrind-full', 'valgrind-ct', 'valgrind-ct-full']:
         flags += ['--with-debug-info']
 
+    if target in ['strubbing']:
+        # Stack scrubbing tests are based on scripted gdb runs and won't work on
+        # an optimized build unfortunately.
+        flags += ['--debug-mode']
+        test_cmd = None
+
     if target in ['coverage', 'sanitizer', 'fuzzers']:
         flags += ['--unsafe-terminate-on-asserts', '--enable-modules=tls_null']
 
@@ -304,7 +311,7 @@ def determine_flags(target, target_os, target_cpu, target_cc, cc_bin, ccache,
         # need to find a way to run the wasm-compiled tests w/o a browser
         test_cmd = None
 
-    if target in ['sanitizer'] and target_cc in ['gcc']:
+    if target in ['sanitizer', 'strubbing'] and target_cc in ['gcc']:
         # Stack scrubbing is supported on GCC 14 and newer, only. This is newer
         # than the current default compiler on GHA's Linux image (ubuntu 24.04).
         # The CI setup has to ensure that we are configured to use a recent
@@ -739,6 +746,7 @@ def main(args=None):
 
         pylint_rc = '--rcfile=%s' % (os.path.join(root_dir, 'src/configs/pylint.rc'))
         pylint_flags = [pylint_rc, '--reports=no']
+        pylint_flags += ['--ignored-modules=gdb'] # 'import gdb' is not available outside gdb...
 
         if is_running_in_github_actions():
             pylint_flags += ["--msg-template='::warning file={path},line={line},endLine={end_line}::Pylint ({category}): {msg_id} {msg} ({symbol})'"]
@@ -758,6 +766,7 @@ def main(args=None):
             'src/scripts/website.py',
             'src/scripts/bench.py',
             'src/scripts/test_python.py',
+            'src/scripts/test_strubbed_symbols.py',
             'src/scripts/test_fuzzers.py',
             'src/scripts/test_cli.py',
             'src/scripts/repo_config.py',
@@ -765,6 +774,7 @@ def main(args=None):
             'src/scripts/python_unittests_unix.py',
             'src/scripts/dev_tools/run_clang_format.py',
             'src/scripts/dev_tools/run_clang_tidy.py',
+            'src/scripts/gdb/strubtest.py',
             'src/editors/vscode/scripts/bogo.py',
             'src/editors/vscode/scripts/common.py',
             'src/editors/vscode/scripts/test.py',
@@ -906,6 +916,10 @@ def main(args=None):
                 cmds.append([py_interp, os.path.join(root_dir, 'src/scripts', script)] +
                             args + test_data_arg + [botan_exe])
 
+        if target in ['strubbing']:
+            cmds.append([py_interp, os.path.join(root_dir, 'src/scripts/test_strubbed_symbols.py'),
+                         '--botan-cli', os.path.join(build_dir, 'botan')])
+
         if target in ['hybrid-tls13-interop-test']:
             cmds.append([py_interp, os.path.join(root_dir, 'src/scripts/test_cli.py'),
                          '--run-online-tests', os.path.join(build_dir, 'botan'), 'pqc_hybrid_tests'])


Test patch:
diff --git a/src/scripts/gdb/strubtest.py b/src/scripts/gdb/strubtest.py
new file mode 100644
index 000000000..e2e31ff4b
--- /dev/null
+++ b/src/scripts/gdb/strubtest.py
@@ -0,0 +1,127 @@
+# (C) 2025 Jack Lloyd
+#     2025 René Meusel, Rohde & Schwarz Cybersecurity
+#
+# Botan is released under the Simplified BSD License (see license.txt)
+#
+
+"""
+User-defined GDB CLI command to help in testing GCC's stack scrubbing
+annotations. See also src/scripts/test_strubbed_symbols.py for details.
+
+This can be used in interactive GDB sessions for debugging and development.
+
+   $> gdb -x src/scripts/gdb/strubtest.py ./botan
+   (gdb) strubtest Botan::SHA_256::compress_digest
+   strubtest setup done
+   (gdb) run hash --algo=SHA-256 readme.rst
+   Success: stackframe of Botan::SHA_256::compress_digest contains 10176 zero bytes after invocation
+   ECD8814EDC180601831FDAD7DEDCF24D57F9F002295346E7D558C941AD318F8E readme.rst
+"""
+
+import gdb
+
+def stack_pointer(frame):
+    return frame.read_register('sp')
+
+def frame_pointer(frame):
+    return frame.read_register('fp')
+
+def report_error(error):
+    gdb.write(f"Error: {error}\n", gdb.STDERR)
+
+def report_status(status):
+    gdb.write(f"{status}\n", gdb.STDOUT)
+
+class PostStrubLocation(gdb.FinishBreakpoint):
+    """
+    This (temporary) breakpoint is placed by the StrubTarget at the end of the
+    "virtual wrapper" GCC introduced to scrub the target's stackframe. When hit
+    it validates that the this stackframe indeed contains zero bytes only.
+    """
+
+    def __init__(self, caller_frame, target_frame, function_name):
+        super().__init__(caller_frame, internal=True)
+        self.function_name = function_name
+        self.payload_stack_memory = (stack_pointer(target_frame),
+                                     frame_pointer(target_frame) - stack_pointer(target_frame))
+
+    def stackframe_memory(self):
+        return gdb.selected_inferior().read_memory(*self.payload_stack_memory)
+
+    def stackframe_size(self):
+        return self.payload_stack_memory[1]
+
+    def is_stackframe_scrubbed(self):
+        return all(b'\x00' == b for b in self.stackframe_memory())
+
+    def stop(self):
+        if not self.is_stackframe_scrubbed():
+            report_error(f"{self.function_name} didn't get its stack frame scrubbed after usage")
+        else:
+            report_status(f"Success: stackframe of {self.function_name} contains {self.stackframe_size()} zero bytes after invcoation")
+
+class StrubTarget(gdb.Breakpoint):
+    """
+    This special breakpoint shall be set to a symbol that was marked with
+    GCC's __attribute__(strub("internal")). When hit, it will note the size of
+    its stackframe and set another temporary breakpoint at the end of the
+    "virtual wrapper" GCC introduced, see PostStrubLocation. There, we'll check
+    if the now-invalidated stackframe indeed contains only zero bytes.
+    """
+
+    def __init__(self, function_name):
+        super().__init__(function_name, internal=True)
+        self.function_name = function_name
+
+    def stop(self):
+        target_frame = gdb.newest_frame()
+        caller_frame = gdb.newest_frame().older()
+
+        # __attribute__( strub("internal") ) creates a "virtual wrapper" around
+        # the annotated function. This wrapper has the same symbol name as the
+        # actual target function. To tell them apart, we simply _assume_ that
+        # the wrapper is always at the lower address in the binary. This may
+        # change in the future or differ across compiler versions!
+        if len(self.locations) > 1:
+            wrapper_address = min(loc.address for loc in self.locations)
+            if wrapper_address == target_frame.pc():
+                return False
+
+        # Set a temporary breakpoint after the "virtual wrapper" (caller_frame)
+        # has returned. This is just after the stack scrubbing was performed.
+        PostStrubLocation(caller_frame, target_frame, self.function_name)
+
+        # Don't stop for interactive inspection at this location
+        return False
+
+class StrubTest(gdb.Command):
+    """
+    User-defined gdb command to set up a stack scrubbing (strub) test allowing
+    to check that the invocation of a given symbol gets its stack cleared after
+    returning. Stack scrubbing is a feature in GCC 14 and newer and is enabled
+    for relevant functions using `./configure.py --enable-stack-scrubbing`.
+    """
+
+    def __init__(self):
+        super().__init__("strubtest", gdb.COMMAND_USER)
+
+    def invoke(self, argstring, _):
+        if not argstring:
+            report_error("a target symbol name is required")
+            return
+
+        bp = StrubTarget(argstring)
+        if bp.pending:
+            bp.delete()
+            report_error(f"the provided symbol '{argstring}' does not appear to be available")
+
+        report_status("strubtest setup done")
+
+# Register the custom command with GDB. It may be invoked from GDB's CLI:
+#
+#  strubtest <annotated symbol, e.g. "Botan::SHA_256::compress_digest">
+#
+# Any time a test program (compiled with --enable-stack-scrubbing --debug-mode)
+# invokes this symbol GDB will either print "Success: ..." or "Error: ...",
+# depending on the outcome of the test.
+StrubTest()
diff --git a/src/scripts/test_strubbed_symbols.py b/src/scripts/test_strubbed_symbols.py
new file mode 100755
index 000000000..f8230f86d
--- /dev/null
+++ b/src/scripts/test_strubbed_symbols.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+
+# (C) 2025 Jack Lloyd
+#     2025 René Meusel, Rohde & Schwarz Cybersecurity
+#
+# Botan is released under the Simplified BSD License (see license.txt)
+#
+
+import subprocess
+import os
+import sys
+import argparse
+import tempfile
+
+from textwrap import indent
+
+SCRIPT_LOCATION = os.path.dirname(os.path.abspath(__file__))
+GDB_EXTENSION = os.path.join(SCRIPT_LOCATION, "gdb", "strubtest.py")
+
+class StrubTest:
+    def __init__(self, symbol, inferior_cmdline, masked_cpuid_bits = None, expect_fail = False):
+        self.symbol = symbol
+        self.inferior_cmdline = inferior_cmdline
+        self.masked_cpuid_bits = masked_cpuid_bits
+        self.expect_fail = expect_fail
+
+    @property
+    def gdb_command(self):
+        return ["gdb", "-x", GDB_EXTENSION,
+                       "-ex", f"strubtest {self.symbol}",
+                       "-ex", "run",
+                       "--batch",
+                       "--args", *self.inferior_cmdline]
+
+    @property
+    def rendered_gdb_command(self):
+        return " ".join([token if " " not in token else f"'{token}'" for token in self.gdb_command])
+
+    @property
+    def environment(self):
+        return {"BOTAN_CLEAR_CPUID": ",".join(self.masked_cpuid_bits)} if self.masked_cpuid_bits else {}
+
+    @property
+    def rendered_environment(self):
+        full_env = os.environ.copy()
+        full_env.update(self.environment)
+        return full_env
+
+    def run(self):
+        print(f"Checking {self.symbol}... ", end="")
+        try:
+            proc = subprocess.run(self.gdb_command, capture_output=True, check=False, env=self.rendered_environment)
+        except subprocess.SubprocessError as ex:
+            return self._fail("subprocess failure", exception=ex)
+
+        if proc.returncode != 0:
+            return self._fail("nonzero error code", proc_result=proc)
+
+        if "Error: " in proc.stderr.decode("utf-8"):
+            return self._fail("fail", proc_result=proc)
+
+        if "Success: " in proc.stdout.decode("utf-8"):
+            return self._succeed(proc_result=proc)
+        else:
+            return self._fail("never invoked", proc_result=proc)
+
+    def _print_debug_report(self, proc_result = None, exception = None):
+        print(f"    ran: {self.rendered_gdb_command}")
+
+        if self.environment:
+            print("    Environment:")
+            print(indent("\n".join([f"{k}={v}" for (k,v) in self.environment.items()]), " " * 6))
+        if exception:
+            print("    Exception:")
+            print(indent(str(exception), " " * 6))
+        if proc_result:
+            if proc_result.stdout:
+                print("    stdout:")
+                print(indent(proc_result.stdout.decode("utf-8"), " " * 6))
+            if proc_result.stderr:
+                print("    stderr:")
+                print(indent(proc_result.stderr.decode("utf-8"), " " * 6))
+
+    def _fail(self, errmsg, proc_result = None, exception = None):
+        if self.expect_fail:
+            print(f"{errmsg} (expected)")
+        else:
+            print(errmsg)
+            self._print_debug_report(proc_result=proc_result, exception=exception)
+        return self.expect_fail
+
+    def _succeed(self, proc_result = None):
+        if not self.expect_fail:
+            print("ok")
+        else:
+            print("ok (unexpected)")
+            self._print_debug_report(proc_result=proc_result)
+        return not self.expect_fail
+
+def dummy_file(size = 1024):
+    with tempfile.NamedTemporaryFile(delete=False) as temp_file:
+        temp_file.write(os.urandom(size))
+        return temp_file.name
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--botan-cli', required=True)
+    args = parser.parse_args()
+
+    cli = args.botan_cli
+    myfile = dummy_file()
+
+    tests = [
+        # This is a self-test, it is expected to fail because the version
+        # information is naturally not annotated for stack scrubbing.
+        StrubTest("Botan::version_string", [cli, "version", "--full"], expect_fail=True),
+
+        # Below is a list of all strub-annotated symbols. Note that we currently run
+        # this on x86_64 only. So platform specific symbols on other architectures are
+        # commented out for completeness and easier future extension.
+
+        StrubTest("Botan::SHA_256::compress_digest",          [cli, "hash", "--algo=SHA-256", myfile]),
+        StrubTest("Botan::SHA_256::compress_digest_x86",      [cli, "hash", "--algo=SHA-256", myfile]),
+        #StrubTest("Botan::SHA_256::compress_digest_armv8",   [cli, "hash", "--algo=SHA-256", myfile]),
+        StrubTest("Botan::SHA_256::compress_digest_x86_avx2", [cli, "hash", "--algo=SHA-256", myfile], masked_cpuid_bits=["intel_sha"]),
+        StrubTest("Botan::SHA_256::compress_digest_x86_simd", [cli, "hash", "--algo=SHA-256", myfile], masked_cpuid_bits=["intel_sha", "avx2"]),
+    ]
+
+    results = [test.run() for test in tests]
+    os.remove(myfile)
+
+    return 1 if any(not ok for ok in results) else 0
+
+if __name__ == '__main__':
+    sys.exit(main())