Repository: Gallopsled/pwntools

Issue:
Unstrip ELF files fetched with `libcdb`
The ELF files fetched from `libcdb` are generally fetched by build-id, e.g. https://gitlab.com/libcdb/libcdb/-/blob/master/hashes/build_id/2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e

There is a tool called `eu-unstrip`, apart of `elfutils` (`apt-get install elfutils`) that can add detached debug information to the libc.

There is also a service for fetching the detached debug info, at e.g. https://debuginfod.systemtap.org/buildid/2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e/debuginfo

We should implement functionality in `libcdb` to attempt to automatically re-symbolize any libc that we fetch.  `debuginfod.systemtap.org` appears to be limited to Ubuntu libc, but that's a common-enough use case to implement the feature.
 
The end result is a `libc.so` which has debug symbols included, which is very useful for when doing exploits that are heap manipulation, since e.g. Pwndbg and GEF need to look up non-exported symbols in order to have accurate mapping of the heap.

More info on the `systemtap.org` server is available here: https://sourceware.org/legacy-ml/systemtap/2020-q1/msg00002.html

Thanks goes to @jagotu for the idea on Discord


Discussion:
The way to expose this is probably to just add the functionality for downloading from `systemtap` in `pwnlib.libcdb` and expose the "un-stripping" as an additional argument to `pwnlib.libcdb.search_by_hash(..., unstrip=True)` since there's no down-side to checking we can have it be default-true.

We SHOULD however cache the result so that we don't repeatedly fetch or try-to-fetch-and-fail.  I think the caching in `pwnlib.libcdb` is broken anyways, this would be an opportunity to address that.
@peace-maker has expressed that he's already got a prototype for some of this, which could be pretty neat!
What do you think about adding some `libcdb.unstrip_libc(filename)` function which you can use to try to unstrip already available libc binaries obtained elsewhere?

It could be called automatically if that `unstrip` parameter added to the `search_by_*` functions is set.
This could be exposed in the commandline tool as well.

What is broken about the caching?
That sounds fine as an implementation choice.

The caching doesn't work, give it a try manually to pull down a libc by
build-ID and look at your ~/.cache/pwntools-X.X directory.  I haven't
looked into why just yet.


On Fri, Feb 26, 2021 at 2:54 PM peace-maker <notifications@github.com>
wrote:

> What do you think about adding some libcdb.unstrip_libc(filename)
> function which you can use to try to unstrip already available libc
> binaries obtained elsewhere?
>
> It could be called automatically if that unstrip parameter added to the
> search_by_* functions is set.
> This could be exposed in the commandline tool as well.
>
> What is broken about the caching?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Gallopsled/pwntools/issues/1818#issuecomment-786861201>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/APYTIFNDIOVKRZJLJCAUWO3TA737PANCNFSM4YGEX4EQ>
> .
>


Resolving patch:
diff --git a/pwnlib/libcdb.py b/pwnlib/libcdb.py
--- a/pwnlib/libcdb.py
+++ b/pwnlib/libcdb.py
@@ -12,9 +12,12 @@
 from pwnlib.context import context
 from pwnlib.elf import ELF
 from pwnlib.log import getLogger
+from pwnlib.tubes.process import process
 from pwnlib.util.fiddling import b64d
+from pwnlib.util.fiddling import enhex
 from pwnlib.util.fiddling import hexdump
 from pwnlib.util.misc import read
+from pwnlib.util.misc import which
 from pwnlib.util.misc import write
 from pwnlib.util.safeeval import const
 from pwnlib.util.web import wget
@@ -22,6 +25,7 @@
 log = getLogger(__name__)
 
 HASHES = ['build_id', 'sha1', 'sha256', 'md5']
+DEBUGINFOD_SERVERS = ['https://debuginfod.systemtap.org/']
 
 # https://gitlab.com/libcdb/libcdb wasn't updated after 2019,
 # but still is a massive database of older libc binaries.
@@ -88,29 +92,13 @@ def provider_libc_rip(hex_encoded_id, hash_type):
 
 PROVIDERS = [provider_libcdb, provider_libc_rip]
 
-def search_by_hash(hex_encoded_id, hash_type='build_id'):
+def search_by_hash(hex_encoded_id, hash_type='build_id', unstrip=True):
     assert hash_type in HASHES, hash_type
 
     # Ensure that the libcdb cache directory exists
-    cache_dir = os.path.join(context.cache_dir, 'libcdb', hash_type)
-
-    if not os.path.isdir(cache_dir):
-        os.makedirs(cache_dir)
-
-    # If we already downloaded the file, and it looks even passingly like
-    # a valid ELF file, return it.
-    cache = os.path.join(cache_dir, hex_encoded_id)
-
-    if os.path.exists(cache):
-        log.debug("Found existing cached libc at %r", cache)
-
-        data = read(cache)
-        if data.startswith(b'\x7FELF'):
-            log.info_once("Using cached data from %r", cache)
-            return cache
-        else:
-            log.info_once("Skipping unavailable libc %s", hex_encoded_id)
-            return None
+    cache, cache_valid = _check_elf_cache('libcdb', hex_encoded_id, hash_type)
+    if cache_valid:
+        return cache
 
     # Run through all available libc database providers to see if we have a match.
     for provider in PROVIDERS:
@@ -128,16 +116,146 @@ def search_by_hash(hex_encoded_id, hash_type='build_id'):
     if not data or not data.startswith(b'\x7FELF'):
         return None
 
+    # Try to find debug info for this libc.
+    if unstrip:
+        unstrip_libc(cache)
+
+    return cache
+
+def _search_debuginfo_by_hash(base_url, hex_encoded_id):
+    # Deferred import because it's slow
+    import requests
+    from six.moves import urllib
+
+    # Check if we tried this buildid before.
+    cache, cache_valid = _check_elf_cache('libcdb_dbg', hex_encoded_id, 'build_id')
+    if cache_valid:
+        return cache
+
+    # Try to find separate debuginfo.
+    url  = '/buildid/{}/debuginfo'.format(hex_encoded_id)
+    url  = urllib.parse.urljoin(base_url, url)
+    data = b""
+    log.debug("Downloading data from debuginfod: %s", url)
+    try:
+        data = wget(url, timeout=20)
+    except requests.RequestException as e:
+        log.warn_once("Failed to fetch libc debuginfo for build_id %s from %s: %s", hex_encoded_id, base_url, e)
+    
+    # Save whatever we got to the cache
+    write(cache, data or b'')
+
+    # Return ``None`` if we did not get a valid ELF file
+    if not data or not data.startswith(b'\x7FELF'):
+        log.warn_once("Could not fetch libc debuginfo for build_id %s from %s", hex_encoded_id, base_url)
+        return None
+
     return cache
 
+def _check_elf_cache(cache_type, hex_encoded_id, hash_type):
+    """
+    Check if there already is an ELF file for this hash in the cache.
+
+    >>> cache, _ = _check_elf_cache('libcdb', '2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', 'build_id')
+    >>> os.unlink(cache) if os.path.exists(cache)
+    >>> filename = search_by_hash('2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', 'build_id', unstrip=False)
+    >>> hex(ELF(filename).symbols.read)
+    '0xe56c0'
+    >>> filename == cache
+    True
+    """
+    # Ensure that the cache directory exists
+    cache_dir = os.path.join(context.cache_dir, cache_type, hash_type)
+
+    if not os.path.isdir(cache_dir):
+        os.makedirs(cache_dir)
+
+    # If we already downloaded the file, and it looks even passingly like
+    # a valid ELF file, return it.
+    cache = os.path.join(cache_dir, hex_encoded_id)
+
+    if not os.path.exists(cache):
+        return cache, False
+    
+    log.debug("Found existing cached ELF at %r", cache)
+
+    data = read(cache)
+    if not data.startswith(b'\x7FELF'):
+        log.info_once("Skipping unavailable ELF %s", hex_encoded_id)
+        return cache, False
+
+    log.info_once("Using cached data from %r", cache)
+    return cache, True
+
+def unstrip_libc(filename):
+    """
+    Given a path to a libc binary, attempt to download matching debug info
+    and add them back to the given binary.
+
+    This modifies the given file.
+
+    Arguments:
+        filename(str):
+            Path to the libc binary to unstrip.
+
+    Returns:
+        :const:`True` if binary was unstripped, :const:`False` otherwise.
+
+    Examples:
+        >>> filename = search_by_build_id('2d1c5e0b85cb06ff47fa6fa088ec22cb6e06074e', unstrip=False)
+        >>> libc = ELF(filename)
+        >>> hex(libc.symbols.read)
+        '0xe56c0'
+        >>> 'main_arena' in libc.symbols
+        False
+        >>> unstrip_libc(filename)
+        True
+        >>> libc = ELF(filename)
+        >>> hex(libc.symbols.main_arena)
+        '0x1d57a0'
+        >>> unstrip_libc(which('python'))
+        False
+        >>> filename = search_by_build_id('06a8004be6e10c4aeabbe0db74423ace392a2d6b', unstrip=True)
+        >>> 'main_arena' in ELF(filename).symbols
+        True
+    """
+    if not which('eu-unstrip'):
+        log.warn_once('Couldn\'t find "eu-unstrip" in PATH. Install elfutils first.')
+        return False
+
+    libc = ELF(filename, checksec=False)
+    if not libc.buildid:
+        log.warn_once('Given libc does not have a buildid. Cannot look for debuginfo to unstrip.')
+        return False
+
+    for server_url in DEBUGINFOD_SERVERS:
+        libc_dbg = _search_debuginfo_by_hash(server_url, enhex(libc.buildid))
+        if libc_dbg:
+            break
+    else:
+        log.warn_once('Couldn\'t find debug info for libc with build_id %s on any debuginfod server.', enhex(libc.buildid))
+        return False
+
+    # Add debug info to given libc binary inplace.
+    p = process(['eu-unstrip', '-o', filename, filename, libc_dbg])
+    output = p.recvall()
+    p.close()
+
+    if output:
+        log.error('Failed to unstrip libc binary: %s', output)
+        return False
+
+    return True
 
-def search_by_build_id(hex_encoded_id):
+def search_by_build_id(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded Build ID, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded Build ID (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -152,15 +270,17 @@ def search_by_build_id(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'build_id')
+    return search_by_hash(hex_encoded_id, 'build_id', unstrip)
 
-def search_by_md5(hex_encoded_id):
+def search_by_md5(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded md5sum, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded md5sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -175,15 +295,17 @@ def search_by_md5(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'md5')
+    return search_by_hash(hex_encoded_id, 'md5', unstrip)
 
-def search_by_sha1(hex_encoded_id):
+def search_by_sha1(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded sha1, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded sha1sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -198,16 +320,18 @@ def search_by_sha1(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'sha1')
+    return search_by_hash(hex_encoded_id, 'sha1', unstrip)
 
 
-def search_by_sha256(hex_encoded_id):
+def search_by_sha256(hex_encoded_id, unstrip=True):
     """
     Given a hex-encoded sha256, attempt to download a matching libc from libcdb.
 
     Arguments:
         hex_encoded_id(str):
             Hex-encoded sha256sum (e.g. 'ABCDEF...') of the library
+        unstrip(bool):
+            Try to fetch debug info for the libc and apply it to the downloaded file.
 
     Returns:
         Path to the downloaded library on disk, or :const:`None`.
@@ -222,7 +346,7 @@ def search_by_sha256(hex_encoded_id):
         >>> hex(ELF(filename).symbols.read)
         '0xeef40'
     """
-    return search_by_hash(hex_encoded_id, 'sha256')
+    return search_by_hash(hex_encoded_id, 'sha256', unstrip)
 
 
 
@@ -279,4 +403,4 @@ def get_build_id_offsets():
     }.get(context.arch, [])
 
 
-__all__ = ['get_build_id_offsets', 'search_by_build_id', 'search_by_sha1', 'search_by_sha256', 'search_by_md5']
+__all__ = ['get_build_id_offsets', 'search_by_build_id', 'search_by_sha1', 'search_by_sha256', 'search_by_md5', 'unstrip_libc']


Test patch: