Package: abi3audit
abi3auditRead our blog post about how we find bugs withabi3audit!abi3auditscans Python extensions forabi3violations and inconsistencies.It can scan individual (unpackaged) shared objects, packaged wheels, or entire
package version histories.IndexMotivationInstallationUsageExamplesLimitationsLicensingMotivationCPython (the reference implementation of Python) defines a stable API and corresponding
ABI ("abi3"). In principle, any CPython extension can be built against this
API/ABI and will remain forward compatible with future minor versions of CPython.
In other words: if you build against the stable ABI for Python 3.5, your
extension should work without modification on Python 3.9.The stable ABI simplifies packaging of CPython extensions, since the packager
only needs to build oneabi3wheel that targets the minimum supported Python
version.To signal that a Python wheel containsabi3-compatible extensions,
the Python packaging ecosystem uses theabi3wheel tag, e.g.:pyrage-1.0.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whlUnfortunately, there isno actual enforcementofabi3compliance
in Python extensions at install or runtime: a wheel (or independent
shared object) that is tagged asabi3is assumed to beabi3, but
is not validated in any way.To make matters worse, there isno formal connectionbetween the flag
(--py-limited-api)
that controls wheel tagging and the build macros
(Py_LIMITED_API)
that actually lock a Python extension into a specificabi3version.As a result: it is very easy to compile a Python extension for the wrongabi3version, or to tag a Python wheel asabi3without actually compiling it
asabi3-compatible.This has serious security and reliability implications: non-stable parts
of the CPython ABI can change between minor versions, resulting in crashes,
unpredictable behavior, or potentially exploitable memory corruption when
a Python extension incorrectly assumes the parameters of a function
or layout of a structure.Installationabi3auditis available viapip:pipinstallabi3auditUsageYou can runabi3auditas a standalone program, or viapython -m abi3audit:abi3audit--help
python-mabi3audit--helpTop-level:usage: abi3audit [-h] [--debug] [-v] [-R] [-o OUTPUT] [-S][--assume-minimum-abi3 ASSUME_MINIMUM_ABI3]SPEC [SPEC ...]Scans Python extensions for abi3 violations and inconsistenciespositional arguments:SPEC                  the files or other dependency specs to scanoptions:-h, --help            show this help message and exit--debug               emit debug statements; this setting also overrides`ABI3AUDIT_LOGLEVEL` and is equivalent to setting itto `debug`-v, --verbose         give more output, including pretty-printed results foreach audit step-R, --report          generate a JSON report; uses --output-o OUTPUT, --output OUTPUTthe path to write the JSON report to (default: stdout)-S, --strict          fail the entire audit if an individual audit stepfails--assume-minimum-abi3 ASSUME_MINIMUM_ABI3assumed abi3 version (3.x, with x>=2) if it cannot bedetectedExamplesAudit a single shared object, wheel, or PyPI package:# audit a local copy of an abi3 extensionabi3auditprocmaps.abi3.so# audit a local copy of an abi3 wheelabi3auditprocmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl# audit every abi3 wheel for the package 'procmaps' on PyPIabi3auditprocmapsShow additional detail (pretty tables and individual violations) while auditing:abi3auditprocmaps--verboseyields:[17:59:46] 👎 procmaps:procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl: procmaps.abi3.souses the Python 3.10 ABI, but is tagged for the Python 3.6 ABI┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓┃ Symbol                  ┃ Version ┃┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩│ PyUnicode_AsUTF8AndSize │ 3.10    │└─────────────────────────┴─────────┘[17:59:47] 💁 procmaps: 2 extensions scanned; 1 ABI version mismatches and 0ABI violations foundGenerate a JSON report for each input:abi3auditprocmaps--report|python-mjson.toolyields:{"specs":{"procmaps":{"kind":"package","package":{"procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl":[{"name":"procmaps.abi3.so","result":{"is_abi3":true,"is_abi3_baseline_compatible":false,"baseline":"3.6","computed":"3.10","non_abi3_symbols":[],"future_abi3_objects":{"PyUnicode_AsUTF8AndSize":"3.10"}}}],"procmaps-0.6.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl":[{"name":"procmaps.abi3.so","result":{"is_abi3":true,"is_abi3_baseline_compatible":true,"baseline":"3.7","computed":"3.7","non_abi3_symbols":[],"future_abi3_objects":{}}}]}}}}Limitationsabi3auditis abest-efforttool, with some of the same limitations asauditwheel. In particular:abi3auditcannot check fordynamicabi3 violations, such as an extension
that callsdlsym(3)to invoke a non-abi3 function at runtime.abi3auditcan confirm the presence of abi3-compatible symbols, but does
not have an exhaustive list of abi3-incompatiblesymbols. Instead, it looks
for violations by looking for symbols that start withPy_or_Py_that
are not in the abi3 compatibility list. This isunlikelyto result in false
positives, butcouldif an extension incorrectly uses those reserved
prefixes.When auditing a "bare" shared object (e.g.foo.abi3.so),abi3auditcannot
assume anything about the minimumintendedabi3 version. Instead, it
defaults to the lowest known abi3 version (abi3-cp32) and warns on any
version mismatches (e.g., a symbol that was only stabilized in 3.6).
This can result in false positives, so users are encouraged to audit entire
wheels or packages instead (since they contain the sufficient metadata).abi3auditconsiders the abi3 version when a symbol wasstabilized,
notintroduced. In other words:abi3auditwill produce a warning
when anabi3-cp36extension contains a function stabilized in 3.7, even
if that function was introduced in 3.6. This isnota false positive
(it is an ABI version mismatch), but it'sgenerallynot a source of bugs.abi3auditchecks both the "local" and "external" symbols for each extension,
for formats that support both. It does this to catch symbols that have been
inlined, such as_Py_DECREF. However, if the extension's symbol table
has been stripped, these may be missed.Licensingabi3auditis licensed under the MIT license.abi3auditincludes ASN.1 and Mach-O parsers generated from
definitions provided by theKaitai Structproject.
These vendored parsers are licensed by the Kaitai Struct authors under the MIT
license.