Gallopsled/pwntools — pull request #158 by zachriggle (2014-10-26 23:28:25+00:00)
Title: pwnlib 2.2 rewrite of context, logging, tubes, various bugfixes

Major pwnlib overhaul to address outstanding issues.

It is **highly** recommended to look at the diff with whitespace disabled (add ?w=1 to URL or `git diff -w`).

# Issues Closed

The following issues are closed by this pull request.

- #151 Recvline regression
- #150 Tubes timeout
- #147 "pip2 not present on debian"
- #141 Context default values and side-effects
- #140 Context module usage
- #135 Logging re-write

# Test Coverage

I have added a great deal of unit tests to `pwntools`, and started a regression testing framework.

## **Unit** Test Coverage

One test fails without a custom-built `binutils-mips`.

```
$ make doctest
...
Doctest summary
===============
591 tests
1 failure in tests
0 failures in setup code
0 failures in cleanup code
```

## **Integration** Testing

I've created a new repository at [zachriggle/pwnlib-regression](https://github.com/zachriggle/pwnlib-regression) which performs regression testing against functionality. This uses a binary which is designed to run shellcode, leak offsets, and in general be useful for testing pwntools. **`bbgp` and `pork` do not work.** I have created a new branch for that repo with the changes necessary to work properly.

Additionally, I've verified that most of the `pwnlib-write-ups` exploits still work, as well as my collection of `io.smashthestack.com` exploits which I cannot publish. All of the published `wargames` exploits work.

# Major Changes

## Documentation

A good deal of documentation has been added and updated.

## pwnlib.context

This has been made into a proper module. The global `context` is now just a globally-scoped object.

The mechanism used for defaults and threading have been updated, so that they're a bit cleaner.

Setting `context.arch` now sets other `context` values to sane defaults.

## pwnlib.log

Uses the normal Python logging subsystem. Log levels for individual modules can now be set, while ``context.log_level`` provides a maximum verbose-ness for logging (same behavior as before). This is useful for when developing to see e.g. verbose `ELF` messages without printing everything a `tube` ever sees.

All animation still works exactly as it did. `log.waitfor` can now be used with the scoping `with` keyword.

The `log_level` module is removed.

## pwnlib.asm

Large amount re-written to completely migrate to binutils, rather than relying on `nasm` for x86 and dealing with the differences. Also auto-detects the names of binaries, as the exact triplet of the installed version may differ. Since we're not relying on any ABIs when assembling shellcode, it doesn't matter if we use `arm-linux-gnueabi` or `arm-none-none`.

## pwnlib.shellcraft

In keeping with the changes to `pwnlib.asm`, all of the i386 shellcode has been rewritten in `gas -Mintel` syntax.

## pwnlib.gdb

Includes `gdb.debug` for debugging over SSH, as shown in #108.

## pwnlib.data

All of the precompiled binutils have been removed. Users should use their package manager to install cross-architecture assemblers/disassemblers.

## pwnlib.util.crc

Instead of performing text processing at-runtime to scrape constants, this information has been cached in `pwnlib/util/crc/known.py`.

## pwnlib.util.packing

Rather than the same repeating copy/pasted/modified code, most of the routines are created dynamically at runtime. This should make maintenance easier, and less prone to copy-paste errors.