Yelp/Testify — pull request #126 by baris (2012-12-14 02:01:45+00:00)
Title: violator collector plugin

This branch adds a plugin to collect unittest "violations" made by tests. The main assumption is that, a test is said to be making unittest violations if it accesses the filesystem with the intend of changing the state of the system (create, write, rm, etc. reading from fs is allowed) or makes network access.

Violations are detected using a Python extention, called catbox (https://github.com/baris/catbox). Catbox is a sandbox implementation for Python modules and uses ptrace to collect (or prevent) system calls of a traced child. Violation collector plugin, running the TestProgram in catbox, forks and traces the program for said unittest violations.

Although I tried to make it explicit where possible and commented the behavior, the plugin's pipeline may be tricky to follow since tests are run in one process and syscall inspection is done in another. There are, infact, in total 3 processes involved in the process: catbox process (parent, making syscall inspection), traced process (child running testify tests), and finally watchdog (the other child of catbox, watches catbox and cleans up the traced process in case catbox is killed with SIGKILL).

Plugin is only enabled with an option. Even when enabled the current behavior is to allow the violating system call but record the violation (and the violator). By default violations are stored in a disk based sqlite database, violations.sqlite. Reporter prints the number of violations and in verbose mode violators by test method. Sqlite database is not cleaned after for further inspection if necessary and cleaned up on the next run.

ViolationStore may be (and will be in later iterations) more efficient to use a queue and insert to database when a test (or even test_case) completes. But I'd like to get this merged as it is and improve performance with a later pull request.