Don't leave cruft around in sys.modules when an import fails.

Closes: #214, #225

Squashed from a bunch of commits by @olasd
Fix flake error