Rebased work around for side effects in setup.py script

This is just a rebase of #1257 so we can merge @xolox's hard work.