Package: qtox
qtoxcreates Bash scripts that simply run the same commands Tox would run but in parallel for each environment.This can lead to a massive speed-up in your daily local workflow.BenchmarksI cloned theFalcon web API framework, rantoxonce, and commented out the scripttools/clean.sh(which error’d out when the unit tests for Python 27 and 36 were run in parallel).Following that, here’s the output oftime toxtook:real    1m9.746s
user    1m18.527s
sys 0m4.732sI then created a bash script usingqtox-epep8 py27 py36 docs  > retox.sh. Here’s the output oftime ./retox.sh:real    0m40.326s
user    1m28.318s
sys 0m3.717sSoqtox’s script ran in 57% the time.Installingqtoxis available on PyPi.If you usepipsiyou can install it with:pipsi install qtoxIf you’re some kind of crazy person who likes polluting your global Python instance you can just callpip install qtoxand I’ll try not to judge you.NoteThe version of tail on OSX doesn’t have the--pidoption, so you’ll need to install it withbrew install coreutils. Scripts generated on osx callgtailinstead oftailfor this reason.Why you shouldn’t call Tox from your dev boxTox is great for making sure tests run a truly isolated environment on a CI platform, but if you’re just trying to run flake8 or the unit tests for the hundredth time today it can be overkill.Tox appears to do a lot of house work before running even the simplest commands. Anecdotally, I’ve always noticed a huge speed improvement when I simply ran a command in the virtualenv tox set up (say,.tox/py27/bin/pytest mypkg) versus when I invoked Tox directly (tox-epy27).The speed savings were significant enough that I ended up writing Bash scripts to run the Tox commands and used that instead of Tox, which always ended up being faster according totime. On the downside, these scripts obviously duplicated information already found in thetox.inifile and often became out of date.The rise of truly amazing static analysis tools in Python such asMyPyexacerbate the problems I see running invoking Tox compared to manually crafting scripts. Tools like MyPy offer the most value when they’re run continuously as development happens. However, because MyPy checks are often put in separate tox environment, it’s easy for people focused on a different problem (say, fixing a unit test) to run only those environments for minutes or hours before they remember to check MyPy, and be left needing to fix a bunch of type errors after they’re convinced they’ve already made their program work at runtime.The best solution is to just run everything as often as possible.qtoxenables this.How to use qtoxImagine a tox file that:formats your code withBlackchecks it withFlake8checks it with MyPyruns unit tests in Python 3.5 and 3.6Note: tox needs to run one time beforeqtoxcan be used, in order for qtox to determine if command line tools are present in the virtualenv’s or if they should be checked for in the Tox’swhitelist_externalssetting.qtoxdoesn’t replace Tox, it just lets you augment it with the ability to re-run it’s commands faster.qtoxcan be used to create a bash script like so:qtox-eblackpep8mypypy35p36>retox.shchmod+xretox.shWhen this script will instantly launch five jobs in parallel and wait on the results in the order you specified (meaning you want the quicker jobs- such as black or flake8- to run first).As it works it’s way through the list, it shows the output of each job in real time. So in this example, blacks output would be seen as it happens, and when black finishes all of flake8’s output that happened in the interim will be shown before it’s current output is displayed, etc.It does this by having every job redirect to a file. When it’s time to consume the results of that job,tailis invoked in another job, which reads from the start of the file and follows it until the process writing to it dies.If a job fails, all other subsequent jobs are simply killed without printing out their output. This keeps things simpler so you don’t have to scroll back up to see what went wrong.It’s up to you to make sure the simpler jobs are put earlier in the list you give toqtox. If you instead put the longer running jobs first, you’ll have to wait for them to finish before seeing feedback from quicker tools such as flake8.Multi Tox Super Projectsqtoxalso supports creating bash scripts for multiple tox projects.Use the-cflag to specify a tox directory, then use-elike normal.Say you have two directories containing two Python projects, both using tox. One is calledacme-lib, and is a reusable library containing core business logic, while the other isacme-rest-api, which usesacme-lib. If you’re working on both at the same time, you may want to simply run all of the Tox tests together, starting with Flake 8 and MyPy tests first.Generating a bash script for that would look something like this:qtox-cacme-lib-epep8mypy-cacme-rest-api-epep8mypy-cacme-lib-epytest-cacme-rest-api-epytest>retox.shchmod+x./retox.shThe tasks that will be waited on / shown the progress for will be in this order:acme-lib’s pep8 and mypy checksacme-rest-api’s pep8 and mypy checksacme-lib’s pytest testsacme-rest-api’s pytest testsTo beat a dead horse, I’ll reiterate that all of these tasks will start simultaneously, meaning the relatively expensive REST API unit tests will start running at the same time as everything else. The simpler checks will simply be waited on first.