Package: b2btest
b2btest - Light framework to setup back-to-back test scriptsThis package is helpfull to prepare and manage
back-to-back tests to get control over software
you don't have proper tests yet.On one side, it includesback2back, a comand line tool
that performs back-to-back tests on any software
that can be called by command line and produces
some output files you want to get warned when they get different.
You can define the set of commands and the outputs to watch in a simple
YAML file.On the other side, it includes comparision functions
and python-unittests like asserts you can use within
your test suites that perform similar workflows, keeping
the reference outputs in a folder and named by the testname.Both the command line tool and the python libraries,
are extendable with plugins to get informative diff for several
file formats: audio, xml, text, pdf...Why back-to-back testingA Back-to-back tests is a black box tests that just compares
that, given an input, you have the same output all the time.
Unit testing and Test Driven Development are a quite more
preferable strategy to test a piece of software.
But often we need to change a piece of software which has
been developed without proper testing.
A quick way to get control over it is setting up a set of
back-to-back tests and then proceeding with the
refactoring with a larger level of confidence than having
no test at all.Note of warning:Don't feel too confident by being backed by back2back tests.
It is black-box testing, so it is hard to ensure full coverage.
You may be changing a behaviour which is not exercised
by the b2b test, and not noticing.Easing the workflow with b2btestWhen b2b tests are hard to run and maintain,
they use to get old and useless.
This script automates most tedious back2back
related task such as setting up, verifying results,
accepting changes, clearing data...Features:You can put under back2back testing eitherthe outcomes (files) of any shell command line (with pipes and so on), orany serializable data in aunittest.TestCasemethodIt is auto-checked, like most Xunit frameworksIt automagically manages the expectation dataOn failure, it generates handy data to evaluate
the changes by providing diffs and keeping
failed and expected data for reference.Provides a handy command line to accept failed
results as proper ones.When the test turns green or it is accepted all
failure related data gets cleared.Comparators and diff generators can be added for your own file type.You can set architecture dependant outputs for the same test.How to installJust use:$pipinstallb2btestIn order to properly diff audio files you should consider installing the audio extension$pipinstallb2btest_audioDependenciesFor xml files, it requireslxmlmodule and bothlibxml2andlibxsltlibraries installedBack2Back testing in your unittestimportunittestimportb2btest# Not used but load a new assertion method for TestCaseimportdatetimeclassMyTest(unittest.TestCase):deftest_this(self):self.assertB2BEqual("data")deftest_that_willallwaysfail(self):self.assertB2BEqual(str(datetime.datetime.now()))if__name__=='__main__':# acceptMode attribute makes the assert accept the results# expectation as newif'--accept'insys.argv:sys.argv.remove('--accept')unittest.TestCase.acceptMode=Trueunittest.main()Back2Back testing of cli programsWhen you are testing back-to-back the output of a command line,
you define a yaml file like this (name it b2bcases.yaml).#!/usr/bin/env back2backdatapath:"b2bdata"# Point it to the directory containing your reference datatestcases:HelloWorld:command:echo Hello World > output.txtoutputs:-output.txtAlwaysChanging:command:date > output.txtoutputs:-output.txtGenerate1KHzSine:# Requires the b2btest_audio packagecommand:sox -n /tmp/sine.wav synth 1.0 sine  1000.0outputs:-/tmp/sine.wavTo list the available test cases:back2backb2bcases.yaml--listTo run them:back2backb2bcases.yamlThe first time you run a test case, it will fail as there is no
expectation, you must to check it is valid and accept it.
Successive failures means that the behaviour has change.
You can accept the new result if the behavioural change is expected.To accept a concrete case:back2backb2bcases.yaml--acceptHelloWorldTo accept all failing tests:back2backb2bcases.yaml--acceptallIf some output depends on the computer architecture or in the platform (windows, mac...)
use the--archand--platformoptions when accepting.
It will generate an independent expectation file for such architecture or platform.back2backb2bcases.yaml--acceptHelloWorld--archOld intefaceIf you want to generate the test cases progamaticaly,
you still are able to use the old python interface.
Instead of a yaml file, write python script like this:Just like in this b2b script does:#!/usr/bin/pythonimportsysfromb2btestimportrunBack2BackProgramdata_path="path/to/b2bdata"back2BackTests=[("myprogram_with_option","./myprogram --option input1.wav input2.wav output.wav otheroutput.wav ",["output.wav","otheroutput.wav",]),("myprogram_with_other_option","./myprogram --other-option input1.wav input2.wav output.wav ignoredoutput.wav ",["output.wav",]),]runBack2BackProgram(data_path,sys.argv,back2BackTests)Save this file asback2back.py, for example, and make it executable.Use the python script directly with the same command line
but without the yaml file.Extra advicesUse continuous integrationPut your tests under a continuous integration system suchTravis-CIBuildBotTestFarmCDashYou might be lazy passing tests but bots aren't.
Connect your bots to your VCS so they test for every commit.Keep your expectations up to dateIf one b2b test gets red, don't keep it for long,
either accept it or roll-back your code.
b2b detect changes, but if you are in a change
you won't notice whether a second one happens.
If your expectation data is backed by a version
control system dare to accept wrong expectation data
until you fix it. But don't forget.Change log1.4.0Audio plugin separated from this code base asb2btest_audio1.3.3Simplified dependency on lxml1.3.2diffaudioas console scriptdiffxmlas console scriptFix: entry points for xml and audio pluginsJust markdown README1.3.1Updated README1.3Avoid larg diffs by telling just the generated file with the failed resultsFix unicode problems in certain python versionsUsing older lxml versions for python<3.51.2CLI: Fix: only the first output was actually checkedPlugin based type sensitive diffSpecific diff for XMLXML and Audio diffing now are extras'extensions' key in yaml testcases to associate custom file extensions to a diff plugin1.1Unit test like usage for back-to-back test Python code instead of command line programs.New commandline toolback2backthat takes a yaml file with the test cases definitions.1.0First github version(There were previous unpublished versions)