Package: pbbt
Table of ContentsOverviewUsing PBBTBuilt-in Test TypesCommon FieldsTest SuiteConditional VariablesInclude TestPython CodeShell CommandWrite to FileRead from FileRemove FileMake DirectoryRemove DirectoryDoctestUnittestPytestCoverageCoverage checkCoverage reportCustom Test TypesCommand-line InterfaceAPI ReferenceOverviewPBBT is a regression test harness forblack-box testing.  It is
suitable for testing complex software components with well-defined input
and output interfaces.input    +----------+   output
o--------> | Software | --------->o
           +----------+In black-box testing, atest caseis a combination ofinputand
expectedoutputdata.  The test harness executes the software with the
given input and verifies that the produced output coincides with the
expected output.Black-box testing could be implemented for many different types of
software.  For example,a database system:the input is a SQL statement, the output is a set
of records;a web service:the input is an HTTP request, the output is an HTTP
response;a command-line utility:the input is a sequence of command-line
parameters andstdin, the output isstdout;a GUI application:different approaches are possible; for instance,
the input could be a sequence of user actions, and the output could be
the state of a particular widget.PBBT is a Python library and an application which allows you to:use built-in test types for testing command-line scripts and Python
code;register custom test types;prepare test input in a succinctYAMLformat;in thetrainmode, run the test cases and record expected output;in thecheckmode, run the test cases and verify that the produced
output coincides with the pre-recorded expected output.PBBT is a free software released under MIT license.  PBBT is created by
Clark C. Evans and Kirill Simonov fromPrometheus Research, LLC.Using PBBTTo install PBBT, you can usepippackage manager:# pip install pbbtThis command downloads and installs the latest version of PBBT fromPython Package Index.  After successful installation, you should be
able to importpbbtPython module and runpbbtcommand-line
utility.To start using PBBT, you need to create a file with input data.  For
example, createinput.yamlwith the following content:py: |
  print "Hello, World!"This file is inYAMLformat, which is a data serialization language
similar toJSON, and, in fact, a superset of JSON.  The file above
could be represented in JSON as:{ "py": "print \"Hello, World!\"\n" }For description of YAML syntax and semantics, seehttp://yaml.org/.Next, execute PBBT intrainingmode to generate expected output data.
Run:$ pbbt input.yaml output.yaml --trainand accept new output when asked.  PBBT will write output data tooutput.yaml:py: print-hello-world
stdout: |
  Hello, World!Now you can start PBBT incheckingmode, in which it executes test
cases and verifies that expected and actual output data coincide:$ pbbt input.yaml output.yamlTo add more test cases toinput.yaml, you need to convert it to atest suite:title: My Tests
tests:
- py: |
    print "Hello, World!"
- sh: echo Hello, World!The file now contains a test suiteMy Testswith two test cases: one
as in the previous example, and another that executes a shell commandecho Hello, World!:sh: echo Hello, World!The output of this test case isstdoutproduced by the shell
command.  To record expected output, runpbbtin training mode
again.Built-in Test TypesOut of the box, PBBT supports a small set of predefined test types:test Python code;test a shell command;file manipulation tests.Also available are special test types:test suite;include;conditional variables.gateway to other test systems.Each test type defines the structure of input and output records, that
is, the set of mandatory and optional fields and the type of field
values.  In this section, we list all available test types and describe
their input fields.Common FieldsThe following optional fields are available for all test types where
they make sense:skip:trueorfalseOntrue, skip this test case.if: variable, list of variables or Python expressionOn avariable name, run this test case only if the variable is
defined and not false.On alist of variables, run this test case only if at least one
variable is defined and not false.On aPython expression, run this test case if the expression
evaluates to true.  You can use any conditional variables in the
expression.unless: variable, list of variables or Python expressionOn avariable name, skip this test case if the variable is defined
and not false.On alist of variables, skip this test case if at least one
variable is defined and not false.On aPython expression, skip this test case if the expression
evaluates to true.  You can use any conditional variables in the
expression.ignore:true,falseor regular expressionOntrue, permit the actual and expected output to be unequal.
The test case must still execute without any errors.On aregular expression, pre-process the actual and expected
output before comparing them:Run the regular expression against the output data and find
all matches.If the regular expression does not contain()subgroups,
erase all the matches from the output.If the regular expression contains one or more()subgroups,
erase the content of the subgroups from the output.The regular expression is compiled withMULTILINEandVERBOSEflags.Example:title: Integration with MySQL
if: has_mysql
tests:
- set:
    MYSQL_HOST: localhost
    MYSQL_PORT: 3306
  unless: [MYSQL_HOST, MYSQL_PORT]
- read: /etc/mysql/my.cnf
  if: MYSQL_HOST == 'localhost'
- py: test-scalar-types.py
  ignore: |
    ^Today:.(\d\d\d\d)-(\d\d)-(\d\d)$
- py: test-array-types.py
  skip: true    # No array type in MySQLTest SuiteA test suite is a collection of test cases.A suite may contain other suites and thus all test suites form a tree
structure.  Apathformed from suite identifiers can uniquely locate
any suite.  We use file-system notation for suite paths (e.g./path/to/suite).Fields:title: textThe title of the suite.suite: identifier (optional)The identifier of the suite.  If not set, generated from the title.tests: list of input recordsThe content of the suite.output: path (optional)If set, the expected output of the suite is loaded from the given
file.Example:title: All Tests
suite: all
output: output.yaml
tests:
- py: ./test/core.py
- py: ./test/ext.py
- title: Database Tests
  tests:
  - py: ./test/sqlite.py
  - py: ./test/pgsql.py
  - py: ./test/mysql.pyIn this example, the path to theDatabase Testssuite is/all/database-tests.Conditional VariablesThis test case defines a conditional variable.Variables could be used inifandunlessclauses to
conditionally enable or disable a test case.  Variables could also be
set or read in Python tests via a global dictionary__pbbt__.Conditional variables could also be set from command line using-Doption.Setting a conditional variable affects all subsequent test cases within
the same test suite.  Variable values are reset on exit from the suite.Fields:set: variable or dictionary of variablesOn avariable name, set the value of the given variable toTrue.On adictionary, set the values of the given variables.Example:title: MySQL Tests
tests:
- set: MYSQL
- set:
    MYSQL_HOST: localhost
    MYSQL_PORT: 3306
  unless: [MYSQL_HOST, MYSQL_PORT]
- py: |
    # Determine the version of the MySQL server
    import MySQLdb
    connection = MySQLdb.connect(host=__pbbt__['MYSQL_HOST'],
                                 port=int(__pbbt__['MYSQL_PORT']),
                                 db='mysql')
    cursor = connection.cursor()
    cursor.execute("""SELECT VERSION()""")
    version_string = cursor.fetchone()[0]
    version = tuple(map(int, version_string.split('-')[0].split('.')))
    __pbbt__['MYSQL_VERSION'] = version
- py: test-ddl.py
- py: test-dml.py
- py: test-select.py
- py: test-new-features.py
  if: MYSQL_VERSION >= (5, 5)Include TestThis test case loads and executes a test case from a file.Fields:include: pathThe file to load.  The file should contain an input test record in
YAML format.Example:title: All Tests
tests:
- include: test/core.yaml
- include: test/ext.yaml
- include: test/sqlite.yaml
- include: test/pgsql.yaml
- include: test/mysql.yamlPython CodeThis test case executes Python code and producesstdout.Fields:py: path or Python codeOnPython code, the source code to execute.On afile name, the file which contains source code to execute.stdin: text (optional)Content of the standard input.except: exception type (optional)If set, indicates that the code is expected to raise an exception of
the given type.Example:title: Python tests
tests:
- py: hello.py
- py: &sum |
    # Sum of two numbers
    import sys
    a = int(sys.stdin.readline())
    b = int(sys.stdin.readline())
    c = a+b
    sys.stdout.write("%s\n" % c)
  stdin: |
    2
    2
- py: *sum
  stdin: |
    1
    -5
- py: *sum
  stdin: |
    one
    three
  except: ValueErrorNote that we use a YAML anchor (denoted by&sum) and aliases
(denoted by*sum) to use the same piece of code in several tests.Shell CommandThis test case executes a shell command and producesstdout.Fields:sh: command or executable with a list of parametersThe shell command to execute.stdin: text (optional)Content of the standard input.cd: path (optional)Change the current working directory to the given path before
executing the command.environ: dictionary of variables (optional)Add the given variables to the command environment.exit: integer (optional)The expected exit code;0by default.Example:title: Shell tests
tests:
- sh: echo Hello, World!
- sh: cat
  stdin: |
    Hello, World!
- sh: [cat, /etc/shadow]
  exit: 1   # Permission deniedWrite to FileThis test case creates a file with the given content.Fields:write: pathThe file to create.data: textThe file content.Example:write: test/tmp/data.txt
data: |
    Hello, World!Read from FileThe output of this test is the content of a file.Fields:read: pathThe file to read.Example:read: test/tmp/data.txtRemove FileThis test case removes a file.  It is not an error if the file does not
exist.Fields:rm: path or list of pathsFile(s) to remove.Example:rm: test/tmp/data.txtMake DirectoryThis test case creates a directory.Parent directories are also created if necessary.  It is not an error if
the directory already exists.Fields:mkdir: pathThe directory to create.Example:mkdir: test/tmpRemove DirectoryThis test case removes a directory with all its content.It is not an error if the directory does not exist.Fields:rmdir: pathThe directory to delete.Example:rmdir: test/tmpDoctestThis test case executesdocteston a set of files.Fields:doctest: path patternFiles with doctest sessions.Example:doctest: test/test_*.rstUnittestThis test case executesunittesttest suite.Fields:unittest: path patternFiles with unittest tests.Example:unittest: test/test_*.pyPytestThis test case executespy.testtest suite.Packagepytestfromhttp://pytest.org/must be installed.Fields:pytest: path patternFiles with py.test tests.Example:pytest: test/test_*.pyCoverageThis test case starts coverage of Python code.Packagecoveragefromhttp://nedbatchelder.com/code/coverage/must
be installed.Fields:coverage: file name orNonePath to the configuration file.data_file: file nameWhere to save coverage data.timid:falseortrue(optional)Use a simpler trace function.branch:falseortrue(optional)Enable branch coverage.source: file paths of package names (optional)Source files or packages to measure.include: file patternsFiles to measure.omit: file patternsFiles to omit.Example:coverage:
source: pbbt
branch: trueCoverage checkThis test case stops coverage and reports measurement summary.Fields:coverage-check: floatExpected coverage percentage.Example:coverage-check: 99.0Coverage reportThis test case stops coverage and saves coverage report to a file.Fields:coverage-report: directoryWhere to save the report.Example:coverage-report: coverageCustom Test TypesIn this section, we discuss how to add custom test types to PBBT.Suppose we want to test a SQL database by running a series of SQL
queries and validating that we get expected output.  To implement this
testing scheme, we need a way to:open a connection to the database;execute a SQL statement and produce a sequence of records.The input file may look like this:title: Database tests
tests:
# Remove the database file left after the last testing session.
- rm: test.sqlite
# Create a new SQLite database.
- connect: test.sqlite
# Run a series of SQL commands.
- sql: |
    SELECT 'Hello, World!';
- sql: |
    CREATE TABLE student (
        id      INTEGER PRIMARY KEY,
        name    TEXT NOT NULL,
        gender  CHAR(1) NOT NULL
                CHECK (gender in ('f', 'i', 'm')),
        dob     DATE NOT NULL
    );
- sql: |
    INSERT INTO student (id, name, gender, dob)
    VALUES (1001, 'Linda Wright', 'f', '1988-10-03'),
           (1002, 'Beth Thompson', 'f', '1988-01-24'),
           (1003, 'Mark Melton', 'm', '1984-06-05'),
           (1004, 'Judy Flynn', 'f', '1986-09-02'),
           (1005, 'Jonathan Bouchard', 'm', '1982-02-12');
- sql: |
    SELECT *
    FROM student
    ORDER BY dob;
- sql: |
    SELECT name
    FROM student
    WHERE id = 1003;In this input file, we use two new types of test cases:connectSpecifies the connection to a SQLite database.sqlSpecifies a SQL statement to execute.We will write a PBBT extension implementing these test types.Create a filesql.pywith the following content:from pbbt import Test, Field, BaseCase, MatchCase, listof
import sqlite3, traceback, csv, StringIO

@Test
class ConnectCase(BaseCase):

    class Input:
        connect = Field(str)

    def check(self):
        self.state['connect'] = None
        try:
            self.state['connect'] = sqlite3.connect(self.input.connect)
        except:
            self.ui.literal(traceback.format_exc())
            self.ui.warning("exception occurred while"
                            " connecting to the database")
            self.ctl.failed()
        else:
            self.ctl.passed()

@Test
class SQLCase(MatchCase):

    class Input:
        sql = Field(str)

    class Output:
        sql = Field(str)
        rows = Field(listof(listof(object)))

    def render(self, output):
        stream = StringIO.StringIO()
        writer = csv.writer(stream, lineterminator='\n')
        writer.writerows(output.rows)
        return stream.getvalue()

    def run(self):
        connection = self.state.get('connect')
        if not connection:
            self.ui.warning("database connection is not defined")
            return
        rows = []
        try:
            cursor = connection.cursor()
            cursor.execute(self.input.sql)
            for row in cursor.fetchall():
                rows.append(list(row))
        except:
            self.ui.literal(traceback.format_exc())
            self.ui.warning("exception occurred while"
                            " executing a SQL query")
            connection.rollback()
            new_output = None
        else:
            connection.commit()
            new_output = self.Output(sql=self.input.sql, rows=rows)
        return new_outputTo use this extension, add parameter-Esql.pyto all PBBT
invocations.  For example:$ pbbt -E sql.py input.yaml output.yaml --trainNow we will explain the content ofsql.pyline by line.The first line imports some classes and decorators we will use:from pbbt import Test, Field, BaseCase, MatchCaseTo register a test type, use@Testdecorator.  Here is a most
general template:@Test
class CustomCase(object):

    class Input:
        some_field = Field(...)
        [...]

    class Output:
        some_field = Field(...)
        [...]

    def __init__(self, ctl, input, output):
        self.ctl = ctl
        self.input = input
        self.output = output

    def __call__(self):
        [...]
        return new_outputThe argument of the decorator must be a class that adheres the following
rules:The structure of input and output records is described with nested
classesInputandOutput.  Record fields are specified usingFielddescriptor.To create a test case, the test harness makes an instance of the class.
The class constructor accepts three arguments:ctlTest harness controller.  It is used for user interaction,
reporting test success or failure, and as a storage for
conditional variables.inputThe input record.outputThe expected output record orNone.The test case is executed by calling its__call__()method.  This
method must run the test case, generate a new output record, and compare
it with the given expected output record.If the expected and actual output record do not coincide:In thecheckmode, the method must report a failure.In thetrainmode, the method may ask the user for permission to
update expected output.  If expected output is to be updated, the
method should return the new output record.PBBT also provides two mixin classes:BaseCaseandMatchCase.
These classes implement most of the necessary functionality for common
types of tests.Let’s reviewconnecttest type, which is implemented byConnectCaseclass:@Test
class ConnectCase(BaseCase):

    class Input:
        connect = Field(str)

    def check(self):
        [...]ConnectCaseis inherited fromBaseCase, which is suitable for
test which produce no output data and are executed for their side
effects.  The nestedInputclass definition is used to declare the
fields of the input record.  In this case, the input record has just one
text fieldconnect, which contains the name of the database.Test types inherited fromBaseCasemust override abstract methodcheck():import sqlite3, traceback
[...]

@Test
class ConnectCase(BaseCase):
    [...]

    def check(self):
        self.state['connect'] = None
        try:
            self.state['connect'] = sqlite3.connect(self.input.connect)
        except:
            self.ui.literal(traceback.format_exc())
            self.ui.warning("exception occurred while"
                            " connecting to the database")
            self.ctl.failed()
        else:
            self.ctl.passed()This code attempts to create a new database connection and store it as a
conditional variableconnect.  If the attempt fails, it callsui.literal()to display the exception traceback andctl.failed()to report test failure.  Otherwise,ctl.passed()is called to
indicate that the test succeeded.Next, let’s reviewsqltest type:@Test
class SQLCase(MatchCase):

    class Input:
        sql = Field(str)

    class Output:
        sql = Field(str)
        rows = Field(listof(listof(object)))

    def run(self):
        [...]

    def render(self, output):
        [...]This test type has both input and output records, which are described
withInputandOutputnested classes.  The input record contains
one fieldsql, a SQL query to execute.  The output record contains
two fields:sqlandrows.  Fieldsqlcontains the same SQL
query and is used for matching the output record with the complementary
input record.  Fieldrowscontains a list of output rows generated
by the SQL query.SQLCaseis inherited fromMatchCase, which is a mixin class
for test types that produce text output.  Test types inherited fromMatchCasemust override two methods:run()Executes the test case and returns the produced output record.render(output)Convert an output record to printable form.InSQLCase,render()is implemented by converting output rows to
CSV format:import csv, StringIO
[...]

@Test
class SQLCase(MatchCase):
    [...]

    def render(self, output):
        stream = StringIO.StringIO()
        writer = csv.writer(stream, lineterminator='\n')
        writer.writerows(output.rows)
        return stream.getvalue()Methodrun()is implemented as follows:@Test
class SQLCase(MatchCase):
    [...]

    def run(self):
        connection = self.state.get('connect')
        if not connection:
            self.ui.warning("database connection is not defined")
            return
        rows = []
        try:
            cursor = connection.cursor()
            cursor.execute(self.input.sql)
            for row in cursor.fetchall():
                rows.append(list(row))
        except:
            self.ui.literal(traceback.format_exc())
            self.ui.warning("exception occurred while"
                            " executing a SQL query")
            connection.rollback()
            new_output = None
        else:
            connection.commit()
            new_output = self.Output(sql=self.input.sql, rows=rows)
        return new_outputCommand-line InterfaceUsage:pbbt [<options>] INPUT [OUTPUT]Here,INPUTandOUTPUTare files which contain input and output
test data respectively.The following options are available:-h,--helpDisplay usage information and exit.-q,--quietShow only warnings and errors.-T,--trainRun tests in the training mode.-P,--purgePurge stale output data.-MN,--max-errorsNHalt afterNtests failed;0means “never”.-DVAR,-DVAR=VALUE,--defineVAR,--defineVAR=VALUESet a conditional variable.-EFILE,-EMODULE,--extendFILE,--extendMODULELoad a PBBT extension from a file or a Python module.-SID,--suiteIDRun a specific test suite.PBBT can also read configuration from the following files:setup.cfgThis file is in INI format with PBBT settings defined in section[pbbt].  The following parameters are recognized:extend,input,output,define,suite,train,purge,max_errors,quiet.pbbt.yamlThis file must be a YAML file with the following keys:extend,input,output,define,suite,train,purge,max-errors,quiet.PBBT can also be executed as a Distutils command:python setup.py pbbtIn this case, PBBT configuration could be specified insetup.cfgor via command-line parameters.API Referencepbbt.maybe(T)Pseudo-type forisinstance(X,...): checks ifXis an
instance ofTor equal toNone.pbbt.oneof(T1, T2,...)Pseudo-type forisinstance(X,...): checks ifXis an
instance ofT1or an instance ofT2, etc.pbbt.choiceof(values)Pseudo-type forisinstance(X,...): checks ifXis equal to
one of the given values.pbbt.listof(T, length=None)Pseudo-type forisinstance(X,...): checks ifXis a list of
elements ofT.pbbt.tupleof(T1, T2,...)Pseudo-type forisinstance(X,...): checks ifXis a tuple
with fields of typesT1,T2, etc.pbbt.dictof(T1, T2)Pseudo-type forisinstance(X,...): checks ifXis a
dictionary with keys of typeT1and values of typeT2.pbbt.raises(E)Use withwithclause to verify that the nested block raises an
exception of the given type.pbbt.raises(E, fn, *args, **kwds)Verifies that the function callfn(*args,**kwds)raises an
exception of the given type.pbbt.Test(CaseType)Registers the given class as a test type.pbbt.Field(check=None, default=REQUIRED, order=None, hint=None)Describes a field of an input or an output record.checkThe type of the field value.defaultDefault value if the field is missing.  If not set, the
field is mandatory.orderIf set, allows you to override the default field order.hintShort description of the field.pbbt.Record(*p_fields,**kv_fields)Base class for all input and output records.TheTestdecorator converts nestedInputandOutputclasses toRecordsubclasses.The following methods could be used or overridden:classmethod __recognizes__(keys)Checks if the set of keys is compatible with the record type.The default implementation checks if the set of keys contains
all mandatory record fields.classmethod __load__(mapping)Generates a record instance from a mapping of record keys and
values.__dump__()Generates a list of field keys and values.__complements__(other)Checks if two records are complementary input and output records
for the same test case.__clone__(**kv_fields)Makes a copy of the record with new values for the given fields.__str__()Generates a printable representation.pbbt.Control(...)Test harness.The test harness object is passed to test cases as the first
argument of the constructor.  The following methods and attributes
could be used by test case objects.Attributes:uiProvides user interaction services.trainingIf set, the harness is in training mode.purgingIf set, the harness is in purging mode.quietIf set, only warnings and errors are displayed.haltedIf set, the test process has been halted.stateConditional variables.selectionSelected suites.Methods:passed(text=None)Attests that the current test case has passed.failed(text=None)Attests that the current test case has failed.updated(text=None)Attests that output data for the current test case has been
updated.halt(text=None)Halts testing process.load_input(path)Loads input test data from the given file.load_output(path)Loads output test data from the given file.dump_output(path, data)Saves output test data to the given file.run(case)Executes a test case.__call__(input_path, output_path)Runs testing process with the given input and output.pbbt.locate(record)Finds the location of the given record.pbbt.Location(filename, line)Position of an input or output record in the YAML document.pbbt.run(input_path, output_path=None, **configuration)Loads test data from the given files and runs the tests.Configuration:uiUser interaction controller.variablesPredefined conditional variables.targetsSelected suites.trainingSet the harness into training mode.purgingPurge stale output data.max_errorsMaximum permitted number of failures before the harness halts.quietDisplay only warnings and errors.pbbt.main()Implements thepbbtcommand-line utility.pbbt.BaseCaseTemplate class suitable for most test types.Subclasses need to override the following methods:check()Runs the test case in check mode.train()Runs the test case in train mode.  The default implementation
simply callscheck().pbbt.MatchCaseTemplate class for test types which produce output.Subclasses need to override the following methods:run()Executes the case; returns produced output.render(output)Converts output record to text.pbbt.UIAbstract class for user interaction services.Methods:part()Starts a new section.section()Starts a subsection.header(text)Shows a section header.notice(text)Shows a notice.warning(text)Shows a warning.error(text)Shows an error.literal(text)Shows a literal text.choice(text, *choices)Asks a single-choice question.pbbt.ConsoleUI(stdin=None, stdout=None, stderr=None)ImplementsUIfor console input/output.pbbt.SilentUI(backend)ImplementsUIfor use with--quietoption.