Package: testharness
===================Test Harness Readme===================:author: Oisin Mulvihill.. contents::Introduction============This is intended to be used as a generic functional test framework.This uses a simple script system, based on the python console, to load and run commands from testmodules. The test modules are python classes that implement the details of how to test the project orprogram. The module provides a simple set of functions that can be called from a script. The functionscan for example, take care of the complexity of starting some service or actually setting/getting somevalue in a database. In the script each function call can then be asserted to test what's returned. Thisthe is used to pass or fail a test.I've used this project to test interactions between network services, database driven applications, GUItesting via pyGTK or in many other situations. I use it mainly to implement user acceptance stories toverify the behavior of a some user functionality. I don't use it to test web pages as there are betterthings out there for this (I use selenium remote control).I originally developed this project back in 2001 and I've used it to test a few commercial products, thatmy company and others have developed. I've got several versions of this project in various places. I'veits time to bring them all together into one release. This is also an exercise for to learn how to writepackages using setuptools.Running the test harness========================Currently the test harness is run out of the pytester directory. The followinginstructions apply only to this situation.* Copy the file ./lib/testharness/config/tester.ini.in to ./tester.ini* Copy the file ./lib/testharness/testharness_log.ini.in to ./testharness_log.ini* Edit the tester.ini if needed.* Edit the log.ini if needed.* From the command line issues the following to run the test harness. All the tests mentioned in the tester.ini will now be run one after another.::PYTHONPATH=lib python ./lib/testharness/scripts/runacceptanceOrgoharnessCommand line arguments======================The test harness takes a number of command line arguments.-c, --config <config_file>--------------------------This specifies the configuration file the test harness is going to use. Thisfile is required and if it is not specified, then the file "tester.ini" islooked for in the current path.-l, --logconfig <log_config>----------------------------This specifies the logging configuration use. This file set up where logmessages go, etc. By default the test harness looks for a file called"log.ini" in the current path.-i, --interactive-----------------This tells the test harness not to start running tests mentioned in thetester ini file, but instead to go into interaction mode. This allows theuser to type the commands in manually.--testpath <path_to_tests>--------------------------This tells the test harness where to find all the tests and test modules itis going to use. If this is not specified the the testharness will look forthe "acceptancetests" directory in the current path.tester.ini configuration-------------------------This file is used to configure the test harness and any modules that areloaded. The TestHarness section contains, as you might expect, settingsused by the test harness. The following is a typical tester.ini::[TestHarness]assert_equals_failure_causes_exit = yestests = my_default_tests_to_run.scriptTestHarness section-------------------assert_equals_failure_causes_exit+++++++++++++++++++++++++++++++++This tells the test harness to stop processing a test script if an assertEqualsfails in a test. If this was "no" the test harness would try to continueprocessing the script. A "yes" value here is the default, "no" is only used indebugging and is not recomended in normal situations.tests+++++This is a list of scripts the test harness will run when it starts. These arefile names seperated by spaces and all on the same line. Currently you cannotspan multiple lines. Note also that the test harness expects to find thesefiles in the --testpath <path_to_tests> directory.Test scripts============All tests usually start with \*.script. Thats just a convention I started using.The files can be called whatever you want.Test script and test harness console commands---------------------------------------------The test harness has a number of built in functions that can be called. Thesefunctions can be called from a script file or from the console. There is no realdistinction.load <module name>------------------This loads a test module into the test harness. Test harness modules are pythonscript files that contain classes derived from TestBase. When you specify themodule you don't specify '.py' extension.*Note*Only class derived from TestBase will be loaded everything else is ignored.help [<function name>]----------------------This function gives help on a command name or lists all functions in theTestHarness.exit----This tells the test harness to exit after call shutdown.rem----This is only really useful inside a script file. Any line starting with this isignored.print <string to be printed>----------------------------This is only really useful inside a script file. This function prints a line oftext to stdout.shutdown--------This function is usually called just before the test harness exits. Thisfunction goes through each loaded module and calls its start() method.start <module name>-------------------This function call a loaded modules start() method.stop <module name>-------------------This function call a loaded modules start() method.list----This function lists all loaded modules in the test harness.inspect <module name>---------------------This function calls a modules inspect() method to print out a list of availablefunction calls.Note: It is up to module writer to actually write the information that isreturned for printing.evaluate <module name> <method call> <arg 1>,<arg 2>,<arg 3>,...<arg X>-----------------------------------------------------------------------This function calls a method in a module. Normally this function isn't called bythe user directly. Instead anything which isn't a built in function is assumedto be a module method call.For example:A module called 'mytest' has one class called 'MainTest'. The 'MainTest'function has a method called 'doMyTest1' and it takes two arguments, a numberand a string.This could be invoked using the evaluate function as follows::evaluate mytest.MainTest doMyTest1 123,this is a stringHowever the following also works::mytest.MainTest doMyTest1 123,this is a string*Notes*Arguments are seperated by commas and these cannot be used by the user inthe script.Arguemts should have no white space before or after them, unless you wantthis as it will be passed to the function.For example::doMyTest1 123,this is a string,4321doMyTest2 123, this is a string  ,4321 ,doMyTest1 will passed the args "123" "this is a string" and "4321"doMyTest1 will passed the args "123" " this is a string  " and "4321 "assertEquals <error message if fails>,<compare value>------------------------------------------------------This function compares the provided <compare value> string, with that returnedby the last user module script call.For example::mytest.MainTest doMyTest1 123,this is a stringThis call returns a number. This function is converted to a string and stored bythe test harness. Now the assertEquals can be used to test the return as follows::assertEquals doMyTest1 didn't work,12If the returned value from doMyTest1 was 12 then assertEquals will pass. IfdoMyTest1 didn't return 12 then the message 'doMyTest1 didn't work' will beprinted.*Note*If the test harness assert config option is 'yes' the a failingassertEquals call will cause the test harness to stop running the script andreturn with a failure code.