Package: selftest
Selftest1. IntroductionA Simple Pythonic Self Test ToolSelftest is a simple and extensible test tool for Python. Selftest was first known as Autotest whereautomeansself. The key features are:tests areordinary functions, anywhere in the application code,gathering and running tests isautomaticand follows the structure of your code,testingstops on first failurewith a standard Python stack trace,The benefit of this approach are:there is no seprate test tree to maintain,dependencies are automatically tested,quicker cycles, more focus.The core is extremely small and support forasync,filters,operators,fixtures,diffs,wildcards,guards, etc are all implemented inhooks.HistorySelftest began, as a recalcitrant move away from the frameworks, with the following decorator above my tests:deftest(f):# the test decoratorprint(f.__qualname__)f()defi_need_testing(x):returnx@test# marks and runs the testdefa_test():assert42==i_need_testing(42)It would just run the test every time I imported it. That turned out to work so well that it grew out to what we have here today.Selftest was first named Autotest, due to conflicting names in the Python Package Index it was renamed to Selftest.FeaturesMeanwhile autotest gained some features. Itworks in a familiarPythonicway, no magic,has one simpleAPIvia the test object,is based onstandard modulesoperator, pdb, logger, difflib, inspect, etc,seamlessly scales frommicroteststosystemtests,discovers tests automatically through theimporting mechanism,crosses module, package and project boundaries easily,makes refactoring easier, even across projects, tests just move with code,executes testsimmediatelyafter discovery, in well-defined order,stops on first failure, fits into a fast REPL-style way of working,supportslevels: unit, integration, performance etc.,there arefixtures(context managers) like in other test tools,async tests and fixturesare fully supported,most functionality is inhookswhich you can extend easily,there is aroottester which can havesubtesters,output is send to alogger.Although autotest promotes an agile, rigorous and Pythonic way of testing, since there is little magic and tests are just functions, you are free to organise them as you wish. You can even do it the Python unittest way, if you want.An exampleSelftest has a global root tester that can have an arbitrarily deep and wide tree of child testers. A typical module uses it as follows:importautotesttest=autotest.get_tester(__name__)defarea(w,h):returnw*h@testdefarea_basics():assert9==area(3,3)assert6==area(2,3)Its creates a subtester usingget_tester(). The resulting tester object is the main access point to all functionality of autotest.  In this case, it is used as a decorator to mark and execute a test function.More on assert later.2. Basic APIGeneralTester objectsSelftest has a hierarchical tree of test runners calledtesterobjects, a bit like Pythons logging facility. The main program is supposed to configure the root (although it doesn’t have to) with various options. Testers lower in the tree can override these options.HooksTester objects only mark and execute tests, optionally with options. Any functionality beyond that is provided by hooks, which themselves are just options that can be overridden. There are hooks for:fixturesoperatorsasync supportfiltersdiffswildcardslevelsThese hooks introduce new functionality by extending the API of the tester object.APIThe API falls apart into five categories:a module level API,a tester object API,core options,hooks API.APIs introduced by hooksModule Level APIThe autotest core consist of two module level functions:basic_config(**options)Sets options for the root tester. This can be called only once, beforeget_tester(). If not called, default options are used. This typicalliy happens in the main of an application or in a program for running tests.get_tester(name=None)When name isNonereturns the root tester. Otherwise it returns a named child of the root.  Name is a potentially hierarchical name separated by dots. Each level in this hierarchy becomes a child of the one preceding it. The last tester object is returned. Thus,get_tester("main.sub")creates a childmainof the root and a childsubof the childmain. It returns the latter.Testers created this way become globally available. A call toget_tester()with the same name repeatedly will return the same tester.Recommended is to usetest = get_tester(__name__)at the start of your module. Using subtesters is a powerful way of organising tests. See the source code of autotest for many examples.Tester Objects APIA tester object as returned fromget_tester()support the following methods:__call__(func)A decorator for marking functions as tests:@testdeffunction_marked_as_test():passThis runs the given function and returnsNone. Thus,function_marked_as_testbecomesNoneand the function is garbage collected subsequently. Keeping the test is possible with an option.__call__(**options)A way for setting options:@test(keep=True,my_option=42)deffunction_marked_as_test():passImportant: This creates ananonymous childtester with given options. This tester is only used once and then discarded. There is a fixtures that gives tests access to it, seeFixturesbelow.If you get creative, you could also run:deffunction_not_marked():passtest(keep=True,my_option=42)(function_not_marked)# ortest(function_not_marked,keep=True,my_option=42)All methods are 100% equivalent. In fact, the full signature is:__call__(*funcs,**options)So you can run multiple test functions with the given options at once.get_child(**options)This function is an alias for__call__(**options). It does exactly the same.child(**options)This creates a child and returns a context manager.test=autotest.get_tester(__name__)withtest.child(level=CRITICAL)ascrit:@critdefa_critical_test_function():passfail(*args,**kwargs)Use as guard in tests. RaisesAssertionErrorwith the givenargs, appendingkwargstoargs.log_stats()Log the current value of the statistics to the configured output. The actual log record contains lots of data, but by default only the message is printed. See__main__.pyfor how to configure loggers.Core OptionsThe core knows four options. Hooks may support additional options. Options can be given to any of these calls:basic_config(**options),__call__(**options),get_child(**options),child(**options).Child testers inherit options from their parents and can override them.optiontypedefaultExplanationkeepbooleanFalseKeep the function instead of discarding it.runbooleanTrueRun immediately.hookslist[]List of hooks that are invoked in order.subprocessbooleanFalseRuns test when inside a subprocess.Normally, autotest runs a test as soon as it discovers it and then discards it. The example below show how tests can be run later by keeping and invoking them.@testdefthis_test_runs_immediately():passassertthis_test_runs_immediatelyisNone@test(keep=True,run=False)defanother_test_for_running_later():passanother_test_for_running_later()Tests do not run in subprocesses which are spawned because the spawn method reimports all needed modules causing all tests to run again and in an endless loop. This only happens when a test spawns of course, but is is disabled by default because these tests run in practically the same context, which add little value.Hooks APIHooks are callable objects, optionally also implementinglookup().  Selftest core only dispatches to the hooks and most useful functionality is implemented in standaard hooks.Installing a hook is done with thehooksoption.withtest.child(hooks=[my_hook])ashooked:@hookeddefsome_test():pass__call__(tester, func)A hook is an ordinary function accepting argumentstesterandfunc. It is called when a test function is discovered, usually when the tester is used as decorator. Thetesterargument supports theOptions APIso hooks can manipulate options in the current tester. It should return the samefuncor a wrapper. If it returnsNoneevaluating stops completely.As an example, here is the complete hook for filtering:deffilter_hook(runner,func):f=runner.option_get('filter','')iffinfunc.__qualname__:returnfuncNote that all hooks get to processfuncin turn, so be nice to them an usefunctools.wrapswhen you wrap.lookup(tester, name)Implemented by a hook that wants to intercept attribute lookups on the tester object. The hook can no longer be a simple function but must be an object understanding both__call__(tester, func)andlookup(tester, name). It is called when an attribute lookup takes place on the tester. When it returns a value, lookup stops. When it raises AttributeError, it continues with the next hook.As an example, here is the hook for diffs, implementing both__call__andlookup(references to diff and print functions omitted for clarity):classDiffHook:def__call__(self,runner,func):returnfuncdeflookup(self,runner,name):ifname=='diff':returndiffifname=='diff2':returndiff2ifname=='prrint':returnprrintraiseAttributeErrorlogrecord(tester, func, record)Implemented by a hook when it wants to change something in the logrecord before it is emitted to the logger. Thelevelshook uses this to add the level of the test to the logrecord as an extra attribute, for example.Options APIThe Options API is meant for hooks manipulating options. Options are hierarchically registered, that is, each tester can have local values for options, and looks up missing ones in its parent.option_get(name, default=None)Returns the value for the option with given name for this tester or its closest parent.option_setdefault(name, default)Get option with name, searching all parents. When missing, sets the option onthistester withdefaultas value and return it.option_enumerate(name)Enumerates all values for the option with the given name, starting with this tester, up to all its parent. List and tuple values are reversed and flattened (concatenated).3) APIs from HooksOperatorsHookoperator.pyintroduces the possibility to use various builtin operators instead of theassertstatement. As a last resort, it looks up methods of the first argument to use as asserting statement. For example:@testdefanother_test():test.all(x>1forxin[1,2,3])# use builtin all()test.startswith("rumbush","rum")# use method of first argumentWhen the given operator returnsFalseaccording tobool()it raisesAssertionErrorwith the actual values of the arguments.This shows how autotest stays close to Python as we know it. It does nothing more than looking up the given attribute in four places:moduleoperator, e.g.:test.gt(2, 1),modulebuiltins, e.g.:test.isinstance('aa',str),moduleinspect, e.g.:test.isfunction(len),the first argument, e.g.:test.isupper("ABC").The benefits of this is that we do not have to learn new methods, that the assert functions are not limited, and that autotest can print the arguments for us on failure.diffAll operators obtained this way support a keyworddiff=<function>that, when present, is invoked with the actual arguments. The result is then given to theAssertionErrorinstead of the actual arguments.@testdefanother_test():a={7,1,2,8,3,4}b={1,2,9,3,4,6}test.eq(a,b,diff=set.symmetric_difference)The code above will raiseAssertionErrorwith as argument:{6, 7, 8, 9}.For more general purpose diff functions, see the hookdiffs.py.Operators is included in the default root tester.FixturesHookfixtures.pyintroduces fixtures as seen in other test tools. Thetest.fixtureattribute registers the next function as a fixture. A fixture is a Pythoncontextmanagerand can be used as such, or it can be specified as a formal argument to a test function. Fixtures accept arguments themselves by using the ‘:’ notation.@test.fixturedefanswer(a=42):yieldawithtest.answerasp:# as a context managertest.eq(42,p)@testdefprobe_the_universe(answer):# as a formal argumenttest.eq(42,answer)@testdefsomething_wrong(answer:43):# with argument 43test.ne(42,answer)test.eq(43,answer)There are standard fixtures for:test - gives access to the current tester, useful for anonymous subtesters,stdout - capturessys.stdout, including that of subprocesses, in aStringIO,stderr - capturessys.stderr, including that of subprocesses, in aStringIO,tmp_path:subpath - creates a temporarypathlib.Pathobject, optionally with a subpath,raises:(Exception, message) - raises AssertionError if given code does not raise given exception with given message,guard - isolates tests by saving and restoring sys.path, sys.meta_path, and sys.modules.An example for usingraises()in two different ways:@testdefshould_raise_error(raises:(AttributeError,"'list' object has no attribute 'a'")):[].awithtest.raises(AttributeError,"'list' object has no attribute 'a'"):[].aFixtures can be async (async def) but async fixtures can only be used in async tests. Async fixtures are executed in the event loop of the async test they are declared for.Fixtures is included in the default root tester.FilteringHookfilter.pysupports the optionfilter=<str>and only executes test whosequalified nameincludes the given<str>.withtest(hooks=[filter_hook])asftest:withftest(filter='moon')asmoon:@moondefphase_of_the_moon_bug():passFiltering is included in the default root tester.DiffsHookdiffs.pyprovides the attributes:test.diff(a b)– a Pythonpprint+difflibbased general purpose diff for use with the operator hook,test.diff2(a, b)– an Selftestprrint+difflibbased diff forPlain Old Data(POD) objects,test.prrint(a)– a pretty printer for POD objects. Use instead of Pythonspprint().Diffs is included in the default root tester.AsyncHookasyncer.pysupportsasynciotests defined withasync def. Async tests can contain other async tests, however due to limitations in Python (being that async is partially a syntax feature and not fully dynamic) this forces nested async tests to be executed in a separate event loop in aseparate thread.Non-nested async tests are run in a new event loop in decurrent thread.The optiontimeout=<timein s>specifies the maximum amount in seconds a test can run. After that anTimeoutErroris raised.The optionslow_callback_duration=<timein s>specifies the time after whichasynciobegins to emit warnings about tests running too long.Async is included in the default root tester.WildcardsHookwildcard.pyintroduces the attributetest.anywhich can be used in structured data comparisons as a wildcard. Its matching can optionally be limited using a function as argument. It is nice to combine this with the operator hook:test.eq([4,test.any,42],[4,-3,42])# succeedstest.ne([4,test.any(test.pos),42],[4,-3,42])# failsWildcards is included in the default root tester.LevelsHooklevels.pyintroduces test levels such asunit,integrationetcetera, together with two options,levelandthreshold, to control
it. It is meant to run only certain tests depending on the context. During
development for example, for reasons of speed, integration and performance
tests can be skipped. The levels are just numbers and a number functions as a
level/threshold, much like as in Pythonslogging.The levels are:levelvaluecritical50unit40integration30performance20The default level isunit. Test levels are provided as attributes on the tester:@test.criticaldefa_critical_test():passTests can also be put at a certain level with an option:@test(level='critical')defa_critical_test():passwithtest.child(level='critical')ascritical:@criticaldefone():passThe defaultthresholdisintegration.The Levels hook adds the following information to any logrecord being generated:the number of the test level (20..50) , astestlevelthe name of the test level (‘unit’, ‘integration’, etc) astestlevelnamethe name of the test level is also prepended to themsgattribute of the logrecordA parent can have thethresholdoption set to one of the levels. This will block execution of tests in these children of levels lower thanthreshold. Note that tests do not have levels, only Testers have.NB:the highestthresholdin chain of parent-children determines the tests to run. This means a child tester can run integration tests (threshold=’integration’), but the root tester can overrule this to run only unit tests.Levels is included in the default root tester.Extended closureThe hookbinder.pyenables binding to a class definition in the making. The namespace of a class being defined is not available inside functions being defined in the class body. See this example:deffunction_a():a=42deffunction_b():asserta=43function_b()# runs finefunction_a()classclass_a:a=42deffunction_b():asserta=43function_b()# NameError: name 'a' is not definedIn order to be able to embed tests in class definitions, the binder hook extends the binding of freevars in test functions to include those of the enclosing class. Sorry for the magic.This hook is enabled by default, but only performs binding when the optionbind=Trueis present.classclass_a:a=42@test(bind=True)deffunction_b():asserta=433. Running TestsDevelopmentRunning tests during development can be done by just running or importing your module:$python<mymodule.py>$python-c"import mymodule"When you only want to develop a submodule, justcddown into that directory and do the same. Only the tests of that submodule (and everything in imports) will be tested.The methode above just prints crude messages and has no way to use options. For that use the main that comes with autotest:$autotest--helpUsage:autotest[options]moduleOptions:-h,--helpshowthishelpmessageandexit-fFILTER,--filter=FILTERonlyruntestswhosequalifiednamecontainsFILTER-t,--threshold=THRESHOLDonlyruntestswhoselevelis>=THRESHOLDFor example to run your tests but not the imported ones from other packages:$autotest--filtermymodulemymoduleIf you want to run the tests for autotests itself, go to the autotest project directory and use:$python-c"import autotest"autotest.selftestThe argumentautotest.selftestlets autotest run its own tests, which are normally skipped.
Also, this avoids using the autotest main program because it is not guaranteed that an installed old version of autotest can run its own test from the future.ProductionDuring production, all tests are automatically run during startup when all needed modules are imported. If an the application configures a specific root, for example by callinglogging.basicConfig(), the tests will automatically log there. Alternatively, you can setup a separateLoggerfor running tests. See Core API.You can als filter tests or run tests for a specific level only. Or suppress them all. See the source code of__main__.pyfor ideas.4. MiscTODOalso implement timeout for synchrounous codelog stats without the loglevel