Package: end2
end² Test Automation FrameworkThe focus of this framework is:A minimal framework only using the standard libraryMore for E2E and/or Functional type of testingFor testing that has heavy logging and needs to analyze failures in logs rather than test case codeFor folks that like programatic ways instead of plugins with configuration filesContentsIntent/PhilosophyFeaturesGetting StartedCLIResource FilesLog ManagerReserved KeywordsIntent/PhilosophyShuffling:By having tests run in random order, we are ensuring that tests don't need to run in a specific order. If test-1 fails, then test-2 will obviously fail, but test-2 is a false negative. It might be better to consider test-1 and test-2 as test steps and just combine test-1 and test-2 in one test case instead. Another plus to Shuffling is the test writer will be able to find out if there are any side effects on the test case side or the SUT and be able to fix what is necessary. This will make them have a better understanding of there own coding, others members coding, and the SUT as well if the side effect is on the SUT itselfCreate you own script entry point:This is the entrypoint for your testing. It is your own python module that you will write that defines what the test parameters are and usesdefault_parser()to add any additional args before you start your testing. You can name it whatever you want but in below examples I refer to it asrun.pyDeclaring:Test case design is very important and the design should speak for itself in the file/module. Declaring the concurrency/run-mode in the file lets everyone know that that particular file can run in parallel. Passing that info in the command line can be confusing over time because not everyone will remember what can and can't run parallel1 set of parameters per suite:When we do a suite run we are only testing 1 system, therefore whatever is needed to communicate to the system should be the same throughout all test cases in that suite. As a result parameters should be the same for all test cases. This always helps keep test cases dry and makes them more step focusedRoot of truth:Single source of truth is a very good thing to have, when the single source is up-to-date and working then everyone will know it is 100% accurate information. By having your test cases as the single-source of truth, you can then publish your truth anywhere necessary and that destination will always have the info of the latest results. So the test cases should speak for themselves and have any doc strings necessary so that everyone can view the latest version of your testingFeaturesTest Runner:Discovers tests at runtimeTest Pattern Matching: Can run individual tests and test modulesRuns tests sequentially and parallelly in 1 runTest FixturesTest Reserved KeywordsTest Module WatcherFixtures:setup packageteardown packagesetup moduleteardown modulesetup testteardown testmetadataparameterizeLogging:Records are timestampedAssertion failures are logged at[ERROR]It will hold folders from the last n test runsEach test module will be in its own folderEach test will be in it's own fileFailed tests will be renamed toFAILED_<test_name>.logHAR loggerGetting StartedUnderstanding the end² Flow (Psuedo Code)defdiscover_package(parent_path):paths=get_all_paths(parent_path)modules=[]forpathinpaths:ifis_dir(path):modules+=discover_package(path)else:modules.append(discover_module(path))returnshuffle(modules)defdiscover_module(path):module=import_module(path)forfunctioninmodule:ifis_test(function):module.add_test(function)returnshuffle(module.tests)defrun_tests(discovered_modules):forpackageindiscovered_packages:package.setup()formoduleinpackage.discovered_modules:module.setup():fortestinmodule.tests:module.setup_test()args,kwargs=test_parameters(logger,package_object)test(*args,**kwargs)module.teardown_test()module.teardown()package.teardown()Simple Example of a Run script#!/usr/bin/env python3fromend2.runnerimportstart_test_runfromend2.arg_parserimportdefault_parserif__name__=='__main__':args=default_parser().parse_args()# You can add your own arguments to default_parser if you want before you# call parse_args()deftest_parameters(logger,package_object)->tuple:# This is how parameters for tests are injected. Whenreturn(create_client(logger),),{}# overriding this you must always return a tuple of tuple# and dict. The logger arg here will be the logger# specific to the test. This method will be called# on every fixture and testtest_suite_result,failed_imports=start_test_run(args,test_parameters)exit(test_suite_result.exit_code)Simple Example of a Test ModuleIn order for a method to become a discoverable test you must prefix your method name withtest_. Each test method will have the same parametersfromend2importRunMode__run_mode__=RunMode.SEQUENTIAL# This is required for every test moduledeftest_1(client):assertclient.get('Hi')isnotNone# assert is used for validation; if assertion fails the test fails and exits on that assertasyncdeftest_2(client):# Both sync and async test methods can exist in the same fileactual=awaitclient.get_stuff()assertactual=="some expected data"defhelper():# Not a test methodreturn{'a':1}Simple Example of Checking Test Case Readiness at Runtimefromend2import(IgnoreTestException,RunMode,SkipTestException)__run_mode__=RunMode.SEQUENTIAL# This is required for every test moduledeftest_1(client,logger):ifnotclient.something_ready():raiseIgnoreTestException()# You may ignore tests are runtime if necessary. No test result will be madeassertclient.get_stuff()logger.info('Hi')asyncdeftest_2(client,logger):# Both sync and async test methods can exist in the same fileifnotclient.something_else_ready():raiseSkipTestException("thing not ready")# You may skip tests are runtime if necessary as well.actual=awaitclient.get_stuff()# A test result will be made with status of skipped and theassertactual=="some expected data"# message of what was supplied in the SkipTestException()logger.info('Hi async')Fixture Example of a Test Modulefromend2import(on_failures_in_module,on_test_failure,parameterize,RunMode,setup,setup_test,teardown,teardown_test)__run_mode__=RunMode.SEQUENTIAL# This is required for every test module@setupdefmy_setup(client):client.do('something during setup')@setup_testdefmy_setup_test(client):client.do('something during setup test')@teardown_testdefmy_teardown_test(client):client.do('something during teardown test')@teardowndefmy_teardown(client):client.do('something during teardown')@on_failures_in_moduledefmy_teardown(client):# Runs once at the end of the test module if you have 1 or more failed test casesclient.do('something')# Parameterize takes 1 argument: list of tuples#  - Each tuple must be the same length@parameterize([('A','B','AB'),(1,2,3),([],[1],[1]),(1.2,2.3,3.5),(True,False,1)])deftest_1(var1,var2,rhs):# Parameterized parameters will come in after all runner.test_parametersassertvar1+var2==rhs@metadata(defect_id='SR-432',case_id='C-23451')# Use metadata when you want to add extra info to your testdeftest_2(client):# This data will also be available to you after the test runassertTrueisTrue@metadata(tags=['yellow','potato'])# tags is a special keyword used for Pattern Matching. As long as atdeftest_3(client):# least 1 tag matches test will run (when using --suite-tag)assertTrueisTruedefcleanup(client):client.do('some cleanup')@on_test_failure(cleanup)# This fixture will run the function in the decorator argument only if the test failsdeftest_4(client):assertTrueisTrueReserved KeywordsThese are optional keyword-only-args that can be added at the end of your test case parameters:end- This is helpful if you have event handling in the app you are testing and need the callback to be called. Only use this if you have to wait for some event otherwise you test will just timeout ifendis not called:deftest_4(client,*,end):defhandler:assertTrueisTrueend()# ends the test caseclient.onSomeEvent(handler)# This test will not finish until end() is called or has timeoutdeftest_4(client,*,end):defhandler:assertTrueisTrueend.fail("This event should not have been called")# ends the test caseclient.onSomeEvent(handler)# This test will not finish until end.fail() is called or has timeoutlogger- The logger used for that specific test casestep- This is so you can record test steps in your test case, that may be useful after your test rundeftest_5(client,*,end):# 1st arg is the description of the step# 2nd arg is the assertion-lambda, which can be None# 3rd arg is the function to call# nth args are the parameters for the functionawaitstep("my first step",lambdax:x.code==201,client.post,{'hi':21})response=awaitstep("my second step",None,client.post,{'hi':22})awaitstep("my third step",None,client.post,{'hi':23})assertresponse.code==201# Works with async as wellasyncdeftest_6(client,*,end):awaitstep("my first step",lambdax:x.code==201,client.post,{'hi':21})response=awaitstep("my second step",None,client.post,{'hi':22})awaitstep("my third step",None,client.post,{'hi':23})assertresponse.code==201package_object- More on this in the next sectionPackages ObjectThis is an object that you can build from within your packages. Since test parameters are always fresh objects you may want to pass data around and be able to access it in packages. This feature is kind of experimental but here are some ideas:Build reports in the middle of runsBuilding metricsExample of Test Package# test_package/__init__.pyfromend2import(setup,teardown)fromend2.fixturesimportpackage_test_parameters@setupdefmy_setup(package_globals):package_globals.stuff=['my_static_stuff']@teardowndefmy_setup(package_globals):package_globals.stuff.clear()@package_test_parametersdefmy_custom_test_parameters(logger,pacakge_object):# Use if you want to override the test_parameters definedreturn(some_other_client(logger),){}# in your 'run.py'# test_package/test_sub_package/__init__.pyfromend2import(setup,teardown)@setupdefmy_setup(package_globals):package_globals.stuff# will be ['my_static_stuff']package_globals.sub_package_stuff=['other stuff']@teardowndefmy_setup(package_globals):package_globals.sub_package_stuff.clear()# test_package/test_sub_package/my_test_module.pyfromend2importRunMode__run_mode__=RunMode.PARALLEL# This is required for every test moduledeftest_1(client,package_globals):assertpackage_globals.stuff==['my_static_stuff']assertpackage_globals.sub_package_stuff=['other stuff']Test GroupsTest groups allow you to organize your tests around setup and teardown. Maybe some of your tests the setup only needs to be run for 2 of your tests. Or maybe you want the same setup for all tests but you want an additional setup for 4 of the tests. Groups are declared as classes and the methods are techincally static but without decorating with@staticmethod# test_package/test_sub_package/test_module.pyfromend2import(RunMode,setup_test,teardown)__run_mode__=RunMode.PARALLEL# This is required for every test module@setup_testdefsetup_all(client):pass# do something at the start of each test.deftest_1(client):assertpackage_globals.stuff==['my_static_stuff']assertpackage_globals.sub_package_stuff=['other stuff']classGroup1:@setup_testdefsetup_all1(client):pass# do an extra something after setup_alldeftest_2(client):passclassGroup2:@setup_testdefsetup_all2(client):pass# do an extra something after setup_all and setup_all1deftest_2(client):passCLIIt is best to run the--helparg on your "run.py" to get the latest information. SincePattern Matchersare a little more complicated below is a more desciptive overviewSuite Pattern MatchersDefaultA suite path is a string that contains the path to the module delimited by a period:To run a single test package:--suite path/to/packageTo run a single test module:--suite path/to/file.pyTo run multiple test packages:--suite path/to/package path2/to/packageTo run multiple test packages and modules:--suite path/to/package path2/to/package path3/to/package/module.py path4/to/package/module.pyIt can also contains filters:::which is to run specific tests in a module:--suite path/to/package/module.py::test_1;which is delimiter for modules:--suite path/to/package/module.py;module2.py,which is a delimiter for tests:--suite path/to/package/module.py::test_1,test_2;module2.py!which means run everything before the!but nothing after:--suite path/to/package/!module.py;module2.pyruns everything inpath/to/packageexceptmodule.pyandmodule2.py--suite path/to/package/module.py::!test_1,test_2;module2.pyrunsmodule2.pyand everything inmodule.pyexcepttest_1andtest_2[n]which will run specific parameterized tests:--suite path/to/package/module.py::test_name[1]runs the 2nd test in the parameterized list--suite path/to/module.py::test_name[2:6]runs tests 2 through 6 in the parameterized list--suite path/to/module.py::test_name[2:6:2]runs the 2nd, 4th, 6th test in the parameterized listTagsTags can be defined by using@metadatain you test as mentionedaboveor at the module. They works pretty similar to theDefault Pattern Matcherbut uses a tag instead of a test name:--suite-tag path/to/module.py::tag_1,tag_2This will include all tests iftag_1ortag2exist in__tags__variable inpath/to/module.pyor the metadata decorator includes the tags field with the mentioned tags--suite-tag path/to/package/tag1,tag2This will include any module that hastag1ortag2exist inpath/to/package--suite-tag path/to/package/tag1,This is the same as above, but how you would use only 1 tag in your string (notice comma at the end)regex and globThese 2 are pretty similar to each and I split module and test the same:--suite-regex <regex for module>::<regex for test>--suite-glob <glob for module>::<glob for test>Last FailedYou can also run only the tests that failed in the last run--suite-last-failedResource Files.end2rc: defines a default value for cli as well as:Aliases: a short name given to a suite that is long. Aliases can also mention other aliasesDisabled Suites: The is a list of disabled suites/tests; this way you don't have to remember which ones to disable. Also the list of suites/tests are centralized here; you won't have to hunt them down in each filelogs/.lastrunrc: defines a list of tests that failed in the last runLog ManagerALog Manageris meant to help organize your logging into timestamped folders that rotate every n number of folders. You can subclassLogManagerif you want, or use the default own. You can use this if you have other tools in you repo that have logging as wellDefault Suite Log ManagerFor Suite runs you will use aSuite Log Manager. The default does what is described below and you can also subclassSuiteLogManagerif you want:Rotates your suite run log foldersLogs INFO to stdinLogs INFO to a standalone file as well and it is not interlacedHas a delimiter for both modules and testsHandles events before and after on:suitemodulesfixturestestsCreates a log subfolder for each moduleCreates a file for both setup and teardown of a moduleCreates a log file for each testMarks (Prefixes) file name as PASSED, FAILED, SKIPPED when test is finished