Package: pytest-adaptavist
pytest-adaptavistThispytestplugin generates test execution results withinZephyr Scale, formerly known as Adaptavist.Table of Contentspytest-adaptavistTable of ContentsInstallationRestrict reportingGetting StartedExamples and FeaturesGeneral WorkflowExamplesContext reportingTest Case OrderExamplesTest Case RangeExamplesSkipping vs. BlockingUsing markersUsing metablock action modesUsing pytest methodsCallbacksInstallationTo install pytest-adaptavist, you can use the following command:$python-mpipinstallpytest-adaptavistTo uninstall pytest-adaptavist, you can use the following command:$python-mpipuninstallpytest-adaptavistTest case selection and ordering (see below) are supported by default.In order to access Adaptavist/Jira and create test runs and results in there, provide credentialsJIRA_SERVER, JIRA_USERNAME, JIRA_PASSWORDas environment variables and run your tests with$pytest--adaptavistRestrict reportingIf you want to report only if a certain local user was used to execute the test (e.g. only report, if the CI user was used), you can use$pytest--adaptavist--restrict-userjenkinsHowever, if this local user does not exist as user in Jira, the "Executed by" field in Jira will stay unassigned.If you want to report only if a certain branch was used to execute the test (e.g. only report, if using the main branch), you can use$pytest--adaptavist--restrict-branchThis defaults to the master branch, but you can change the branch.$pytest--adaptavist--restrict-branch----restrict-branch-namedevelopmentYou can also apply this settings to a pytest.ini:restrict_user=jenkins
restrict_branch=1
restrict_branch_name=developmentGetting Startedpytest-adaptavist searches for test methods named liketest_<test_case_key>ortest_<test_case_key>_<step>wheretest_case_keyis the key of the Jira test case excluding the project key (e.g. "T1") andstepdefines a single test script step (if existing). In order to build real test case key strings from test methods, the corresponding project key needs to be specified for each relevant class or single test methods by using markers (seeexamplesbelow). Alternatively,test_case_keycan be given as it appears in Adaptavist, but with hyphens replaced by underscores (e.g. "TEST_T1"). Each of these kind of test methods is marked as Adaptavist test case for reporting appropriate results into Adaptavist test management. Any other test methods are processed as usual. If you don't want to stick to this convention, you can decorate any test method with the testcase marker and specifyproject_key,test_case_keyandtest_step_keythere.Finally, pytest-adaptavist needs eithertest_run_keyto use an existing test run orproject_keyto create a new test run every time with collected test cases linked to it. In order to work properly, either of these parameters need to be specified at the very start of the test session. If both parameters are empty, neither test runs nor test results are created in Adaptavist test management. Please also note that any of these parameters mentioned here and in the following documentation can either be set programmatically by storing them in the environment, be defined in pytest.ini or be provided as part of a json config file (./config/global_config.json).Examples and FeaturesGeneral Workflowpytest-adaptavist collects test cases (and single test steps) as mentioned above and prepares them for Adaptavist reporting.test_run_keyis used to specify an existing test run. In this case, you must either link collected test cases to that test run in Adaptavist. Not linked but collected test cases are skipped. Or you have to use theappend_to_cycleoption to automatically append found test cases that are not linked yet.Alternatively, ifproject_keyis given andtest_run_keyis left empty, pytest-adaptavist creates a new test run every time with collected test cases linked to it. In this case,test_run_suffixcan be used to create a meaningful test run name. In addition,test_plan_keyis available to link the new created test run to an existing test plan.New test plans can be created by specifyingtest_plan_suffixwhich is used as identifier to find existing test plans and must be unique. If there is a matching test plan, it will be used for creating new test runs. Else a new test plan is created within the given project and new test runs are linked to that. If both test plan key and suffix are missing, test runs are created just for the given project.Default naming convention for new test plans and test runs within pytest_adaptavist:new test plans are named like<project key> <test plan suffix>new test runs are named like<test plan name or project key> <test run suffix>If no test run suffix is set,test run <current date>will be used.If you want to use a different schema, you can configuretest_run_nameortest_plan_namein your pytest.ini and use the following placeholders:project_keytest_plan_keytest_run_keytest_environmenttest_plan_foldertest_run_foldertest_plan_suffixtest_run_suffixlocal_usere.g.test_run_name = %(project_key) Regression Test on %(test_environment)test_case_keyscan be used as an option to run only a subset of implemented test cases. All others are skipped in this case. For new created test runs these test cases are excluded while for existing test runs the appropriate test case results stay as they are (if existing).In addition,test_case_keysmay contain test cases that are not implemented in the current python test script. This can be useful in cases where the new test run also needs to include manual test cases (e.g. for later execution). Furthermore, it is even possible to just create a new test run with only test cases that are not (yet) implemented.If either of these parameters is missing, pytest-adaptavist tries to read appropriate values from pytest.ini or config file (global_config.json).Specifying a project key for relevant test classes or test methods can be done by using markers:@pytest.mark.project(project_key="my project")If project markers are not used, pytest-adaptavist is usingproject_keyto build test case key strings.To send additional data (comments, attachments) to Adaptavist, test methods can be extended by using plugin's meta data parameter.(Note that for blocked or skipped methods, attachments will be ignored and only comments will be added to the test result).ExamplesThis simple snippet shows some implementation of step 2 of test casemyproject-T1usingmeta_data:@pytest.mark.project(project_key="myproject")classTestClass(object):deftest_T1_2(self,meta_data):meta_data["comment"]="unexpected result"attachment=io.StringIO()attachment.write("this is just a simple attachment")meta_data["attachment"]=attachmentmeta_data["filename"]="content.txt"assertFalseWith each test step report the status of overall (or parent) test result is adjusted automatically, depending on the status of all other test step results.If a step fails, the status of the overall test is set toFailand stays atFailno matter of upcoming results within that test run. Furthermore, an appropriate information about failure is appended to overall test result as additional comment. Given the example above, there would be astep 2 failed: unexpected resultnote added to the overall test result comment.If there is a test method for the overall test, its (optional) comment is prepended to the existing comment. Given the following example, the final comment for test casemyproject-T1would beall good<br />step 2 failed: unexpected result:@pytest.mark.project(project_key="myproject")classTestClass(object):deftest_T1_2(self,meta_data):meta_data["comment"]="unexpected result"attachment=io.StringIO()attachment.write("this is just a simple attachment")meta_data["attachment"]=attachmentmeta_data["filename"]="content.txt"assertFalsedeftest_T1(self,meta_data):meta_data["comment"]="all good"assertTrueHowever, the final status would beFailbecause not all test steps passed.The same example using the testcase marker:@pytest.mark.project(project_key="myproject")classTestClass(object):@pytest.mark.testcase(test_case_key="T1",test_step_key=2)deftest_unexpected_result(self,meta_data):meta_data["comment"]="unexpected result"attachment=io.StringIO()attachment.write("this is just a simple attachment")meta_data["attachment"]=attachmentmeta_data["filename"]="content.txt"assertFalse@pytest.mark.testcase(test_case_key="T1")deftest_all_good(self,meta_data):meta_data["comment"]="all good"assertTrueContext reportingIn addition to implementing test methods for single test steps, it is possible to combine them within the appropriate test case method by using context blocks:classTestClass(object):@pytest.mark.project(project_key="myproject")deftest_T1(self,meta_block):# processing step 1withmeta_block(1)asmb:mb.data["comment"]="unexpected result"attachment=io.StringIO()attachment.write("this is just a simple attachment")mb.data["attachment"]=attachmentmb.data["filename"]="content.txt"pytest.assume(True)# processing step 2withmeta_block(2)asmb:mb.data["comment"]="unexpected result"attachment=io.StringIO()attachment.write("this is just a simple attachment")mb.data["attachment"]=attachmentmb.data["filename"]="content.txt"pytest.assume(False)@pytest.mark.project(project_key="myproject")deftest_T2(self,meta_block):# processing test case onlywithmeta_block()asmb:mb.data["comment"]="all good"attachment=io.StringIO()attachment.write("this is just a simple attachment")mb.data["attachment"]=attachmentmb.data["filename"]="content.txt"pytest.assume(True)It is highly recommended to use pytest.assume instead of assert in this case, as assert would immediately abort from current test method just skipping any other steps or context blocks. Alas, there might be cases, where it does make sense to abort though. In these cases, using assert is the better choice.To simplify the overall handling, there's a helper method that combines meta data (e.g. comments, attachments) and assertion. Basically it checks given condition, aborts or continues appropriately and adds comments and attachments optionally:classTestClass(object):@pytest.mark.project(project_key="myproject")deftest_T2(self,meta_block):withmeta_block()asmb:# if precondition is False, stop session (results in status "Blocked" for this and any following test cases)mb.check(precondition,message="precondition failed",action_on_fail=mb.Action.STOP_SESSION)# if precondition is False, stop further execution of this test (results in status "Blocked")mb.check(precondition,message="precondition failed",action_on_fail=mb.Action.STOP_METHOD)withmeta_block(1)asmb1:# if condition is False, just collect failed assumption and continue (just like 'assume', results in test step status "Fail")mb1.check(condition,message="condition failed",action_on_fail=mb1.Action.FAIL_CONTEXT)# if condition is False, stop further execution of this step (results in status "Blocked")mb1.check(condition,message="condition failed",action_on_fail=mb1.Action.STOP_CONTEXT)# if condition is False, stop further execution of this test (just like 'assert', results in status "Fail")mb1.check(condition,message="condition failed",action_on_fail=mb1.Action.FAIL_METHOD)withmeta_block(2)asmb2:attachment=io.StringIO()attachment.name="context.txt"attachment.write("this is just a simple attachment")# default: if condition is False, just collect failed assumption and continue (just like 'assume', results in test step status "Fail")mb2.check(condition,message="condition failed",attachment=attachment)# attachment is added anywayIn addition, this method supports the following parameters:message_on_passwhich can be used to report passed conditions as wellmessage_on_failwhich is the same asmessage(just for convenience)descriptionas option to add details about test results (f.e. can be a html table or more)Test Case OrderBy default the execution of test methods in pytest runs alphabetical over test class names and then from top to bottom.With pytest-adaptavist this order can be changed by usingtest_case_order(ortest_case_keys, see also above). In this case the corresponding test methods are executed according to the given order, followed by all remaining test methods. Moreover, this can also be used when creating test runs automatically, as in this case the new test run is created with test cases linked in the given order.Alternatively, if an existing test run is specified bytest_run_key, the corresponding test methods are executed according to the order of test cases in the given test run, followed by all other test methods.Note thattest_case_orderoverrules the test case order of the given test run as well as the order specified bytest_case_keys. This might be helpful in cases, where the default order should be changed temporarily. Iftest_case_orderis not specified, the order will be as defined bytest_run_keyor - if a new test run should be created -test_case_keys.ExamplesAssume there is a project TEST with exactly two test cases TEST-T1 and TEST-T2 while a test implementation contains methods in the following order (top to bottom):test_T4,test_T1,test_T2,test_T3,test_myspecialtest.All these methods will be executed in the same order (from top to bottom). But a potentially new created test run will only contain TEST-T1 and TEST-T2 (in this order). And only the results of TEST-T1 and TEST-T2 will be send to Adaptavist.For the next run a test case order is specified like TEST-T2, TEST-T3, TEST-T18.
Now the methods are executed in this order:test_T2,test_T3,test_T4,test_T1,test_myspecialtest.As we can see, all the test cases specified in the test case order are executed first followed by all the others. Of course, TEST-T18 is ignored as there is no implementation found.Again, a potentially new created test run will only contain TEST-T2 and TEST-T1 (in this order).For cases where a new test run should be created including only a subset of test cases, it is enough to specifytest_case_keysonly by using the required order. All listed test cases are linked (and executed, if implemented) in exactly this order.Test Case RangeIn addition to specify a list of test cases to be executed it is possible to define ranges of test cases by usingtest_case_range.ExamplesDefining["TEST-T2", "TEST-T5", "TEST-200", "TEST-299"]as a range will include any test cases from TEST-T2 to TEST-5 and from TEST-200 to TEST-299.Similar to the use oftest_case_keysall others are skipped in this case. For new created test runs these test cases are excluded while for existing test runs the appropriate test case results stay as they are (if existing).Skipping vs. BlockingThe execution of test cases (methods) or even single steps can be skipped, eitherstaticallyordynamically.Whilestatic skippingis done f.e. by specifyingtest_case_keys(only the listed test cases will be executed) or by applying appropriate markers as described below,dynamic skippingis happening based on some condition.Basically, when a test case or step is skipped the status stays untouched (typically it will be "Not Executed").According to Adaptavist test management, test cases with containing at least one step passed and all others not executed (yet) will have the status "In Progress" while test runs/cycles containing at least one test case being "Not Executed" or "In Progress" will have the status "In Progress". This can be challenging when checking if a test runs/cycles is still running or is finished with some test cases skipped. To solve this there is the option of blocking which is actually the same as skipping but with marking the test case (or step) as "Blocked". Having any test cases or steps that have been aborted or not executed marked as "Blocked" will lead to a test run/cycle status "Done" once it is finished.Using markersMethods to skip or block test cases (methods) statically are provided by the markerspytest.mark.skip(part of pytest module) andpytest.mark.block(defined in pytest-adaptavist):classTestClass(object):@pytest.mark.skip(reason="not implemented yet")@pytest.mark.project(project_key="myproject")deftest_T1(self,meta_block):...@pytest.mark.block(reason="not implemented yet")@pytest.mark.project(project_key="myproject")deftest_T2(self,meta_block):...Note that these markers work for classes as well.Using metablock action modesMethods to skip or block test cases (or steps) dynamically, e.g. depending on some condition, are provided by the meta block's helper method:classTestClass(object):@pytest.mark.project(project_key="myproject")deftest_T2(self,meta_block):withmeta_block()asmb:# if precondition is False, stop session (results in status "Blocked" for this and any following test cases)mb.check(precondition,message="precondition failed",action_on_fail=mb.Action.STOP_SESSION)# if precondition is False, stop further execution of this test (results in status "Blocked")mb.check(precondition,message="precondition failed",action_on_fail=mb.Action.STOP_METHOD)withmeta_block(1)asmb1:# if condition is False, stop further execution of this step (results in status "Blocked")mb1.check(condition,message="condition failed",action_on_fail=mb1.Action.STOP_CONTEXT)...Using pytest methodsAnother methods to skip or block test cases (or steps) dynamically, e.g. depending on some condition, arepytest.skip(part of pytest module) andpytest.block(defined in pytest-adaptavist):defmy_internal_function_1(...):...ifnotprecondition:pytest.skip(msg="precondition failed")...defmy_internal_function_2(...):...ifnotprecondition:pytest.block(msg="precondition failed")...Note that these methods are not needed usually, alas sometimes they can be useful inside of helper functions where test case context or information is missing.CallbacksFor additional custom functionality pytest-adaptavist provides a small set of callbacks:defmeta_block_condition_cb(signature,condition,reference):"""Called with each call of mb.checkproviding a signature like <item name>_<step index>_<#calls>, the condition (True|False) and a reference message (message_on_pass|message_on_fail)."""passdefmeta_block_cb(signature,status):"""Called when exiting a meta block (resp. context block)providing a signature like <item name>_<step index> and the final status ("passed", "failed", etc.)."""passExamples below show how to work with them:defmy_func(a,b,c,d,**kwargs):"""The actual function to be called."""print("my_func",a,b,c,d,kwargs)defmy_callback_example_1(func,*args):"""Wrapper for the condition callback."""def_cb(**kwargs):condition=kwargs.pop("condition",None)ifnotcondition:returnfunc(*args,**kwargs)returnNonereturn_cbdefmy_callback_example_2(func,*args):"""Another example of a callback wrapper using lambda expression."""returnlambda**kwargs:func(*args,**kwargs)ifnotkwargs.pop("condition",False)elseNonedefmy_callback_example_3(func,*args):"""Wrapper for meta block callback."""def_cb(**kwargs):status=kwargs.pop("status",None)ifstatus!="passed":returnfunc(*args,**kwargs)returnNonereturn_cbdefmy_callback_example_4(func,*args):"""Another example of a callback wrapper using lambda expression."""returnlambda**kwargs:func(*args,**kwargs)ifkwargs.pop("status",None)!="passed"elseNone@pytest.fixture(scope="function",autouse=True)defmy_fixture_example(request):"""Function-scoped fixture to set the callbacks."""request.node.meta_block_condition_cb=my_callback_example_1(my_func,17,18,19)request.node.meta_block_cb=my_callback_example_3(my_func,17,18,19)