Package: pytest-ver
SummaryThis module is used to write verification test cases
within a Python & pytest environment. FDA compliant reports can
then be generated.The verification test cases are written in Python which uses
the pytest module to invoke the test cases. During
execution, various data is captured. The data is then used by
the report facility to generate Test Protocol, Test Report
and Trace Matrix documents in docx, pdf and text formats. A
summary is also generated indicating percentage of requirements
that are complete, passing/failing etc.This repo is used to develop pytest-ver. See the matching
repohttps://bitbucket.org/arrizza-public/pytest-ver-tutorial/src/master/for
full details of how to use it. There is a User Guide with full
details there. That repo tests a simulated IV pump
application, including a set of requirements and test scripts
that use pytest-ver. A sample set of reports can be generated
and reviewed.There are also a couple of repos that have used pytest-ver as
proof-of-concept:https://bitbucket.org/arrizza-public/socket-oneline/src/master/https://bitbucket.org/arrizza-public/gui-api-tkinter/src/master/UsageSee repohttps://bitbucket.org/arrizza-public/pytest-ver-tutorial/src/master/for full instructions on how to use pytest-verUser Guide.docx in the pytest-ver-tutorial repo for additional detailsQuick Guidethe file test_sample.py shows various examples of using pytest-ver.to invoke all tests use:./doitor invoke a subset of test cases:# the "-k" switch is part of pytest./doit-ktest_0# invoke only passing tests that are fully automated./doit-k"not semi and not fails"# invoke only semi-manual tests./doit-k"semi"invoking the test_sample scriptThe test is run in doit by:functionrun_it()The report is run in doit by:functionrun_report()see out/ver directory for pdf or docx reportsInstallationThe doc/installation_*.md files contain additional information
for installing pytest-ver
for various supported platforms:installation_macos.md : MAC OSinstallation_msys2.md : MSYS2 on Windowsinstallation_ubu.md   : UbuntuPython environmentNote that the set_env.sh script sets the python environment for
various platforms and sets variables$pyexe - the python executable$pybin - the location of the environment bin directorysourceset_env.shsource"${pybin}/activate"# ... skip ...$pyexehelpers/ver_report.pyReport Outputthe output files are in the out/ver directorylsout/ver
pytest_ver.log# the log outputpytest_ver.txt# output from doit script*_protocol.json# data generated during the test case runsummary.docx# docx version of summary documentsummary.pdf# pdf version of summary documentsummary.txt# text version of summary documenttest_protocol.*# test protocol document in various formatstest_report.*# test report document in various formatstrace.*# trace matrix document in various formatsCheck conftest.pyIf you want to use the pytest-ver command line switches from pytest,
ensure you call the cli_addoption() and cli_configure() functions in conftest.pyfrompytest_verimportpth# -------------------defpytest_addoption(parser):pth.cfg.cli_addoption(parser)# -------------------defpytest_configure(config):pth.cfg.cli_configure(config)Writing a test caseimport unittest and pytest as normal. Then import pytest_ver:importunittestimportpytestfrompytest_verimportpthNote: 'pth' is a global that holds a reference to PytestHarness.
The harness holds references to all classes needed during a test run.next create a normal unit test for pytest/unitest# -------------------classTestSample(unittest.TestCase):# --------------------@classmethoddefsetUpClass(cls):pth.init()# -------------------defsetUp(self):pass# -------------------deftearDown(self):pass# --------------------@classmethoddeftearDownClass(self):pth.term()To create a protocol use pth.proto.protocol() with a protocol
id and descriptionTo create steps within that protocol, use pth.proto.step()# --------------------deftest_0(self):# declare a new protcol id and it's descriptionpth.proto.protocol('tp-000','basic pass/fail tests')pth.proto.set_dut_serialno('sn-0123')pth.proto.step('try checks with 1 failure')pth.ver.verify_equal(1,2,reqids='SRS-001')pth.ver.verify_equal(1,1,reqids='SRS-002')pth.ver.verify_equal(1,1,reqids='SRS-003')pth.proto.comment('should be 1 failure and 2 passes')at this point, there is one protocol TP-000 that has 1 step.Use doit to run it:./doit-ktest_0OutputCheck the stdout or the out/ver/pytest_ver.txt file:indicates that a failure occuredthe return code from the script is non-zeroReport documentsCheck the generated documents in the out/ver/ dirctory.summary.pdf should indicate:there are a total of 7 requirements (see srs_sample.json)there are 2 passing requirements wghich is 28.8% of all
requirementsthere is 1 failing requirement which is 14.3% of all
requirementsthere are 4 requirements that were not tested which is
57.1% of all requirementstest_report.pdf and/or test_protocol.pdf should indicate:the test run type is "dev" so this was not a formal runthe test run id is "dev-001". This can be set in cfg.json to
track individual test runsthe date time the document was generatedThere should be one protocol TP-000The location of the protocol is test_sample.py(line number)The protocol had only 1 step which tested requirement SRS-001The report document shows the expected and actual values and
that result was a FAIL,
and the location of the failing verify() functionThe report document shows a commentThere is table after the protocol showing the requirement
SRS-001 and it's descriptionNote the header and footer information comes from the cfg.json
filePytest markersyou can use pytest markers as normal# --------------------# @pytest.mark.skip(reason='skip')@pytest.mark.smoketest1deftest_init2(self):pth.proto.protocol('tp-002','test the init2')pth.proto.step('verify1 everything is equal')pth.ver.verify(1==1,reqid='SRS-001')# note: this is the second time this requirment is verified# --------------------# @pytest.mark.skip(reason='skip')deftest_init3(self):pth.proto.protocol('tp-003','test the init3')pth.proto.step('verify1 everything is equal')pth.ver.verify(1==1,reqid='SRS-004')Verification FunctionsTo create verification tests use pth.ver.verify()# note: you can use normal pytest and unitest functions# but their results won't show up in the reportself.assertEqual(x,y)# do a verification against a requirementpth.ver.verify_equal(x,y,reqid='SRS-001')pth.ver.verify_equal(x,1,reqid='SRS-001')# since all verifys passed, this step's result is PASSpth.proto.step('verify2')pth.ver.verify(False,reqid='SRS-002')pth.ver.verify(True,reqid='SRS-002')pth.ver.verify(True,reqid='SRS-002')# since one verify failed, this step's result is FAILpth.proto.step('verify3')pth.ver.verify(True,reqid='SRS-003')pth.ver.verify(True,reqid='SRS-003')pth.ver.verify(False,reqid='SRS-003')# since one verify failed, this step's result is FAILSee doc/User Guide.docx for a full list of verification functionsverify(actual)# verify actual is trueverify_true(actual)# verify actual is trueverify_false(actual)# verify actual is falseverify_equal(expected,actual)# verify actual == expectedverify_not_equal(expected,actual)# verify actual != expectedverify_none(actual)# verify actual is Noneverify_is_none(actual)# verify actual is Noneverify_not_none(actual)# verify actual is not Noneverify_in(actual,exp_list)# verify actual is in the expected listverify_not_in(actual,exp_list)# verify actual is not in the expected listverify_lt(left,right)# verify left < rightverify_le(left,right)# verify left <= rightverify_gt(left,right)# verify left > rightverify_ge(left,right)# verify left >= rightverify_reqex(actual,regex)# verify actual matches the regexverify_not_reqex(actual,regex)# verify actual does not matche the regexverify_delta(expected,actual,abs_tolerance)# verify actual == expected within +/- toleranceverify_not_delta(expected,actual,abs_tolerance)# verify actual outside +/- toleranceverify_delta_pct(expected,actual,pct_tolerance)# verify actual == expected within +/- percentverify_not_delta_pct(expected,actual,pct_tolerance)# verify actual outside +/- percentA pass does not generate any stdoutA fail reports various informationthe location of the failurethe expected value (and it's python type)the actual value (and it's python type)a traceback at the time of the failuresFAILURE:attest_sample.py(37)Expected(int):1Actual(int):2test_sample.py:37intest_0()->pth.ver.verify_equal(1,2,reqids='SRS-001')pytest_ver/lib/verifier.py:98inverify_equal()->self._handle_fail(rs)pytest_ver/lib/verifier.py:412in_handle_fail()->raiseAssertionError(f'at {rs.location}{msg}')The test_report.txt document shows some additional information:the protocol id and description and its locationwhich step failedthe date time stamp (dts) when the failure occurredthe requirement id====protocol:TP-000basicpass/failtestslocation:test_sample.py(33)Step1:trycheckswith1failure>dts:2022-12-1106:00:52>result:FAIL>actual:2>actualraw:2>expected:1>expectedraw:1>reqids:{'SRS-001':1}>location:test_sample.py(37)Generate Reportto generate a report use: pth.report()see helpers/ver_report.pyimportosimportsyssys.path.insert(1,os.path.join('.'))frompytest_verimport*# noqa# generate the reportpth.cfg.cli_parse()# force iuvmode to bepth.cfg.cli_set('iuvmode',False)pth.init(report_mode=True)pth.report()pth.term()to invoke it:$pyexehelpers/ver_report.py