Package: testman
TestManA manager for automated testing by humansRationaleAlthough automated testing inherently requires code, and Python has a very readable syntax, it is still not the best abstraction layer for "humans" who focus on testing and not on code.Still testers still want to describe tests in a way that they can easily be automated - if possible without having to rely on developers to (en)code them.TestMan tries to offer a simple framework that brings together both worlds, by offering a simple test definition interface (aka a DSL) combined with a simple way for developers to provide additional, reusable testing functions, all with the power of Python underneath.A Word of CautionTestMan usesevalto perform the tests and assertionsyouspecify.This means that it can run arbitrary code.Don't just feed any TestMan script to it.A Quick ExampleImagine you want to check that an email gets send and delivered succesfully.The following TestMan script defines this test and can be written by a non-development-minded testing human:uid:gmailname:Sending an email and checking that it was deliveredconstants:UUID:uuid.uuid4variables:TIMESTAMP:testman.util.utcnow | isoformatsteps:-name:Sending an emailperform:testman.testers.mail.sendwith:server:smtp.gmail.com:587username:GMAIL_USERNAMEpassword:GMAIL_PASSWORDrecipient:GMAIL_USERNAMEsubject:A message from TestMan ({UUID})body:~gmail_body.txt-name:Checking that the email has been deliveredperform:testman.testers.mail.popwith:server:pop.gmail.comusername:GMAIL_USERNAMEpassword:GMAIL_PASSWORDassert:any mail.Subject == "A message from TestMan ({UUID})" for mail in resultSo this test consists of twosteps:Step 1: Sending an email (with a unique marker)
Step 2: Checking that the email has been delivered (checking the marker)For each step, an action toperformis executed optionallywitharguments. Finally, one or moreassertionscan be expressed to validate the method's result.A developerwouldonly have to provide a module consisting of the following two functions, one for sending and one for fetching mail:importsmtplibimportpoplibimportemailimportemail.policydefsend(server=None,username=None,password=None,recipient=None,subject=None,body=None):msg="\r\n".join([f"From:{recipient}",f"To:{recipient}",f"Subject:{subject}","",body])server=smtplib.SMTP(server)server.ehlo()server.starttls()server.login(username,password)server.sendmail(username,[recipient],msg)server.close()defpop(server=None,username=None,password=None):conn=poplib.POP3_SSL(server)conn.user(username)conn.pass_(password)messages=[conn.retr(i)foriinrange(1,len(conn.list()[1])+1)]conn.quit()result=[]formsginmessages:m=email.message_from_bytes(b"\n".join(msg[1]),policy=email.policy.default)r={}fork,vindict(m.items()).items():r[k]=str(v)r["Body"]=m.get_content()result.append(r)returnresultNotice that these functions merely implement the execution of the test action. The assertions are handled by TestMan. This way it is simple to extend TestMan with custom methods to gather information and have TestMan perform assertions over it.I stressedwould, since these basic tester functions are part of TestMan's included set of testers - which was also visible from theperformedfunction name ;-)To execute this test TestMan offers both a Python and CLI way. Say that the TestMan script is saved asexamples/gmail.yamland we have a set two environment variables (GMAIL_USERNAME and GMAIL_PASSWORD), e.g. in a.envfile, we now can execute the script from the Python REPL...>>>importyaml,json>>>fromdotenvimportload_dotenv,find_dotenv>>>load_dotenv(find_dotenv(usecwd=True))True>>>fromtestmanimportTest>>>withopen("examples/gmail.yaml")asfp:...script=yaml.safe_load(fp)...>>>mytest=Test.from_dict(script,work_dir="examples/")>>>mytest.execute()>>>print(json.dumps(mytest.results,indent=2)){"Sending an email": {"start": "2022-09-21T07:07:44.982532","end": "2022-09-21T07:07:46.204026","output": null,"info": null,"status": "success","skipped": false},"Checking that the email has been delivered": {"start": "2022-09-21T07:07:46.204132","end": "2022-09-21T07:07:48.345193","output": [{"Return-Path": "<someone@email.com>","Received": "from ip6.arpa (d54.access.telenet.be. [4.19.19.25])","Message-ID": "<fd4dc.9827@mx.google.com>","Date": "Sun, 18 Sep 2022 11:31:17 -0700","From": "someone@email.com","To": "someone@email.com","Subject": "A message from TestMan (1da6f05f-d819-44be-a07b-4d2cbe6acd04)","Body": "Hello,\n\nThis is an automated message from TestMan.\nTime of sending: 2022-09-18T18:31:16.128970\n\nregards,\nTestMan"}],"info": null,"status": "success","skipped": false}}... and from the command line ...%python-mtestmanloadexamples/gmail.yamlexecuteresults_as_json[2022-09-21 09:11:19 +0200] ▶ in /Users/xtof/Workspace/testman/examples[2022-09-21 09:11:20 +0200] ✅ Sending an email[2022-09-21 09:11:22 +0200] ✅ Checking that the email has been delivered[2022-09-21 09:11:22 +0200] ◀ in /Users/xtof/Workspace/testman{"Sending an email": {"start": "2022-09-21T07:07:44.982532","end": "2022-09-21T07:07:46.204026","output": null,"info": null,"status": "success","skipped": false},"Checking that the email has been delivered": {"start": "2022-09-21T07:07:46.204132","end": "2022-09-21T07:07:48.345193","output": [{"Return-Path": "<someone@email.com>","Received": "from ip6.arpa (d54.access.telenet.be. [4.19.19.25])","Message-ID": "<fd4dc.9827@mx.google.com>","Date": "Sun, 18 Sep 2022 11:31:17 -0700","From": "someone@email.com","To": "someone@email.com","Subject": "A message from TestMan (1da6f05f-d819-44be-a07b-4d2cbe6acd04)","Body": "Hello,\n\nThis is an automated message from TestMan.\nTime of sending: 2022-09-18T18:31:16.128970\n\nregards,\nTestMan"}],"info": null,"status": "success","skipped": false}}The examples folder contains another TestMan script, that uses some mocked up test functions to illustrate some of the possibilities:%python-mtestmanloadexamples/mock.yamlexecute[2022-09-21 09:11:55 +0200] ▶ in /Users/xtof/Workspace/testman/examples[2022-09-21 09:11:55 +0200] ✅ Test that an envrionment variable as argument is returned[2022-09-21 09:11:55 +0200] 🚨 Test that an incorrect argument fails - 'result.hello == "world"' failed for result={'hello': 'WORLD'}[2022-09-21 09:11:55 +0200] 🚨 Test that a random value is bigger than 0.7 - 'result > 0.7' failed for result=0.16237524253063063[2022-09-21 09:11:55 +0200] 🚨 Test that two keys of a random dict match criteria - 'result.result1 < 0.5' failed for result={'result1': 0.5824170196145589, 'result2': 0.6443538640468358, 'result3': 0.5201206405472322}[2022-09-21 09:11:55 +0200] ✅ Test that any values in a random list is < 0.5[2022-09-21 09:11:55 +0200] 🚨 Test that all of the dicts in a list have result1 < 0.5 - 'all item.result1 < 0.5 for item in result' failed for result=[{'result1': 0.5336479045489211, 'result2': 0.30981993872330327, 'result3': 0.27661254405288216}, {'result1': 0.23920511249278598, 'result2': 0.7080886447360248, 'result3': 0.1361173238448583}, {'result1': 0.719807760896399, 'result2': 0.21743598455790847, 'result3': 0.6627068916970688}][2022-09-21 09:11:55 +0200] ✅ Test that values inside a nested dict can be asserted[2022-09-21 09:11:55 +0200] ✅ Test that object properties and methods can be asserted[2022-09-21 09:11:55 +0200] ◀ in /Users/xtof/Workspace/testmanAnd if you install Testman from PyPi, it will also install thetestmancommand line script.%pipinstalltestman%testmanversion0.0.3%testmanload../testman/examples/mock.yamlexecute[2022-09-21 09:12:37 +0200] ▶ in /Users/xtof/Workspace/testman/examples[2022-09-21 09:12:37 +0200] ✅ Test that an envrionment variable as argument is returned[2022-09-21 09:12:37 +0200] 🚨 Test that an incorrect argument fails - 'result.hello == "world"' failed for result={'hello': 'WORLD'}[2022-09-21 09:12:37 +0200] 🚨 Test that a random value is bigger than 0.7 - 'result > 0.7' failed for result=0.111074170853391[2022-09-21 09:12:37 +0200] 🚨 Test that two keys of a random dict match criteria - 'result.result2 > 0.5 or result.result2 < 0.1' failed for result={'result1': 0.1970010221173848, 'result2': 0.20193458254324714, 'result3': 0.25430481982617814}[2022-09-21 09:12:37 +0200] ✅ Test that any values in a random list is < 0.5[2022-09-21 09:12:37 +0200] 🚨 Test that all of the dicts in a list have result1 < 0.5 - 'all item.result1 < 0.5 for item in result' failed for result=[{'result1': 0.36567629773296983, 'result2': 0.7890211368675198, 'result3': 0.28117765912895376}, {'result1': 0.6011158365310331, 'result2': 0.9422161128407222, 'result3': 0.9990457704008657}, {'result1': 0.1991118692356576, 'result2': 0.8839600916890447, 'result3': 0.7788144324166787}][2022-09-21 09:12:37 +0200] ✅ Test that values inside a nested dict can be asserted[2022-09-21 09:12:37 +0200] ✅ Test that object properties and methods can be asserted[2022-09-21 09:12:37 +0200] ◀ in /Users/xtof/Workspace/testman_testCustom Test FunctionsTo implement your own test functions, simply write a function and use its FQN asperformparameter:>>>importyaml>>>importjson>>>fromtestmanimportTest,Step>>>>>>defhello(name):...returnf"hello{name}"...>>>steps=yaml.safe_load("""...- name: say hello...perform: __main__.hello...with:...name: Christophe...assert: result == "hello Christophe"...""")>>>>>>mytest=Test("hello world test",[Step.from_dict(step)forstepinsteps])>>>mytest.execute()>>>print(json.dumps(mytest.results,indent=2)){"say hello": {"start": "2022-10-01T15:07:16.366154","end": "2022-10-01T15:07:16.366565","output": "hello Christophe","info": null,"status": "success","skipped": false}}The TestMan Steps DSLTestMan uses a nested dictionary structure as its domain specific language to encode the steps to take during the test. I personally prefer to write them inyaml, yet this is purely optional and a personal choice. As long as you pass a dictionary to TestMan, it will happily process it.The dictionary looks like this:steps = [
  {
    "name" : "a description of the step used to refer to it",
    "perform" : "the fully qualified name of a function",
    "with" : {
      "key" : "value pairs of arguments passed to the function as named arguments",
    },
    "assert" : "optionally, a single string or list of strings to be evaluated to assert the result (see below for the full description of the assertion DSL)",
    "continue" : "yes, optionally, allows to indicate that in case of an assertion failure, the steps should be continued, whereas the default is to stop processing the steps on first failure",
    "always": "yes, optionally, allows to ensure that a step is always performed, even if it previously was successful, whereas the default is to not perform a step if it previously was successful"
  }
]A Typical WorkflowI've designed TestMan with a specific workflow in mind: managing a set of tests that all take some time to complete and therefore need to be run multiple times, until all tests are done.So a typical workflow would struturally look like:load tests into a test suiteexecuteexecute...until all tests in the suite have completedIn the mean time, the persisted test information can be used to visualize the status.The following example, using the examples in the repository, illustrates this:%testmanstatemongodb://localhost:27017/testman/suitesload../testman/examples/mock.yamlload../testman/examples/gmail.yaml%testmanstatemongodb://localhost:27017/testman/suitessuitesummarymock:  {"unknown": 8, "success": 0, "ignored": 0, "pending": 0, "failed": 0, "summary": "8 in progress"}gmail: {"unknown": 2, "success": 0, "ignored": 0, "pending": 0, "failed": 0, "summary": "2 in progress"}%testmanstatemongodb://localhost:27017/testman/suitessuiteexecutesummarymock:  {"unknown": 0, "success": 5, "ignored": 1, "pending": 2, "failed": 0, "summary": "2 in progress"}gmail: {"unknown": 0, "success": 2, "ignored": 0, "pending": 0, "failed": 0, "summary": "all done"}%testmanstatemongodb://localhost:27017/testman/suitessuiteexecutesummarymock:  {"unknown": 0, "success": 5, "ignored": 1, "pending": 2, "failed": 0, "summary": "2 in progress"}gmail: {"unknown": 0, "success": 2, "ignored": 0, "pending": 0, "failed": 0, "summary": "all done"}%testmanstatemongodb://localhost:27017/testman/suitessuiteexecutesummarymock:  {"unknown": 0, "success": 6, "ignored": 1, "pending": 1, "failed": 0, "summary": "1 in progress"}gmail: {"unknown": 0, "success": 2, "ignored": 0, "pending": 0, "failed": 0, "summary": "all done"}%testmanstatemongodb://localhost:27017/testman/suitessuiteexecutesummarymock:  {"unknown": 0, "success": 6, "ignored": 1, "pending": 1, "failed": 0, "summary": "1 in progress"}gmail: {"unknown": 0, "success": 2, "ignored": 0, "pending": 0, "failed": 0, "summary": "all done"}%testmanstatemongodb://localhost:27017/testman/suitessuiteexecutesummarymock:  {"unknown": 0, "success": 7, "ignored": 1, "pending": 0, "failed": 0, "summary": "all done"}gmail: {"unknown": 0, "success": 2, "ignored": 0, "pending": 0, "failed": 0, "summary": "all done"}After loading two tests into the MongoDBsuitescollection, the initial status shows that 8+2=10 steps in two tests (mock and gmail) need to performed.After every execution, more steps have been performed succesfully or have been ignored, until alle steps have been completed as requested.The execution can be triggered from a function/cron job/... that is called every minute, or every hour, thus enabling long-during test suite executions.An More Elaborate ExampleAn example that showcases some more of the features of TestMan isexamples/postbin.yaml:uid:postbinname:use postbin to test calling an APIconstants:POSTBIN:https://www.toptal.com/developers/postbinsteps:-name:Create a binperform:requests.post | testman.unwrap.requests.jsonwith:url:"{POSTBIN}/api/bin"assert:-result.status_code == 201-"'binId'inresult.json"-name:Post something to binperform:requests.postwith:url:"{POSTBIN}/{STEP[0].json.binId}"params:hello:worlddata:more:-data_0-data_1assert:-result.status_code == 200always:yes-name:Check content of binperform:requests.get | jsonwith:url:"{POSTBIN}/api/bin/{STEP[0].json.binId}/req/shift"assert:-result.query.hello  == "world"-result.body.more[1] == "data_1"always:yesOutput UnwrappingTestMan doesn't like objects as output. Simply because it wants to be able to serialize output to a simple JSON representation and back. So, when the output of a performed function is an object, TestMan tries to unwrap this into something more like a dict. So without your intervention, an object will be transformed to its__dict__.Thepostfunction from therequestsmodule returns aResponseobject. So by default, TestMan will return a dict containint the folowing keys:>>>importrequests>>>r=requests.get("http://google.be")>>>r.__dict__.keys()dict_keys(['_content', '_content_consumed', '_next', 'status_code', 'headers', 'raw', 'url', 'encoding', 'history', 'reason', 'cookies', 'elapsed', 'request', 'connection'])This might be perfect for your test step. If you'd like to inspect, let's say the returned JSON, you'd have to calljson.parse()your self.Theperformstatement takes optional post-processors. Post-processors are appended to the function, separated by slashes. A post-processor can be a property name, a function name, a string to be applied to a dict or a function.In case of the requests module, we can therefore append| jsonto the function to perform and have the resulting Response object unwrapped using its.json()method.Another approach is to write your own unwrapping function. In the postbin exampletestman.unwrap.requests.jsonis simply defined as:defjson(response):return{"status_code":response.status_code,"json":response.json()}It simply extracts thestatus_codealong with thejsonpart in a simple dict.String InterpollationUsing curly braces, existing variables, constants, environment variables,... can be dynamically inserted into strings:"{POSTBIN}/{STEP[0].json.binId}"includes both thePOSTBINconstant aswell as the implicitSTEPvariable that contains the last outputs of all previous steps.