Package: santa-rest-test
[![pipeline status](https://gitlab.com/lisael/santa/badges/master/pipeline.svg)](https://gitlab.com/lisael/santa/commits/master)[![coverage report](https://gitlab.com/lisael/santa/badges/master/coverage.svg)](https://gitlab.com/lisael/santa/commits/master)# SantaDeclarative JSON-REST api test framework, so simple, it's hard to belive.- Santa is a concept a 5yo can understand.- Santa brings so much magic in your life, it's hard to believe before you see.- Santa loves you all equally. QA people, POs, developers, everyone.- Writing your wish-list to Santa doesn't involve any Electron-ic or Atom-ic stuff. Just a Notepad.- Santa gifts are not limited to what's available on the internet. You can craft your own orlet the elves fulfill your sweetest dreams.- Don't tell Santa he's my fantasy Postman, he'd get upset.- You'd better not upset Him.## Features- Declarative, git-friendly, YAML syntax- Ansible-like architecture and lazy jinja templating- Validator/extractor/context_processors- Leverage asyncio (with aiohttp client) to parallelize runs- Interactive test case builder- Almost everything is easily customisable- custom validators- custom extractors- custom context processors- custom tasks- custom jinja functions## Quick start### Install```pip install santa-rest-test```### Developer's installIf you want to contribute to Santa or you wish to run the quick start project,you should install Santa from source:```shvirtualenv --python=/usr/bin/python3 santacd santa. bin/activategit clone https://gitlab.com/lisael/santa.gitcd santapip install -e .```### Test projectTo run the testing project (a modifed version of DRF tutorial):```shmake testproj_run```Alternatively, use the docker image:```shmake docker_run_testproj```### Create a project```shmkdir myprojectcd myprojectmkdir {suite,context}```### Add a contextA context is a nested map that is passed from one task to the next one. Eachtask may use and/or modify the context.```sh$EDITOR context/local.yml``````yaml# myproject/context/local.yml---domain: localhostscheme: httpurls:default:port: 8080scheme: "{{ scheme }}"host: "{{ domain }}"headers:Authorization: "{{ basic_auth(users.admin.username, users.admin.password) }}"Content-Type: "application/json"users:admin:username: adminpassword: admin```This defines a context that is available in all tests.The context layout is free, the only requirements are:1. The root object is a map2. keys of the root map do not start with `_` (reserved for internal use)However, some good practices may emerge and this kind of inventory per type ofobjects we are to manipulate has proven useful.The values are templated. It's important to understand that the templating is lazyand is applied when the value is read, and re-evaluated each time the value is read.The laziness has two interesting consequences:1. Not all variable have to be defined when the first test starts.- However, it must be defined at the time it's used2. When variables are changed, the value of dependent variables is re-evaluated- e.g. if `users.admin.password` changes, `urls.default.headers.Authorization` isstill correct### Add a test suite```sh$EDITOR suite/snippets.yml``````yaml---- test:name: List snippetsurl:$<: urls.defaultpath: snippets/```The URL is not fully defined here, as it extends the context variable `url.default` usingthe special `$<` key.  Only the API endpoint (aka. `path`) is missing in `url.default`.No method is defined in our `test`, santa assumes `GET`.### Run the testsThe basic syntax of the command is `santa test <context> <suite>````shsanta test local snippets```The result shows one test run and did not fail:```suite/snippets.yml: List snippets: OKSummary**************************************************tests: 1    success: 1    skipped: 0    failed: 0**************************************************```### Reusing the contextThis local test is nice (albeit small, we're going to expand the test coverage later), butwe'd like to run this tests against multiple environments.Hopefully, the context can be extended. This provides a clean way to reuse parts of a basecontext in child contexts.Let's define a base context:```sh$EDITOR context/base.yml``````yaml---domain: "{{ undefined('Domain must be set') }}"scheme: httpsurls:default:scheme: "{{ scheme }}"host: "{{ domain }}"headers:Authorization: "{{ basic_auth(users.admin.username, users.admin.password) }}"Content-Type: "application/json"users:admin:username: "{{ undefined('Admin username') }}"password: "{{ undefined('Admin passord') }}"````undefined` is a builtin template function (as is `basic_auth`) that raises an`UndefinedError` if the value is accessed. These values are mandatory ininherited contexts.Now we can re-write our context to extend base.yml```sh$EDITOR context/local.yml``````yaml---extends: basedomain: localhostscheme: httpurls:default:port: 8080users:admin:username: adminpassword: admin```Run this:```shsanta test local snippetssuite/snippets.yml: List snippets: OKSummary**************************************************tests: 1    success: 1    skipped: 0    failed: 0**************************************************```### AssertionsSo far, we did call our API endpoint, but apparently we did not check anything in the resultof the call. Well actually, we did. The `test` task validate the HTTP status of the responseagainst sensible defaults (depending on the method).We can add a simple validator that checks if expected keys are present in the result:```sh$EDITOR suite/snippets.yml``````yaml- test:name: List snippetsurl:$<: urls.defaultpath: snippets/validate:- json:contains:- count- next- previous- results```Validators are listed in the `test` argument `validate````shsanta test local snippetssuite/snippets.yml: List snippets: OKSummary**************************************************tests: 1    success: 1    skipped: 0    failed: 0**************************************************```Note that the default HTTP status validator is still run. To override the defaultstatus validator, just add your own:```yamlvalidate:- status: [417]```### Going postalPosting an new snippet is as simple as:```yaml- test:name: Post test snippetmethod: POSTurl:$<: urls.defaultpath: snippets/json_body:title: test.pycode: |from math import sin, pisin(pi/2)linenos: falselanguage: pythonstyle: vim```This works, but we're repeating ourselves a bit, don't we?The `url` part is the same as in our "List snippets" test. And what if we decide toadd a `/blog/` in our API and to move snippets under `/pastebin/` for namespacingsake ? Would we change each and every tests that call `/snippets/` to `/pastebin/snippets` ?We also want to test many ways to call snippets, each one uses all or a part of the `json_body`we defined here. It would be nice to re-use a json payload with sensible default.It's time to leverage the context to enforce the DRY principle. We add the `endpoints` mappingthat defines all our endpoints (URLs and payloads). Note that these endpoint are shared amongstall environments, so we define them once in the base context.```sh$EDITOR context/base.yml``````yaml---domain: "{{ undefined("Domain must be set") }}"scheme: httpsurls:default:scheme: "{{ scheme }}"host: "{{ domain }}"headers:Authorization: "{{ basic_auth(users.admin.username, users.admin.password) }}"Content-Type: "application/json"users:admin:username: "{{ undefined("Admin username") }}"password: "{{ undefined("Admin passord") }}"endpoints:snippets:url:$<: urls.defaultpath: snippets/json:title: ""code: ""linenos: falselanguage: nullstyle: vim```Change the suite accordingly:```sh$EDITOR suite/snippets.yml``````yaml- test:name: List snippetsurl:$<: endpoints.snippets.urlvalidate:- json:contains:- count- next- previous- results- test:name: Post test snippetmethod: POSTurl:$<: endpoints.snippets.urljson_body:$<: endpoints.snippets.jsontitle: test.pycode: |from math import sin, pisin(pi/2)language: python```-- BEGIN TROLL --It may seem strange to some users to define JSON values in yaml. It's also harder to copy andpaste real payloads.I, for myself, think that JSON is not a human language and should never be used as ahuman-to-machine format.Anyway, remember that JSON is valid YAML ;).-- END OF TROLL (quickly resolved, hopefully) --```shsanta test local snippets``````suite/snippets.yml: List snippets: OKsuite/snippets.yml: Post test snippet: OKSummary**************************************************tests: 2    success: 2    skipped: 0    failed: 0**************************************************```### Manipulating the contextThere are 4 ways to update a context:1. Define another context that `extends` the first one2. Pass an `--extra-var` option at the invocation of the suite3. Extract data from the result JSON in a test4. Manipulate the context with `context_processors` defined in the suite.We've already show the first way and the second is pretty clear. These two methods arestatic in the sense that they update the context before the suite is run. The third andfourth methods, on the other hand are dynamic. They are defined in the suite itself andreact to events occuring during the tests.#### ExtractorsSay that we want to keep track of the snippet we've just posted. On a POST, the APIgive us the id of the created snippet in the result JSON. Using an extractor, we canbind a context variable to any value. Let's add an extractor to the POST:```sh$EDITOR suite/snippets.yml``````yaml# ...- test:name: Post test snippetmethod: POSTurl:$<: endpoints.snippets.urljson_body:$<: endpoints.snippets.jsontitle: test.pycode: |from math import sin, pisin(pi/2)language: pythonextract:- jq:pattern: .idbind: snippets.first.id- jq:pattern: .urlbind: snippets.first.full_url```We use the jq extractor that binds a context variable to the result of the givenjq pattern.Note that if a key is not found on the path of the bond variable, it is simplycreated, e.g. those extractors created `snippets`, `snippets.first`, `snippets.first.id`(bound to the new snippet id) and `snippets.first.full_url` (bound to the new snippeturl)#### Context processorsIf we are to call the new created snippet, we now have to define the URL like this```yaml- test:name: Get first testurl:$<: endpoints.snippets.urlpath: "{{ endpoints.snippets.url}}{{ snippets.first.id }}/"```For the sake or DRYness, we can register this url in `snippets.first` context mapping.That's exactly what context processors are for:```sh$EDITOR suite/snippets.yml``````yaml# ...- context:- copy:src: endpoints.snippets.urldest: snippets.first.url- update:snippets.first.url.path: "{{ endpoints.snippets.url.path }}{{ snippets.first.id }}/"````context` is a new task type, that accesses the context. It's a list of context processors thatare run in the order of definition. Here, at first we copy the snippets URL into the new snippet'sinfo mapping and then we update the path to point to the id of the snippet.We can check that the contex processor did its job using the `message` task that prints nice formatedmessages or raw yaml representing the context:```sh$EDITOR suite/snippets.yml``````yaml# ...- message:message: The full url is {{ snippets.first.full_url }}context:- snippets``````shsanta test local snippets``````suite/snippets.yml: List snippets: OKsuite/snippets.yml: Post test snippet: OKsuite/snippets.yml: context: OKThe full url is http://localhost:8080/snippets/15/snippets:first:full_url: http://localhost:8080/snippets/15/id: 15url:headers:Authorization: 'Basic YWRtaW46YWRtaW4='Content-Type: application/jsonhost: localhostpath: snippets/15/port: 8080scheme: httpsuite/snippets.yml: : OKSummary**************************************************tests: 2    success: 2    skipped: 0    failed: 0**************************************************```Now we can use the API to manipulate the first snippet with less boilerplate YAML:```sh$EDITOR suite/snippets.yml``````yaml# ... (remove the message to avoid log pollution)- test:name: Get new snippetmethod: GETurl:$<: snippets.first.urlvalidate:- json:partial:language: python``````shsanta test local snippets``````suite/snippets.yml: List snippets: OKsuite/snippets.yml: Post test snippet: OKsuite/snippets.yml: context: OKsuite/snippets.yml: Get new snippet: OKSummary**************************************************tests: 3    success: 3    skipped: 0    failed: 0**************************************************```### More context tricks.The snippets API is open to multiple users with various permissions. For example, a snippethas an `owner` field. Only the owner of a snippet can edit the snippet. To tests this we need toimpersonate multiple users. The authentication is HTTP basic auth, and we defined the `default` URL(which in turn is the base of all URLs) like this:```yaml# context/base.yml---# ...urls:default:port: 8080scheme: "{{ scheme }}"host: "{{ domain }}"headers:Authorization: "{{ basic_auth(users.admin.username, users.admin.password) }}"Content-Type: "application/json"# ...```The problem that arise here is that the authenticated user is hard-coded in the context: `admin`.The solution is to use the laziness and late evaluation of the context:```sh$EDITOR context/base.yml``````yaml# ...urls:default:port: 8080scheme: "{{ scheme }}"host: "{{ domain }}"headers:Authorization: "{{ basic_auth(users.current.username, users.current.password) }}"Content-Type: "application/json"# ...```The trick consists in using the undefined user `current` to generate the authenticationheader. Now we have to set this user to any defined user in a `context` task and allsubsequent requests will correctly authenticate.We have to add the `context` task at the top of the snippets suite:```sh$EDITOR suite/snippets.yml``````yaml---- context:- copy:src: users.admindest: users.current# ...```### More endpointsLet's test the other endpoint, `/users/`. With all we've already setup it's easy.First, we add the endpoint definition in the base context```sh$EDITOR context/base.yml``````yaml# ...endpoints:snippets:url:$<: urls.defaultpath: snippets/json:title: ""code: ""linenos: falselanguage: nullstyle: vimusers:url:$<: urls.defaultpath: users/```Then we create a suite:```sh$EDITOR suite/users.yml``````yaml---- context:- copy:src: users.admindest: users.current- test:name: Get usersurl:$<: endpoints.users.url```Wait a minute... That's not DRY, we're repeating the copy of the user `admin` to`current`. There must be a way to avoid this.Suites can include each other. We can use this feature to avoid duplication of code.Create a suite called `log_as/admin`:```sh$EDITOR suite/log_as/admin.yml``````yaml---- context:- copy:src: users.admindest: users.current```Include this suite in `users.yml` and `snippets.yml````sh$EDITOR suite/users.yml``````yaml---- include: log_as/admin- test:name: Get usersurl:$<: endpoints.users.url``````sh$EDITOR suite/snippets.yml``````yaml---- include: log_as/admin- test:# ...```Avoiding code duplication is more future-proof. At the moment, the API implementsonly basic HTTP authentication. If we decide to implement token authentication or evenOAuth dance, we only have to change the implementation of `suite/log_as/admin.yml`, andthe `include` task abstract the login logic away.We can try the new endpoint tests:```shsanta test local users``````suite/log_as/admin.yml: context: OKsuite/users.yml: include: OKsuite/users.yml: Get users: OKSummary**************************************************tests: 1    success: 1    skipped: 0    failed: 0**************************************************```Note that the log lines are printed when the task is completed. That's why the log issomewhat hard to follow. The first line is the included context processor, the secondone tels that the inclusion did complete, and the third is the actual test.The logging logic may change in the future. However, if you have to perform a fineanalysis of a suite result, use alternate outputs like json or yaml:```shsanta test local users -o yaml```### Run all the suites at onceWhen we are writing a suite, it's nice to be able to run only what we're currentlyworking at. (e.g. users or snippets). But when the tests are run, we want to run themall at once. We can use the include task to create a collection of suites:```sh$EDITOR suite/all.yml``````yaml---- include: snippets- include: users```Run this...```shsanta test local all```Guess what... We're somehow repeating ourselves. Each of the included suiteincludes `log_as/admin` repeating the context update. It's not thatproblematic here as we use basic HTTP auth and `log_as/admin` only access thelocal memory through the context, but logging as an admin user may require aHTTP roundtrip or two.The naive solution is to include `log_as/admin` at the top of `all.yml` and toremove it from `snippets.yml` and `users.yml`. It works, but we would loose theability to run the suites in isolation.The `require` task is what we need. `require` works like include except that itkeeps track of what suite was required and don't repeat the suite.```sh$EDITOR suite/users.yml``````yaml---- require: log_as/admin- test:# ...``````sh$EDITOR suite/snippets.yml``````yaml---- require: log_as/admin- test:# ...```Run this...```shsanta test local all``````suite/log_as/admin.yml: context: OKsuite/snippets.yml: require: OKsuite/snippets.yml: List snippets: OKsuite/snippets.yml: Post test snippet: OKsuite/snippets.yml: context: OKsuite/snippets.yml: Get new snippet: OKsuite/all.yml: include: OKsuite/users.yml: require: SKIPPEDsuite/log_as/admin.yml already completedsuite/users.yml: Get users: OKsuite/all.yml: include: OKSummary**************************************************tests: 4    success: 4    skipped: 0    failed: 0**************************************************```Internally the completed required suite are kept into the context. This should preventfrom login as user1 -> login as user2 -> login as user1 again. `require` accepts asecond argument that resets previously required suites:```yaml- require:suite: log_as/user1reset: log_as/*````reset` accept globs, but excludes it's own `suite` from the expanded names.### Run suites in parallelIn continuous integration process, the tests must be fast and should raise issuesas soon as possible. Large functional test suite are slow. To mitigate this issue,santa allows parallel tests.Now we two have problems.Parallel execution is never simple to handle and this feature must be used with muchprecautions. To help the user to wrap their head around parallel tests, santa usesa straightforward technique, the context is simply duplicated and the new context ispassed to the new "thread", with no possibility to share context values among threads.This implementation is limiting, and may force the user to repeat themselves but it'smuch safer and avoids a whole class of client-side race conditions.However it's not a silver bullet and the user must care about server-side raceconditions. For example, if a thread counts the snippets in the database whileanother one creates snippets and a third thread delete snippets, the first thread isin trouble to get consistent and reproducible results. We can't do anything on theclient side to mitigate this uncertainty.That said, with careful implementation of isolated threads, concurrency may substantiallyspeed up your test suite.To run tests in parallel, just replace `include` with `fork`:```sh$EDITOR suite/all.yml``````yaml---- fork: snippets- fork: users``````shsanta test local all``````suite/log_as/admin.yml: context: OKsuite/snippets.yml: require: OKsuite/log_as/admin.yml: context: OKsuite/users.yml: require: OKsuite/snippets.yml: List snippets: OKsuite/users.yml: Get users: OKsuite/snippets.yml: Post test snippet: OKsuite/snippets.yml: context: OKsuite/snippets.yml: Get new snippet: OKsuite/all.yml: fork: OKsuite/all.yml: fork: OKSummary**************************************************tests: 4    success: 4    skipped: 0    failed: 0**************************************************```But... `require` did not work as intended, it looks like the second requirementof `log_as/admin` was not skipped.Remember that the context is forked in `fork` and also that `require` keeps trackof completed suites in the context.The context in the suite `users` is not aware of the completion of the requirementsin `snippets`. Moreover, there is no guarantee about the order of execution of thetwo `require` tasks.The solution is to require `log_as/admin` in the umbrella suite:```sh$EDITOR suite/all.yml``````yaml---- require: log_as/admin- fork: snippets- fork: users```Now, the admin is logged in at the time the contexts fork and the completedrequired suites are known in forked contexts:```shsanta test local all``````suite/log_as/admin.yml: context: OKsuite/all.yml: require: OKsuite/snippets.yml: require: SKIPPEDsuite/log_as/admin.yml already completedsuite/users.yml: require: SKIPPEDsuite/log_as/admin.yml already completedsuite/snippets.yml: List snippets: OKsuite/users.yml: Get users: OKsuite/snippets.yml: Post test snippet: OKsuite/snippets.yml: context: OKsuite/snippets.yml: Get new snippet: OKsuite/all.yml: fork: OKsuite/all.yml: fork: OKSummary**************************************************tests: 4    success: 4    skipped: 0    failed: 0**************************************************```Right. Both thread did skip the admin login as it was already done.## CustomizationCreate a plugin package```shmkdir pluginstouch plugins/__init__.py```Add a custom extractor```# the name of the file is not important. Just put stuff where it belong$EDITOR plugins/extractor.py``````pythonfrom santa.extractor import Extractorfrom santa.parser import IntFieldclass Accumulate(Extractor):""" Given an array of int as `src` extract the sum of its values """# if not set, the yaml name defaults to the lowered classname# __yaml_name__ = "accumulate"src = StringField()bind = ContextValueField()async def __call__(self, response, ctx):data = await response.json()array = data[self.src]ctx.bind(self.bind, sum(array), create=True)```This class defines an extractor as well as the yaml code required tocall it.```yaml---- test:name: Get the sum of stuffmethod: GETurl:extends: urls.defaultpath: /api/v12/stuffextract:- accumulate:src: stuffbind: stuff.count```Custom validators and context processors can be implemented the sameway.