Package: enhancedschemathesis
Schemathesis is a tool for testing your web applications built with Open API / Swagger specifications.It reads the application schema and generates test cases which will ensure that your application is compliant with its schema.The application under test could be written in any language, the only thing you need is a valid API schema in a supported format.Supported specification versions:Swagger 2.0Open API 3.0.xMore API specifications will be added in the future.Built with:hypothesishypothesis_jsonschemapytestInspired by wonderfulswagger-conformanceproject.If you are looking for more information, then there is an article about Schemathesis:https://code.kiwi.com/schemathesis-property-based-testing-for-api-schemas-52811fd2b0a4InstallationTo install Schemathesis viapiprun the following command:pipinstallenhancedschemathesisGitter:https://gitter.im/kiwicom/schemathesisUsageThere are two basic ways to use Schemathesis:Command Line InterfaceWriting tests in PythonCLI is pretty simple to use and requires no coding, in-code approach gives more flexibility.Command Line InterfaceTheschemathesiscommand can be used to perform Schemathesis test cases:schemathesisrunhttps://example.com/api/swagger.jsonIf your application requires authorization then you can use--authoption for Basic Auth and--headerto specify
custom headers to be sent with each request.To filter your tests by endpoint name, HTTP method or Open API tags you could use-E,-M,-Toptions respectively.CLI supports passing options tohypothesis.settings. All of them are prefixed with--hypothesis-:schemathesisrun--hypothesis-max-examples=1000https://example.com/api/swagger.jsonTo speed up the testing process Schemathesis provides-w/--workersoption for concurrent test execution:schemathesisrun-w8https://example.com/api/swagger.jsonIn the example above all tests will be distributed among 8 worker threads.If you’d like to test your web app (Flask or AioHTTP for example) then there is--appoption for you:schemathesisrun--app=importable.path:app/swagger.jsonYou need to specify an importable path to the module where your app instance resides and a variable name after:that points
to your app.Note, app factories are not supported. The schema location could be:A full URL;An existing filesystem path;In-app endpoint with schema.This method is significantly faster for WSGI apps, since it doesn’t involve network.For the full list of options, run:schemathesis--help# Orschemathesisrun--helpDockerSchemathesis CLI also available as a docker imagedockerrunkiwicom/schemathesis:stablerunhttp://example.com/schema.jsonTo run it against localhost server add--network=hostparameter:dockerrun--network="host"kiwicom/schemathesis:stablerunhttp://127.0.0.1/schema.jsonPre-run CLI hookSometimes you need to execute custom code before the CLI run, for example setup an environment,
register custom string format strategies or modify Schemathesis behavior in runtime you can use--pre-runhook:schemathesis--pre-runimportable.path.to.modulerunhttps://example.com/api/swagger.jsonNOTE. This option should be passed before therunpart.The passed value will be processed as an importable Python path, where you can execute your code.
An example -https://github.com/kiwicom/schemathesis#custom-string-strategiesRegistering custom checks for CLITo add a new check for the Schemathesis CLI there is a special functionimportschemathesis@schemathesis.register_checkdefnew_check(response,case):# some awesome assertions!passThe registered check should accept aresponsewithrequests.Response/schemathesis.utils.WSGIResponsetype andcasewithschemathesis.models.Casetype.After registration, your checks will be available in Schemathesis CLI and you can use them via-ccommand line option.schemathesis--pre-runmodule.with.checksrun-cnew_checkhttps://example.com/api/swagger.jsonIn-codeTo examine your application with Schemathesis you need to:Setup & run your application, so it is accessible via the network;Write a couple of tests in Python;Run the tests viapytest.Suppose you have your application running onhttp://0.0.0.0:8080and its
schema is available athttp://0.0.0.0:8080/swagger.json.A basic test, that will verify that any data, that fit into the schema will not cause any internal server error could
look like this:# test_api.pyimportrequestsimportschemathesisschema=schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")@schema.parametrize()deftest_no_server_errors(case):# `requests` will make an appropriate call under the hoodresponse=case.call()# use `call_wsgi` if you used `schemathesis.from_wsgi`# You could use built-in checkscase.validate_response(response)# Or assert the response manuallyassertresponse.status_code<500It consists of four main parts:Schema preparation;schemathesispackage provides multiple ways to initialize the schema -from_path,from_dict,from_uri,from_fileandfrom_wsgi.Test parametrization;@schema.parametrize()generates separate tests for all endpoint/method combination available in the schema.A network call to the running application;case.calldoes it.Verifying a property you’d like to test; In the example, we verify that any app response will not indicate a server-side error (HTTP codes 5xx).NOTE. Look forfrom_wsgiusagebelowRun the tests:pytesttest_api.pyOther properties that could be tested:Any call will be processed in <50 ms - you can verify the app performance;Any unauthorized access will end with 401 HTTP response code;Each test function should have thecasefixture, that represents a single test case.ImportantCaseattributes:method- HTTP methodformatted_path- full endpoint pathheaders- HTTP headersquery- query parametersbody- request bodyYou can use them manually in network calls or can convert to a dictionary acceptable byrequests.request:importrequestsschema=schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")@schema.parametrize()deftest_no_server_errors(case):kwargs=case.as_requests_kwargs()response=requests.request(**kwargs)For each test, Schemathesis will generate a bunch of random inputs acceptable by the schema.
This data could be used to verify that your application works in the way as described in the schema or that schema describes expected behavior.By default, there will be 100 test cases per endpoint/method combination.
To limit the number of examples you could usehypothesis.settingsdecorator on your test functions:fromhypothesisimportsettings@schema.parametrize()@settings(max_examples=5)deftest_something(client,case):...To narrow down the scope of the schemathesis tests it is possible to filter by method or endpoint:@schema.parametrize(method="GET",endpoint="/pet")deftest_no_server_errors(case):...The acceptable values are regexps or list of regexps (matched withre.search).WSGI applications supportSchemathesis supports making calls to WSGI-compliant applications instead of real network calls, in this case
the test execution will go much faster.app=Flask("test_app")@app.route("/schema.json")defschema():return{...}@app.route("/v1/users",methods=["GET"])defusers():returnjsonify([{"name":"Robin"}])schema=schemathesis.from_wsgi("/schema.json",app)@schema.parametrize()deftest_no_server_errors(case):response=case.call_wsgi()assertresponse.status_code<500Explicit examplesIf the schema contains parameters examples, then they will be additionally included in the generated cases.paths:get:parameters:-in:bodyname:bodyrequired:trueschema:'#/definitions/Pet'definitions:Pet:additionalProperties:falseexample:name:Doggoproperties:name:type:stringrequired:-nametype:objectWith this Swagger schema example, there will be a case with body{"name": "Doggo"}.  Examples handled withexampledecorator from Hypothesis, more info about its behavior ishere.If you’d like to test only examples provided in the schema, you could utilize--hypothesis-phases=explicitCLI option:$schemathesisrun--hypothesis-phases=explicithttps://example.com/api/swagger.jsonOr add this decorator to your test if you use Schemathesis in your Python tests:fromhypothesisimportsettings,Phase...@schema.parametrize()@settings(phases=[Phase.explicit])deftest_api(case):...NOTE. Schemathesis does not support examples in individual properties that are specified inside Schema Object.
But examples inParameter Object,Media Type ObjectandSchema Objectare supported.
See below:...paths:/users:parameters:-in:queryname:fooexample:bar# SUPPORTED!schema:type:stringexample:spam# SUPPORTED!post:requestBody:content:application/json:schema:type:objectproperties:foo:type:stringexample:bar# NOT SUPPORTED!Direct strategies accessFor convenience you can explore the schemas and strategies manually:>>>importschemathesis>>>schema=schemathesis.from_uri("http://0.0.0.0:8080/petstore.json")>>>endpoint=schema["/v2/pet"]["POST"]>>>strategy=endpoint.as_strategy()>>>strategy.example()Case(path='/v2/pet',method='POST',path_parameters={},headers={},cookies={},query={},body={'name':'\x15.\x13\U0008f42a','photoUrls':['\x08\U0009f29a','\U000abfd6\U000427c4','']},form_data={})Schema instances implementMappingprotocol.Changing data generation behaviorIf you want to customize how data is generated, then you can use hooks of three types:Global, which are applied to all schemas;Schema-local, which are applied only for specific schema instance;Test function specific, they are applied only for a specific test function;Each hook accepts a Hypothesis strategy and should return a Hypothesis strategy:importschemathesisdefglobal_hook(strategy):returnstrategy.filter(lambdax:x["id"].isdigit())schemathesis.hooks.register("query",hook)schema=schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")defschema_hook(strategy):returnstrategy.filter(lambdax:int(x["id"])%2==0)schema.register_hook("query",schema_hook)deffunction_hook(strategy):returnstrategy.filter(lambdax:len(x["id"])>5)@schema.with_hook("query",function_hook)@schema.parametrize()deftest_api(case):...There are 6 places, where hooks can be applied and you need to pass it as the first argument toschemathesis.hooks.registerorschema.register_hook:path_parametersheaderscookiesquerybodyform_dataIt might be useful if you want to exclude certain cases that you don’t want to test, or modify the generated data, so it
will be more meaningful for the application - add existing IDs from the database, custom auth header, etc.NOTE. Global hooks are applied first.Lazy loadingIf you have a schema that is not available when the tests are collected, for example it is build with tools
likeapispecand requires an application instance available, then you can parametrize the tests from a pytest fixture.# test_api.pyimportschemathesisschema=schemathesis.from_pytest_fixture("fixture_name")@schema.parametrize()deftest_api(case):...In this case the test body will be used as a sub-test viapytest-subtestslibrary.NOTE: the used fixture should return a valid schema that could be created viaschemathesis.from_dictor otherschemathesis.from_variations.Extending schemathesisIf you’re looking for a way to extendschemathesisor reuse it in your own application, thenrunnermodule might be helpful for you.
It can run tests against the given schema URI and will do some simple checks for you.fromschemathesisimportrunnerevents=runner.prepare("http://127.0.0.1:8080/swagger.json")foreventinevents:# do something with eventrunner.preparecreates a generator that yields events of different kinds -BeforeExecution,AfterExecution, etc.
They provide a lot of useful information about what happens during tests, but handling of these events is your responsibility.
You can take some inspiration from SchemathesisCLI implementation.
See full description of events in thesource code.If you want to use Schemathesis CLI with your custom checks, look atthis sectionThe built-in checks list includes the following:Not a server error. Asserts that response’s status code is less than 500;Status code conformance. Asserts that response’s status code is listed in the schema;Content type conformance. Asserts that response’s content type is listed in the schema;Response schema conformance. Asserts that response’s content conforms to the declared schema;You can provide your custom checks to the execute function, the check is a callable that accepts one argument ofrequests.Responsetype.fromdatetimeimporttimedeltafromschemathesisimportrunner,modelsdefnot_too_long(response,case:models.Case):assertresponse.elapsed<timedelta(milliseconds=300)events=runner.prepare("http://127.0.0.1:8080/swagger.json",checks=[not_too_long])foreventinevents:# do something with eventCustom string strategiesSome string fields could use custom format and validators,
e.g.card_numberand Luhn algorithm validator.For such cases it is possible to register custom strategies:Createhypothesis.strategies.SearchStrategyobjectOptionally provide predicate function to filter valuesRegister it viaschemathesis.register_string_formatstrategy=strategies.from_regex(r"\A4[0-9]{15}\Z").filter(luhn_validator)schemathesis.register_string_format("visa_cards",strategy)Unittest supportSchemathesis supports Python’s built-inunittestframework out of the box,
you only need to specify strategies forhypothesis.given:fromunittestimportTestCasefromhypothesisimportgivenimportschemathesisschema=schemathesis.from_uri("http://0.0.0.0:8080/petstore.json")new_pet_strategy=schema["/v2/pet"]["POST"].as_strategy()classTestSchema(TestCase):@given(case=new_pet_strategy)deftest_pets(self,case):response=case.call()assertresponse.status_code<500Schema validationTo avoid obscure and hard to debug errors during test runs Schemathesis validates input schemas for conformance with the relevant spec.
If you’d like to disable this behavior use--validate-schema=falsein CLI andvalidate_schema=Falseargument in loaders.DocumentationFor the full documentation, please seehttps://schemathesis.readthedocs.io/en/latest/(WIP)Or you can look at thedocs/directory in the repository.Local developmentFirst, you need to prepare a virtual environment withpoetry.
Installpoetry(check out theinstallation guide) and run this command inside the project root:poetryinstallFor simpler local development Schemathesis includes aaiohttp-based server with the following endpoints in Swagger 2.0 schema:/api/success- always returns{"success": true}/api/failure- always returns 500/api/slow- always returns{"slow": true}after 250 ms delay/api/unsatisfiable- parameters for this endpoint are impossible to generate/api/invalid- invalid parameter definition. Usesintinstead ofinteger/api/flaky- returns 1/1 ratio of 200/500 responses/api/multipart- accepts multipart data/api/teapot- returns 418 status code, that is not listed in the schema/api/text- returnsplain/textresponses, which are not declared in the schema/api/malformed_json- returns malformed JSON withapplication/jsoncontent type headerTo start the server:./test_server.sh8081It is possible to configure available endpoints via--endpointsoption.
The value is expected to be a comma separated string with endpoint names (success,failure,slow, etc):./test_server.sh8081--endpoints=success,slowThen you could use CLI against this server:schemathesisrunhttp://127.0.0.1:8081/swagger.yaml==================================Schemathesistestsessionstarts=================================platformLinux--Python3.7.4,schemathesis-0.12.2,hypothesis-4.39.0,hypothesis_jsonschema-0.9.8rootdir:/hypothesisprofile'default'->database=DirectoryBasedExampleDatabase('/.hypothesis/examples')Schemalocation:http://127.0.0.1:8081/swagger.yamlBaseURL:http://127.0.0.1:8081Specificationversion:Swagger2.0collectedendpoints:2GET/api/slow.[50%]GET/api/success.[100%]==============================================SUMMARY==============================================not_a_server_error2/2passedPASSED=========================================2passedin0.29s=========================================Running testsYou could run tests viatox:tox-pall-oorpytestin your current environment:pytesttest/-nautoContributingAny contribution in development, testing or any other area is highly appreciated and useful to the project.Please, see theCONTRIBUTING.rstfile for more details.Python supportSchemathesis supports Python 3.6, 3.7 and 3.8.LicenseThe code in this project is licensed underMIT license.
By contributing toschemathesis, you agree that your contributions
will be licensed under its MIT license.