Package: schemathesis
Schemathesis: catch crashes, validate specs, and save timeDocumentation:https://schemathesis.readthedocs.io/en/stable/Chat:https://discord.gg/R9ASRAmHnAWhy Schemathesis?Schemathesis is a tool that automates your API testing to catch crashes and spec violations. Built on top of the widely-usedHypothesisframework for property-based testing, it offers the following advantages:🕒Time-SavingAutomatically generates test cases, freeing you from manual test writing.🔍ComprehensiveUtilizes fuzzing techniques to probe both common and edge-case scenarios, including those you might overlook.🛠️FlexibleSupports OpenAPI, GraphQL, and can work even with partially complete schemas. Only the parts describing data generation or responses are required.🎛️CustomizableCustomize the framework by writing Python extensions to modify almost any aspect of the testing process.🔄ReproducibleGenerates code samples to help you quickly replicate and investigate any failing test cases.Quick DemoWith a summary right in your PRs:Getting StartedChoose from multiple ways to start testing your API with Schemathesis.💡 Your API schema can be either a URL or a local path to a JSON/YAML file.💻 Command-Line InterfaceQuick and easy for those who prefer the command line.PythonInstall via pip:python -m pip install schemathesisRun testsstrun--checksallhttps://example.schemathesis.io/openapi.jsonDockerPull Docker image:docker pull schemathesis/schemathesis:stableRun testsdockerrunschemathesis/schemathesis:stablerun--checksallhttps://example.schemathesis.io/openapi.json🐍 Python LibraryFor more control and customization, integrate Schemathesis into your Python codebase.Install via pip:python -m pip install schemathesisAdd to your tests:importschemathesisschema=schemathesis.from_uri("https://example.schemathesis.io/openapi.json")@schema.parametrize()deftest_api(case):case.call_and_validate()💡 See a complete working example project in the/exampledirectory.:octocat: GitHub IntegrationGitHub ActionsRun Schemathesis tests as a part of your CI/CD pipeline.Add this YAML configuration to your GitHub Actions:api-tests:runs-on:ubuntu-22.04steps:-uses:schemathesis/action@v1with:schema:"https://example.schemathesis.io/openapi.json"# OPTIONAL. Add Schemathesis.io token for pull request reportstoken:${{ secrets.SCHEMATHESIS_TOKEN }}For more details, check out ourGitHub Actionrepository.💡 See ourGitHub Tutorialfor a step-by-step guidance.GitHub AppReceive automatic comments in your pull requests and updates on GitHub checks status. Requires usage of our SaaS platform.Install theGitHub app.Enable in your repository settings.Software as a ServiceIf you prefer an all-in-one solution with quick setup, we have afree tieravailable.How it worksHere’s a simplified overview of how Schemathesis operates:Test Generation: Using the API schema to create a test generator that you can fine-tune to your testing requirements.Execution and Adaptation: Sending tests to the API and adapting through statistical models and heuristics to optimize subsequent cases based on responses.Analysis and Minimization: Checking responses to identify issues. Minimizing means simplifying failing test cases for easier debugging.Stateful Testing: Running multistep tests to assess API operations in both isolated and integrated scenarios.Reporting: Generating detailed reports with insights and cURL commands for easy issue reproduction.Research Findings on Open-Source API Testing ToolsOur study, presented at the44th International Conference on Software Engineering, highlighted Schemathesis's performance:Defect Detection: identified a total of755 bugsin16 services, finding between1.4× to 4.5× more defectsthan the second-best tool in each case.High Reliability: consistently operates seamlessly on any project, ensuring unwavering stability and reliability.Explore the full paper athttps://ieeexplore.ieee.org/document/9793781or pre-print athttps://arxiv.org/abs/2112.10328Testimonials"The world needs modern, spec-based API tests, so we can deliver APIs as-designed. Schemathesis is the right tool for that job."Emmanuel Paraskakis -Level 250"Schemathesis is the only sane way to thoroughly test an API."Zdenek Nemec -superface.ai"The tool is absolutely amazing as it can do the negative scenario testing instead of me and much faster! Before I was doing the same tests in Postman client. But it's much slower and brings maintenance burden."Luděk Nový -JetBrains"Schemathesis is the best tool for fuzz testing of REST API on the market. We are at Red Hat use it for examining our applications in functional and integrations testing levels."Dmitry Misharov -RedHat"There are different levels of usability and documentation quality among these tools which have been reported, where Schemathesis clearly stands out among the most user-friendly and industry-strength tools."Testing RESTful APIs: A Survey -a research paper by Golmohammadi, at alContributingWe welcome contributions in code and are especially interested in learning about your use cases.
Understanding how you use Schemathesis helps us extend its capabilities to better meet your needs.Feel free to discuss ideas and questions throughGitHub issuesor on ourDiscord channel.
For more details on how to contribute, see ourcontributing guidelines.Let's make it better together 🤝Your feedback is essential for improving Schemathesis.
By sharing your thoughts, you help us develop features that meet your needs and expedite bug fixes.Why Give Feedback: Your input directly influences future updates, making the tool more effective for you.How to Provide Feedback: Usethis formto share your experience.Data Privacy: We value your privacy. All data is kept confidential and may be used in anonymized form to improve our test suite and documentation.Thank you for contributing to making Schemathesis better! 👍Commercial supportIf you're a large enterprise or startup seeking specialized assistance, we offer commercial support to help you integrate Schemathesis effectively into your workflows.
This includes:Quicker response time for your queries.Direct consultation to work closely with your API specification, optimizing the Schemathesis setup for your specific needs.To discuss a custom support arrangement that best suits your organization, please contact our support team atsupport@schemathesis.io.Additional contentPapersDeriving Semantics-Aware Fuzzers from Web API Schemasby@Zac-HDand@Stranger6667Description: Explores the automation of API testing through semantics-aware fuzzing. Presented at ICSE 2022.Date: 20 Dec 2021ArticlesAuto-Generating & Validating OpenAPI Docs in Rust: A Streamlined Approach with Utoipa and SchemathesisbyidentecoDescription: Demonstrates OpenAPI doc generation with Utoipa and validating it with Schemathesis.Date: 01 Jun 2023Testing APIFlask with schemathesisby@pamelafoxDescription: Explains how to test APIFlask applications using Schemathesis.Date: 27 Feb 2023Using Hypothesis and Schemathesis to Test FastAPIby@amalshajiDescription: Discusses property-based testing in FastAPI with Hypothesis and Schemathesis.Date: 06 Sep 2022How to use Schemathesis to test Flask API in GitHub Actionsby@lina-is-hereDescription: Guides you through setting up Schemathesis with Flask API in GitHub Actions.Date: 04 Aug 2022Using API schemas for property-based testing(RUS) about Schemathesis by@Stranger6667Description: Covers the usage of Schemathesis for property-based API testing.Date: 07 Sep 2021Schemathesis: property-based testing for API schemasby@Stranger6667Description: Introduces property-based testing for OpenAPI schemas using Schemathesis.Date: 26 Nov 2019VideosAPI Fuzzing: What it is and why you should use itbyJosé Haro PeraltaDescription: A comprehensive overview and demo of Schemathesis.Date: 14 Feb 2023Schemathesis tutorialwith an accompanyingvideobyRed HatDescription: Provides a hands-on tutorial for API testing with Schemathesis.Date: 09 Feb 2023Effective API schemas testingfrom DevConf.cz by@Stranger6667Description: Talks about using Schemathesis for property-based API schema testing.Date: 24 Mar 2021API-schema-based testing with schemathesisfrom EuroPython 2020 by@hultnerDescription: Introduces property-based API testing with Schemathesis.Date: 23 Jul 2020LicenseThis project is licensed under the terms of theMIT license.