Package: verifit
AboutThis is a Python library aimed at developers, that simplifies setting up and using automatic system testing for several types of projects.I've named itverifitas a contraction ofVerify It!, i.e. "Make sure your system works fine!"IntroductionWhy Developer-Based Automatic System Testing?Automatic system (or application-level) testing done by developers is a must for every project.  Even if you have a dedicated QA team, automatic system testing is a very valuable tool for making sure your product works as expected:It allows developers to be very confident of the quality of the products that they ship.It lowers the risk of breaking older functionality as you add a new one.It makes bug investigation easier - if you know that the behavior shown in the bug passes in your automated tests, then the source must be elsewhere, such as misconfiguration or wrong parameter.It makes QA team's life easier by helping them design their tests.Why This Libray?For the above to work, though, the developers must be maintaining their tests along with the code base, or even better, do TDD.This library aims to make a developer's life easier by providing a handful of commonly used options, such as environment variables or making authorized HTTPS requests.  The goal is to get started with automated tests right away!Side Note: Black Box TestingThere are multiple ways of writing automated tests, and this library is not tied to any particular methodology.However, one particular testing methodology I favor isblack box testing, in which you treat your system as a black box, andnever check its internals.  I find it has quite a few advantages, and I thought I'd list some of them here:Easy to put in place.  You just hit your system in the same way its clients would.  No need to write code that checks databases, message queues, or who knows what.Can serve as acceptance or end-to-end testing.It's not affected by implementation changes in the system that's being tested.OverviewThis library supports any type of tests that you write in Python, using any testing framework.For our sample tests, we use the PyTest runner & framework.Normally, if you're using PyTest and a few other standard Python modules, you can test pretty much anything quite fast, without needing a library.  However, as you start doing this, you soon realize you're going to need some things over and over again in your tests.  This small library provides a few helpers for making a developer's automatic testing easier:Quick configuration & data sharing between tests.A cache for speeding up things such as getting access tokens, or any other data that is being reused.Log in and caching the access token.Some helper tools, like date, JWT, accessing dictionary properties, etc.Web-Sockets testing.  (I couldn't find a Python package that makes testing Web-Sockets as easy as calling a couple of functions.)This repo consists of the actual library, which is insrc/verifit/, and some sample tests from which you can inspire when writing your own.Quick Start with Writing Tests1. Install Required LibrariesInstall Python 3.6 or higher.Install PyTest.Copy therequirements from the sample teststo your PyTest project.  Adjust them to suit your needs, and then install them.2. Read the Sample Tests & ConfigsWe have atests/directory that shows how to use this library. Please make sure to read this section while browsing that folder. It will get you started real quick.2.1. Sample ConfigsThe following configs are made in order to run the sample tests:requirements.txt: Listsverifit, as well as other packages used by it or by our sample tests..dev.env: Hosts configuration for our tests, such as endpoints or users.__init__.py: Needed for local imports.conftest.py: Needed for local imports.2.2. Sample TestsThe sample tests use dummy online services or commands in order to show how to test various types of applications, and how to use our lib:post_service/post/test_post.py. Tests simple APIs using HTTP and GraphQL servers.post_service/drivers/test_post_via_drivers.py. The same astest_post.py, but showcasing using different drivers.If you need to run your tests through multiple channels, then it's best if you keep your tests unified, and add a couple of layers to help directing the tests through the right channel.Assume you need to 'Post' items via two channels, 'foo', and 'bar'.  The layers would be:Test declarationSketches the high-level test case: do the action - in this case, 'post' an item with some data - and verify the resultThe test never calls directly to any of the drivers, but to a Domain-Specific Language, aka DSLBoththeactionand theverificationare done via the DSL - this is important, becauseonly the particular driver can possibly know how to do the action or check its results; attempting to abstract this would add needless overheadDSL implementationBased on the 'driver' parameter, it calls to the corresponding driverWe intentionally are missing one driver implementation, which is skipped both in the test and in the DSLDrivers implementationEach driver 'posts' items in the way it knowsWe have a couple of drivers as a demoThe files:drivers.pycontains the list of our particular driversThe test itself is straightforward, as it imports the DSL, calls it, and verifies the result.  Note that the test is skipped for the 'bar' driver, because in our example, we are pretending that the 'bar' channel has no way to 'Post an item.dsl_post.py: The DSL uses the driver lib to call to the right driver from a driver map it provides.  In our case, one driver is omitted from the map, because we know it cannot 'post'.driver_*.py: The '*' driver to 'post' an item.  One driver is intentionally missing, to showcase how you handle such situations.To run this test via the 'bar' driver, do this:DRIVER=bar pytest . -k 'post_via_drivers'By default,DRIVERisfoo.shopping_service/login/test_login.py. Here we test log in to a server.shopping_service/login/test_products.py. This one does the following:Log in from cache.  This returns the cached token if it is valid, or logs in and caches the token before returning it.Call to a private endpoint, passing in an authorization header with a cached access token.shopping_service/login/test_carts.py. It shows Gherkin features and BDD.echo_service/test_echo_service.py.  Shows how to simplify testing WebSockets using our library.Note: As of2023-04-26, the online server that we were using for this sample test stopped accepting the API key that they provide.  As a result, we've marked this test as skipped.date_service/test_date.py.  Shows how to test CLI programs using this lib: it runs the shell commanddatewith some arguments, and verifies the resulting output.self_check/test_self.py.  Tests the lib itself.kitchen_service/test_kitchen_service.py.  Showcase doing a Web UI test using Cypress.IO and their kitchen sink sample page.To prepare for running the sample tests that are included with this project, do this:cdtests
pipinstall-rrequirements.txtTo run all tests:pytest.To run only tests that are related to posts and shopping:pytest.-k'post_service or shopping_service'3. Write Some TestsUsing the sample configs and tests as an inspiration, go ahead and write and run your own!ReferenceThe entire library is coded using Functional Programming principles.  Thus, you will see:Pure functions, that return the same result given the same input, and that do not have side effects.Immutability & disciplined state - functions do not alter state outside them, and there's no shared state between functions.Currying - at most one argument for each function.  Note that we're using Python's built-inkwargsas the "single argument".Higher-order functions - that return another function.The memoize pattern used to make sure a single instance of a thing exists (in our case the config store).No global variables, not even a singleton - memoize takes care of this.EnvironmentThe environment configuration is parsed usingdotenv.  This automatically parses.<ENV>.env, where<ENV>is the environment, by defaultdev.  In absence of this file, you can pass in the required configuration as environment variables, e.g.:POST_SERVICE_1_URL=https://jsonplaceholder.typicode.com DRIVER=foo POST_SERVICE_2_URL=https://graphqlzero.almansi.me/api pytest . -k post.Helperscache.py.  Implements a simple cache, that is being stored in a file named.<env>-cache.jsonfrom thetestsdirectory.config.py.  Loads configuration via thedotenvpackage, and returns a unique store for getting/setting values across tests.date_tools.py.  Some simple date tools.driver.py.  The driver lib is pretty simple:It gets the current driver from the environmentIt calls the corresponding function from a driver name / function dictIt also offers a helper function to skip a test for a particular driverjson_web_token.py.  Decoding and extracting data from a JWT.login.py.  Functions to:Log in using a driver, which is a plain function.  We use a driver for this because logging in is particular to the API that each project has.  The driver does the actual call to the API endpoint for login, and returns the access token.  The framework caches the access token and its expiration date.Log in from a cached token using a driver.Building authorization values:For HTTP headers.For the Python GraphQL client.prop.py: Access dictionary properties without raising exceptions, and with default values.retrieve.py. Shortcut functions for calling to HTTP or GraphQL endpoints, with unified logging.web_sockets.py.  Simplifies Web-Sockets testing by offering functions for listening in background for received packages, and sending data.DevelopmentLibraryIf you want to develop, build, and upload this library to PyPI, do this:First, set up a few things:Install Python 3.6 or higher.Install the requirements fromrequirements.txt.The lib code is insrc/verifit.  Make sure to use proper.imports to avoid importing from the installedverifitlibrary accidentally.Once your code changes are ready and documented, do the following to upload:Commit everything to GIT.Check Manifest. Run:check-manifestIncrease the version inpyproject.toml:[project]version="X.Y.Z"Increase the minimum version intests/requirements.txt:verifit >= X.Y.ZBuild package. Run:python-mbuildCheck build. Run:twinecheckdist/verifit-X.Y.Z*Upload to Test.PyPI. Run:twineupload-rtestpypidist/verifit-X.Y.Z*Install manually from Test.PyPI. Run:pipinstall-ihttps://test.pypi.org/simpleverifit==X.Y.Z(First time it may fail, in this case, rerun the above command.)Run the sample tests:cdtests/&&pytest.If everything goes well, upload to PyPI. Run:twineuploaddist/verifit-X.Y.Z*Install sample tests requirements. Run:cdtests/&&pipinstall-rrequirements.txt(First time it may fail, in this case, rerun the above command.)Run the sample tests again:cdtests/&&pytest.If all went well, they should pass.You can now push to GIT.Sample TestsSetup:Go totests/kitchen_service/uiand runyarn install.Install sample tests requirements. Run:cdtests/&&pipinstall-rrequirements.txtChange tests code, then make sure to test, document, & push your changes.