Package: pytest-automation
Pytest AutomationFor automating test creation. This plugin lets you send a list of tests (arguments defined in yaml's), to python methods. For creating an agile test suite.This plugin is compatible to run alongside vanilla pytest tests, without interfering with them. The custom CLI arguments for this plugin won't affect those tests (Including --skip-all).How to setup the Test Suite1) Install the plugin2) Add both required filespytest-config.ymlpytest-managers.py3) Write the yaml testsyaml requirementswriting yaml tests exampleyaml test philosophy4) Using conftest.py for extra FlexibilityAdding CLI OptionsRunning scripts before/after the SuiteFull list of HooksHow to run testsRunning the TestsHow to Build from SourceHow to setup the Test Suite1) Install the pluginRun the following:python3-mpipinstallpytest-automation2) Add both required filesIt doesn't matter where in your project these exist, but names are case-sensitive, andexactlyone of each should exist. If you need to ignore a sub-directory with it's own pytest suite, use--ignore <dir>(More info on ignoringhere).pytest-config.ymlThis file defines where each individualyml testgets sent, to which method in thepytest-managers.pyfile.This also allows you to have multiple types of tests in the same file. (Useful for example, having a test_known_bugs.yml you can exclude from pipelines).This file is required to have onetest_typeskey. This holds a list, of each type of test (ortest type) you'd like in your suite. Each element in the list, is in the format {"test title": {all_test_info}} (Shown inthis examplebelow).pytest-config.yml example:# Contents of pytest-config.ymltest_types:-For running addition tests:required_keys:["x_add","y_add"]method:test_PythonsAddition-For running factorial tests:required_keys:factor_numrequired_in_title:test-factormethod:test_NumpysFactorvariables:throw_on_negative:FalsePossible filters in eachtest type:required_keys: The yml test must contain ALL these keys to run with thistest type.required_in_title: Check if the test title contains this string (case insensitive). NOTE: With basic values, it's easy to accidentally match new tests later on. Best practice is to use something like "test-[something]", instead of just "[something]".Eachyml testwill go through thetest_typeslistin order, and the following things will happen:ONLY keys that are declared will be checked. Also you can have multiplerequired_*keys in the sametest type, and they ALL have to match for the test to run.Note: This means if it hasNOrequired_*keys, ALL tests will match it, sonotests will continue pass thattest type.If the test is matched to that type, the function undermethodwill be looked for inpytest-managers.pyand called.If NOtest typeis matched, that test will fail, and the next will run.In the pytest-configexampleabove, the test_PythonsAddition will only be called, if that yml test contains both the x_add and y_add keys. With the test_NumpysFactor, it'll only be called if that yml test has the factor_num key, AND it's in the test_factorials.yml file.Test Type Variables:In the pytest-configexampleabove, you can see the following key in the secondtest type:variables:throw_on_negative:FalseThisvariableskey is optional. It'll pass it's contents into eachpytest-managersmethod, under thetest_type_varsparam. This is useful for declaring url's, endpoints, etc. More info on what arguments get passed to themethodhere.pytest-managers.pyWhen ayml testis matched withtest type, that test type'smethodis imported from this file, and ran.pytest-managers.py example# Contents of pytest-managers.pyfromcustom_addimportrun_add_testfromcustom_factorimportrun_fact_test# The methods here matchs the 'method' key in 'pytest-config.yml' example. (Required)deftest_PythonsAddition(**args):run_add_test(**args)# *OR* just run test here:test_info=args["test_info"]asserttest_info["x_add"]+test_info["y_add"]==test_info["answer"]deftest_NumpysFactor(**args):run_fact_test(**args)# *OR* just run test here:test_factor=args["test_info"]["factor_num"]assertfactor(test_factor)==args["test_info"]["answer"]Like with the first line of each method, it'srecommendedto have the testing code in another file, and just call it from this one. This helps keeps the suite organized for longer test methods. Even if you import other methods, ONLY the methods defined in this file can be loaded from themethodkey inpytest-config.yml.Args passed into each TestEach test in pytest-managers.py should only accept**argsas their one param. That'll allow the plugin to add extra keys in the future, without breaking older tests. The following keys are currently guaranteed:config: Apytest config (ext link)object. For interacting with pytest itself (i.e getting whichcli optionswere used when running the suite)test_info: The parameters from the yml file. This is passed into the python manager, and what makes each test unique. More infohere.test_type_vars: How to declare variables for atest type, and not have them hard-coded/duplicated in each test. (i.e. what api endpoint to target). More infohere.3) Write the yaml testsThis is where you define each individual test. Each test is matched to atest type, then ran.yaml requirements:All yaml names must start with "test_", and end with either ".yml" or ".yaml".The list of tests, must be under aSINGLEtestskey, inside the file. If more than one key exists, they'll override each other, and you won't run all your tests.Each test in the list, is a single dict of the format {"test title": {ALL_TEST_INFO}}writing yaml tests example:# Contents of test_MyCustomMath.yml# These examples match the "pytest-config.yml" example, with required_keys above.tests:-Basic addition test:x_add:5y_add:7answer:12-Factorial Basic test:factor_num:4answer:24-test-factor Factorial special case zero:factor_num:0answer:1-Negative addition test:x_add:5y_add:-7answer:-2The first test gets matched to the additiontest typein the pytest-configexample, containing the two required keys.The second andwouldget matched to the factorial test,exceptit doesn't have "test-factor" in it's title, likerequired_in_titlesays it should, so it doesn't get matched toanythingand fails.The third testdoeshave "test-factor" in it's title, so it runs as normal.The fourth test gets matched to the additiontest type, so it runs with thatmethodin thepytest-config.yml.IMPORTANT NOTE: Before passing each yml test to theirmethod, the plugin will move the titleintothe info, withtitlebecoming key. So thetitlekey is reserved:# Before passing into the test, this test info:-Negative addition test:x_add:5y_add:-7answer:-2# Will automatically become:-title:Negative addition testx_add:5y_add:-7answer:-2# To make accessing each item easier to access.(Example on how to access thetest_infovalueshere).yaml test philosophy:One key idea behind organizing tests into yamls, is you can move each individual yml test between files, and the test will still behave as expected.This means you can have "test_known_bugs.yml" to exclude from build pipelines, or "test_prod_only.yml" that only gets executed against your prod environment. etc.This also means we can't decide whichtest typeto run a test with, based on what file it's in. Otherwise, as soon as you moved a test from this file, to "known_bugs.yml", it's behavior might change.4) Usingconftest.pyfor extra FlexibilityYou're able to usepytest hooks (ext link)to run commands before the suite, add CLI options, etc; by declaring them in aconftest.pyfile inside your project.(NOTE: Fixtures not currently supported, but hopefully coming soon!)Adding CLI OptionsAdd thepytest_addoption (ext link)hook for declaring CLI arguments.# Contents of conftest.pydefpytest_addoption(parser):# addoption is built on top of argparse, same syntax:parser.addoption("--api",action="store",default="local",help="Which API to hit when running tests (LOCAL/DEV/TEST/PROD, or url).")# Add other 'parser.addoption' calls here for each argumentThen each tests can look at what was the user passed in, through theconfig.# Contents of some test file, called by pytest-managers.pydeftest_CheckQueries(**args):api=args["config"].getoption("--api")# ... Continue to run whatever checks after this ...Running scripts before/after the SuiteAdd thepytest_sessionstart (ext link)orpytest_sessionfinish (ext link)hooks for adding startup/tear down logic.# Contents of conftest.pydefpytest_sessionstart(session):# Maybe you need a directory for dumping temp files:temp_dir="some/tmp/dir"ifos.path.isdir(temp_dir):shutil.rmtree(temp_dir)os.mkdir(temp_dir)defpytest_sessionfinish(session,exitstatus):# Maybe send a email if the suite failspassFull list of HooksYou can find the full listhere (ext link).How to run testsRunning the Tests:pytest<pytestandpluginargshere><PATH><customconftestargshere># Example:pytest-nauto-s-tbshort--dfknown_bugs.--apidevelCommon pytest CLI args:-n int=> The number of threads to use. Make sure tests are thread-safe. (Default = 1, installpytest-xdist (ext link)to use).-s=> If python prints anything, show it to your console.-x=> Quit as soon as the first test fails(-v|-vv|-vvv) => How much info to print for each test--tb ("short" | "long" | ...)=> How much error to print, when a test fails. (Other options available, more infohere (ext link))--ignore DIR=> Ignore this directory from your suite. Works both with vanilla pytest tests, and pytest-automation files. Useful if you pull another repo into yours, and it has it's own test suite. (More infohere (ext link)).Custom pytest-automation args:Filter what tests to run:--only-run-name str,--dont-run-name str(--on str/--dn str) => (Can use multiple times) If the name of the testcontainsthis value, only/don't run accordingly.--only-run-file str,--dont-run-file str(--of str/--df str) => (Can use multiple times) If the file the test is incontainsthis value, only/don't run accordingly.--only-run-type str,--dont-run-type str(--ot str/--dt str) => (Can use multiple times) Looks at the title inpytest-config.yml, and if itcontainsthis value, only/don't run accordingly.skip-all: Skips all pytest-automation yaml tests. (Doesn't skip vanilla pytest methods).PATH:The path to start collecting tests /conftest.pyfiles from.Normally just "." for current directory. (i.e. 'pyest . ')Custom conftest CLI args:Any argumentsyoudefine in your projectsconftest.pyfile. More infohere.How to Build from Source1) Create your environment:# Upgrade pip to latest and greatestpython3-mpipinstall--upgradepip# Install tool for creating environmentspython3-mpipinstallvirtualenv# Create the environmentvirtualenv--python=python3~/PytestAuto-env# Jump inside it. (You'll need to run this for each new shell)source~/PytestAuto-env/bin/activateYou should see your terminal start with "(PytestAuto-env)" now.2) Install the required packages:# Because of the --python=python3 above, you can now just run 'python'# Install the packages needed to run setup.py:python-mpipinstall<Path-to-this-repo-root># OR if you want to run the test suite:python-mpipinstall<Path-to-this-repo-root>/requirements.txt3) Install it:Or run this after each change to the source.# NOTE: The "--upgrade" is needed incase it's already installed.#  (i.e. Don't use cached version).python-mpipinstall--upgrade<Path-to-this-repo-root>