Package: zopkio
Zopkio - A Functional and Performance Test Framework for Distributed Systems============================================================================.. image:: https://travis-ci.org/linkedin/Zopkio.svg?branch=master:target: https://travis-ci.org/linkedin/Zopkio.. image:: https://coveralls.io/repos/linkedin/Zopkio/badge.svg?branch=master&service=github:target: https://coveralls.io/github/linkedin/Zopkio?branch=masterZopkio is a test framework built to support at scale performance and functionaltesting.Installation------------Zopkio is distributed via pipTo install::(sudo) pip install zopkioIf you want to work with the latest code::git clone git@github.com:linkedin/zopkio.gitcd zopkioOnce you have downloaded the code you can run the zopkio unit tests::python setup.py testOr you can install zopkio and run the sample test::(sudo) python setup.py installzopkio examples/server_client/server_client.pyN.B the example code assumes you can ssh into your own box using yourssh keys so if your are having issues with the tests failing check yourauthorized_keys.In the past there have been issues installing one of our dependencies (Naarad)if you encounter errors installing naarad seehttps://github.com/linkedin/naarad/wiki/InstallationBasic usage-----------Use the zopkio main script::zopkio testfileZopkio takes several optional arguments::--test-only [TEST_LIST [TEST_LIST ...]]run only the named tests to help debug broken tests--machine-list [MACHINE_LIST [MACHINE_LIST ...]]mapping of logical host names to physical namesallowing the same test suite to run on differenthardware, each argument is a pair of logical name andphysical name separated by a =--config-overrides [CONFIG_OVERRIDES [CONFIG_OVERRIDES ...]]config overrides at execution time, each argument is aconfig with its value separated by a =. This has thehighest priority of all configs-d OUTPUT_DIR, --output-dir OUTPUT_DIRSpecify the output directory for logs and test results.By default, Zopkio will write to the current directory.--log-level LOG_LEVELLog level (default INFO)--console-log-level CONSOLE_LEVELConsole Log level (default ERROR)--nopassword          Disable password prompt--user USER           user to run the test as (defaults to current user)Testing with Zopkio-------------------Zopkio provides the ability to write tests that combine performance andfunctional testing across a distributed service or services.Writing tests using Zopkio should be nearly as simple as writing tests in xUnitor Nose etc.  A test suite will consist of a single file specifying fourrequired pieces:#. A deployment file#. One or more test files#. A dynamic configuration file#. A config directoryFor simplicity in the first iteratation this is assumed to be json or a pythonfile with a dictionary called  *test*.Deployment~~~~~~~~~~The deployment file should be pointed to by an entry in the dictionary called*deployment_code*. Deplyoment is one of the key features of Zopkio.Developers can write test inwhich they bring up arbtrary sets of services on multiple machines and thenwithin the tests exercise a considerable degree of control over these machines.The deployment section of code will be similar to deployment in other testframeworks but because of the increased complexity and the expectation of reuseacross multiple test suites, it can be broken into its own file.A deployment file can contain four functions:#. ``setup_suite``#. ``setup``#. ``teardown``#. ``teardown_suite``As in other test frameworks, ``setup_suite`` will run before any of tests,``setup`` will run before each test, ``teardown`` will run if ``setup`` ransuccessfully regardless of the test status, and ``teardown_suite`` will run if``setup_suite`` ran successfully regardless of any other conditions. The maindistinction in the case of this framework will be in the extended libraries tosupport deployment.In many cases the main task of the deployment code is creating a Deployer.This can be done using the SSHDeployer provided by the framework or throughcustom code. For more information about deployers see the APIs. The ``runtime``module provides a helpful ``set_deployer(service_name)`` and``get_deployer(service_name)``.  In addition to allowing the deployers to beeasily shared across functions and modules, using these functions will allowthe framework to automatically handle certain tasks such as copying logs fromthe remote hosts.  Once the deployer is created it can be used in both thesetup and teardown functions to start and stop the services.Since the ``setup`` and ``teardown`` functions run before and after each test atypical use is to restore the state of the system between tests to preventtests from leaking bugs into other tests.  If the ``setup`` or ``teardown``fails we will skip the test and mark it as a failure. In an effort to avoidwasting time with a corrupted stack there is a configuration``max_failures_per_suite_before_abort`` which can be set to determine how manytimes the frameworke will skip tests before autmatically skipping the remainingtests in that suite.In addition the entire suite is rerun parameterized by the configurations (Seeconfigs_) there is a second config ``max_suite_failures_before_abort``which behaves similarly.Test Files~~~~~~~~~~Test files are specified by an entry in the test dictionary called *test_code*,which should point to a list of test files.For each test file, the framework will execute any function with *test* in thename (no matter the case) and track if the function executes successfully. Inaddition if there is a function ``test_foo`` and a function ``validate_foo``,after all cleanup and log collection is done, if ``test_foo`` executed successfullythen ``validate_foo`` will be executed and tested for successful execution ifit fails, the original test will fail and the logs from the post execution willbe displayed. Test can be run in either a parallel mode or a serial mode. Bydefault tests are run serially without any specified order. However each test filemay specify an attribute *test_phase*. A test_phase of -1 is equivalent to serialtesting. Otherwise all tests with the same test_phase will be run in paralleltogether. Phases proceed in ascending order.Dynamic Configuration File~~~~~~~~~~~~~~~~~~~~~~~~~~The dynamic configuration component may be specified as either*dynamic_configuration_code* or *perf_code*. This module contains a numberof configurations that can be used during the running of the tests to provideinputs for the test runner. The required elements are a function to return Naaradconfigs, and functions to return the locations of the logs to fetch from theremote hosts. There are also several configs which can be placed either in thismodule as attributes or in the Master config file. The main focus of this moduleis support for Naarad. The output of the loadgeneration can be any format supported by Naarad including JMeter and CSV. Theperformacnce file can also contain rules for Naarad to use to pass/fail thegeneral performance of a run (beyond rules specific to individual tests).  Toget the most from Naarad, a Naarad config file can be provided (seehttps://github.com/linkedin/naarad/blob/master/README.md section Usage). Inorder to have Naarad support the module should provide a function``naarad_config()``. There are also two functons``machine_logs()`` and ``naarad_logs()`` that should return dictionariesfrom ``unique_ids`` to the list of logs to collect.  Machine logs are theset of logs that should not be processed by naarad... _configs:Configs-------Being able to test with different configurations is extremely important. Theframework distinguishes between three types of configs:#. master config#. test configs#. application configsMaster configs are properties which affect the way zopkio operates. Current propertiesthat are supported include:* ``max_suite_failures_before_abort``* ``max_failures_per_suite_before_abort``* ``LOGS_DIRECTORY``* ``OUTPUT_DIRECTORY``Test configs are properties which affect how the tests are run. They are specificto the tests test writer and accessible from``runtime.get_config(config_name)`` which will return the stored value or theempty string if no property with that name is present. These are the propertiesthat can be overrode by the ``config-overrides`` command line flag.some of the test configs that zopkio recognizes are:* ``loop_all_tests``* ``show_all_iterations``* ``verify_after_each_test``'loop_all_tests' repeats the entire test suite for that config for the specified number of times'show_all_iterations' shows the result in test page for each iteration of the test.'verify_after_each_test' forces the validation before moving onto the next testApplication configs are properties which affect how the remote services areconfigured. There is not currently an official way to copy these configs to remotehosts separately from the code, although there are several utilities to support it.In order to allow the same tests to run over multiple configurations, theframework interprets configs accoriding to the following rules.  All configsare grouped under a single folder.  If this foldercontains at least one subfolder, then the config files at the top level areconsidered defaults and for each subfolder of the top folder, the entire testsuite will be run using the configs within that folder (plus the defaults andconfig overrides). This is the case in which``max_suite_failures_before_abort`` will be considered. Otherwise the suitewill be run once with the top level config files and overrides.Example Tests-------------1) command : zopkio examples/server_client/server_client.py- Runs bunch of tests with multiple clients and servers deployed2) command : zopkio examples/server_client/single_server_multipleiter_inorder.py --nopassword- The individual tests have the TEST_PHASE set to be 1,2,3 respectively. This enforces order.- To run multiple iterations set loop_all_tests to be <value> in config.json file- To validate each run of the test before moving to next one set verify_after_each_test in configs- To show the pass/fail for each iteration set show_all_iterations to be true in configs- sample settings to get mulitple runs for this test#. "show_all_iterations":true,#. "verify_after_each_test":true,#. "loop_all_tests":2,3) command : zopkio examples/server_client/server_client_multiple_iteration.py- The base_tests_multiple_iteration.py module has TEST_ITER parameter set to 2.- This repeats all the tests twice but does not enfore any ordering4) command : zopkio examples/server_client/client_resilience.py- This is an example of the test recipe feature of zopkio. See test_recipes.py for recipe and test_resilience.py for example used here- This tests the kill_recovery recipe to which you pass the deployer, process list, optional restart func, recovery func and timeout- Zopkio will kill a random process of the deployer and verifies if the system can recover correctly based on recovery function before the timeout value