Package: pysurge
pysurgeHave existing Python code? Want to use it to set up some simple performance tests quickly? Trypysurge, a python-based tool that allows you to quickly write simple performance tests in python.ExampleSee theexamplein this project to see how a simple HTTP load test can be quickly configured and executed.InstallationThis project is available on pypi:pip install pysurgePerformance tests... in Python?Yes, Python has its limitations when it comes to concurrency and scale. However, sometimes you don't need more scale than what Python can deliver. This is especially true when testing cloud-based applications that "scale out" (i.e., spin up more replicas when demand increases). Load testing can be done at lower scale against a single replica and results can be extrapolated.There's a lot of complex performance test tools out there that are designed to load test applications at very high scale. This project aims to offer a "quick and easy" solution for those who don't need the massive scale. The scale you can achieve is of course limited by the demands of your test, python's concurrency drawbacks (i.e., the GIL), and your hardware's specifications (number of CPU cores being a key factor).Many test engineers already have code written in Python. The goal of this project is to prevent engineers from having to significantly re-write their logic just to get it running at higher scale. If the code is thread-safe, it can be imported and executed concurrently usingpysurge.How doespysurgework?Writing a TestCase classTests are defined by the test writer to inherit from thepysurge.TestCaseclass with the following functions:startup&shutdown-- these define "per-session" setup and teardown tasks that will happen once per test case class. Here you can also define variables/functions/etc. that all the fired test instances will share.setup&teardown-- define the "per-test" setup and teardown tasks that are executed for every unique test instance that is fired offrun-- defines the actual test run logicthe__init__can also define custom kwargs -- useful if you want to tweak the behavior of the test slightly.Test metricsA test case class can store float/int metrics in a dict calledself.metrics. For example, you may want to store a metric that tracks the "response time" of your service every time the test fires. In your test'srunfunction, you may have code that looks something like this:start_time = time.time()
do_stuff()
end_time = time.time()
self.metrics['response_time'] = end_time - start_timeAt the end of the test run, the metric stored for every instance that fired is averaged together and reported.Configuring the pysurge runpysurgeuses a YAML configuration file which defines:which test classes you want to runthe rate (# of tests per sec) to fire them off atany kwargs that should be passed to theTestCase.__init__-- useful if you want to run the same test class concurrently but with different optionsInitiating the runYou can initiate a test run with:$ pysurge -c </path/to/config.yaml> -d <run duration in minutes>What happens during the run?To make use of thread pools as efficiently as possible,pysurgeusesmultiprocessingto create a child process for each CPU core available on your system. A threadpool is created for eachTestClasswithin every child process, and the test "fire rate" is then distributed evenly across each child process. For example, if you define a test case calledMyTestthat should be fired "once every second", and you have 4 CPU cores, the following will happen:Four child processes will be createdA threadpool will be created in each child process which runs the code fromMyTestEach threadpool will fire the test at a rate of "1 every 4 sec"Each child process staggers the start of its test fires by 1 second to make the "firing rate" even.Once the test run has "ramped up" and all thread pools have begun firing, this results in 1 instance ofMyTestfiring every 1 second.As you add more test classes, more thread pools are added to each child process. The number of child processes will remain constant.Since the tests are executed in thread pools, they cannot be overly CPU intensive -- ideal candidates are quicker I/O bound tests interacting with remote APIs/services. Tests should take care to use CPU resources efficiently -- i.e. focus on I/O bound tasks (like interacting with remote network services) and remember to 'sleep' while looping to avoid sucking up CPU cycles.Stopping the runYou can sendSIGINT/ctrl+Cto end the run prematurely. This will NOT abruptly end the application, but rather it will trigger all child processes to begin shutting down cleanly. Otherwise, the run will execute for thedurationspecified at run time.Viewing resultspysurgeoutputs logs from all running threads tostdoutIt will also produce a rudimentary report at the end of its run.The below example is for a test calledUploadTest. Two instances of this test were configured to run:UploadTestwith no kwargsUploadTestwith kwargslegacy=TrueA metric calledupload->db_time_in_secwas stored every time the test fired. The average time is reported in the report.2018-09-14 18:51:37,590 MainProcess MainThread INFO ---- Results for test '<UploadTest kwargs={}> -- upload test, legacy: False' ----
2018-09-14 18:51:37,590 MainProcess MainThread INFO >>> Ran for 1849.129022 sec, fired 6988 total tests
2018-09-14 18:51:37,590 MainProcess MainThread INFO >>> Successes: 6987, failures: 1
2018-09-14 18:51:37,591 MainProcess MainThread INFO ---- Metrics for test '<UploadTest kwargs={}> -- upload test, legacy: False' ----
2018-09-14 18:51:37,591 MainProcess MainThread INFO >>> upload->db_time_in_sec: 2.668292
2018-09-14 18:51:37,591 MainProcess MainThread INFO
2018-09-14 18:51:37,591 MainProcess MainThread INFO ---- Results for test '<UploadTest kwargs={'legacy': True}> -- upload test, legacy: True' ----
2018-09-14 18:51:37,591 MainProcess MainThread INFO >>> Ran for 1849.129022 sec, fired 7004 total tests
2018-09-14 18:51:37,591 MainProcess MainThread INFO >>> Successes: 7002, failures: 2
2018-09-14 18:51:37,591 MainProcess MainThread INFO ---- Metrics for test '<UploadTest kwargs={'legacy': True}> -- upload test, legacy: True' ----
2018-09-14 18:51:37,591 MainProcess MainThread INFO >>> upload->db_time_in_sec: 1.025963
2018-09-14 18:51:37,591 MainProcess MainThread INFOWhat doespysurgeNOT do?Very large scale testsDistributed load testing (orchestrating and firing tests across multiple hosts)Advanced reporting