Package: hedra
Hedra - Testing at scalePackageHedraVersion0.7.19Webhttps://hedra.devDownloadhttps://pypi.org/project/hedra/Sourcehttps://github.com/scorbettUM/hedraKeywordsperformance, testing, async, distributed, graph, DAG, workflowHedra is a Python performance and scalable unit/integration testing framework that makes creating and running complex test workflows easy.These workflows are written as directed acrylic graphs in Python, where each graph is specified as a collection of Python classes referred to asstages. Each Stage may then specify async Python methods which are then wrapped in Python decorators (referred to ashooks), which that Stage will then execute. The hook wrapping a method tells Hedra both what the action does and when to execute it. In combination, stages and hooks allow you to craft test workflows that can mimic real-world user behavior, optimize framework performance, or interact with a variety of Hedra's powerful integrations.Why Hedra?Understanding how your application performs under load can provide valuable insights - allowing you to spot issues with latency, memory usage, and stability. However, performance test tools providing these insights are often difficult to use and lack the ability to simulate complex user interaction at scale. Hedra was built to solve these problems by allowing developers and test engineers to author performance tests as sophisticated and scalable workflows. Hedra adheres to the following tenants:Speed and efficiency by defaultRegardless of whether running on your personal laptop or distributed across a cluster, Hedra isfast, capable of generating millions of requests or interactions per minute and without consuming excessive memory. Hedra pushes the limits of Python to achieve this, embracing the latest in Python async and multiprocessing language features to achieve optimal execution performance.Run with ease anywhereAuthoring, managing, and running test workflows is easy. Hedra includes integrations with Git to facilitate easy management of collections of graphs viaProjects, the ability to generate flexible starter test templates, and an API that is both fast and intuitive to understand. Distributed use almost exactly mirrors local operation, reducing the learning curve for more complex deployments.Flexibility and and painless extensibilityHedra ships with support for HTTP, HTTP2, Websockets, and UDP out of the box. GraphQL, GRPC, and Playwright are available simply by installing the (optional) dependency packages. Hedra offers JSON and CSV results output by default, with 28 additional results reporting options readily available by likewise installing the required dependencies.Likewise, Hedra offers a comprehensive plugin system. You can easily write a plugin to test your Postgresql database or integrate a third party service, with CLI-generated templates to guide you and full type hints support throughout. Unlike other frameworks, no additional compilation or build steps are required - just write your plugin, import it, and include it in the appropriate Stage in your test graph.Requirements and Getting StartedHedra has been tested on Python versions 3.8.6+, though we recommend using Python 3.10+. You should likewise have the latest LTS version of OpenSSL, build-essential, and other common Unix dependencies installed (if running on a Unix-based OS).Warning: Hedra has currently only been tested on the latest LTS versions of Ubuntu and Debian. Other official OS support is coming Mar. 2023.InstallingTo install Hedra run:pip install hedraVerify the installation was was successful by running the command:hedra --helpwhich should outputCreating your first graphGet started by running Hedra's:hedra graph create <path/to/graph_name.py>command in an empty directory to generate a basic test from a template. For example, run:hedra graph create example.pywhich will output the following:and generate the the test below in the specifiedexample.pyfile:fromhedraimport(Setup,Execute,action,Analyze,JSONConfig,Submit,depends,)classSetupStage(Setup):batch_size=1000total_time='1m'@depends(SetupStage)classExecuteHTTPStage(Execute):@action()asyncdefhttp_get(self):returnawaitself.client.http.get('https://<url_here>')@depends(ExecuteHTTPStage)classAnalyzeStage(Analyze):pass@depends(AnalyzeStage)classSubmitJSONResultsStage(Submit):config=JSONConfig(events_filepath='./events.json',metrics_filepath='./metrics.json')We'll explain this graph below, but for now  - replace the string'https://<url_here>'with'https://httpbin.org/get'.Before running our test, if on a Unix system, we may need to set the maximum number of open files above its current limit. This can be done
by running:ulimit -n 256000note that you can provide any number here, as long as it is greater than thebatch_sizespecified in theSetupStageStage. With that, we're ready run our first test by executing:hedra graph run example.pyHedra will load the test graph file, parse/validate/setup the stages specified, then begin executing your test:The test will take a minute or two to run, but once complete you should see:You have officially created and run your first test graph!DevelopmentLocal development requires at-minimum Python 3.8.6, though 3.10.0+ is recommended. To setup your environment run:python3 -m venv ~/.hedra && \
source ~/.hedra/bin/activate && \
git clone https://github.com/scorbettUM/hedra.git && \
cd hedra && \
pip install --no-cache -r requirements.in && \
python setup.py developTo develop or work with any of the additional provided engines, references the dependency tables below.Engines, Personas, Algorithms, and ReportersMuch of Hedra's extensibility comes in the form of both extensive integrations/options and plugin capabilities for four main framework features:EnginesEngines are the underlying protocol or library integrations required for Hedra to performance test your application (for example HTTP, UDP, Playwright). Hedra currently supports the following Engines, with additional install requirements shown if necessary:EngineAdditional Install OptionDependenciesHTTPN/AN/AHTTP2N/Ah2, hpackHTTP3 (unstable)pip install hedra[http3]aioquicUDPN/AN/AWebsocketN/AN/AGRPCpip install hedra[grpc]grpcio grpco-tools, h2, hpackGraphQLpip install hedra[graphql]gqlGraphQL-HTTP2pip install hedra[graphql]gql, h2, hpackPlaywrightpip install hedra[playwright] && playwright installplaywrightPersonasPersonas are responsible for scheduling when@action()or@task()hooks execute over the specified Execute stage's test duration. No additional install dependencies are required for Personas, and the following personas are currently supported out-of-box:PersonaSetup Config NameDescriptionApproximate Distribution (unstable)approximate-distributionHedra automatically adjusts the batch size after each batch spawns according to the concurrency at the current distribution step. This Persona is only available to and is selected by default if a Variant of an Experiment is assigned a distribution.BatchedbatchedExecutes each action or task hook in batches of the specified size, with an optional wait between each batch spawningConstant Arrival Rateconstant-arrivalHedra automatically adjusts the batch size after each batch spawns based upon the number of hooks that have completed, attempting to achievebatch_sizecompletions per batchConstant Spawn Rateconstant-spawnLikeBatched, but cycles through actions before waitingbatch_intervaltime.DefaultN/ACycles through all action/task hooks in the Execute stage, resulting in a (mostly) even distribution of executionNo-Waitno-waitCycles through all action/task hooks in the Execute stage with no memory usage or other waits.WARNING: This persona may cause OOM.RampedrampedStarts at a batch size ofbatch_gradient*batch_size. Batch size increases by the gradient each batch with an optional wait between each batch spawningRamped Intervalramped-intervalExecutesbatch_sizehooks before waitingbatch_gradient*batch_intervaltime. Interval increases by the gradient each batchSortedsortedExecutes each action/task hook in batches of the specified size and in the order provided to each hook's (optional)orderparameterWeightedweightedExecutes action/task hooks in batches of the specified size, with each batch being generated from a sampled distribution based upon that action's weightAlgorithmsAlgorithms are used by HedraOptimizestages to calculate maximal test config options likebatch_size,batch_gradient, and/orbatch_interval. All out-of-box supported algorithms usescikit-learnand include:AlgorithmSetup Config NameDescriptionSHGshgUsesscikit-learn's Simple Global Homology (SHGO) global optimization algorithmDual Annealingdual-annealingUsesscikit-learn's Dual Annealing global optimization algorithmDifferential Evolutiondiff-evolutionUsesscikit-learn's Differential Evolution global optimization algorithmPoint Optimizer (unstable)point-optimizerUses a custom least-squares algorithm. Can only be used by assigning a distribution to a Variant stage for an Experiment.ReportersReporters are the integrations Hedra uses for submitting aggregated and unaggregated results (for example, to a MySQL database via the MySQL reporter). Hedra currently supports the following Reporters, with additional install requirements shown if necessary:EngineAdditional Install OptionDependenciesAWS Lambdapip install hedra[aws]boto3AWS Timestreampip install hedra[aws]boto3Big Querypip install hedra[google]google-cloud-bigqueryBig Tablepip install hedra[google]google-cloud-bigtableCassandrapip install hedra[cassandra]cassandra-driverCloudwatchpip install hedra[aws]boto3CosmosDBpip install hedra[azure]azure-cosmosCSVN/AN/ADatadogpip install hedra[datadog]datadogDogStatsDpip install hedra[statsd]aio_statsdGoogle Cloud Storagepip install hedra[google]google-cloud-storageGraphitepip install hedra[statsd]aio_statsdHoneycombpip install hedra[honeycomb]libhoneyInfluxDBpip install hedra[influxdb]influxdb_clientJSONN/AN/AKafkapip install hedra[kafka]aiokafkaMongoDBpip install hedra[mongodb]motorMySQLpip install hedra[sql]aiomysql, sqlalchemyNetDatapip install hedra[statsd]aio_statsdNew Relicpip install hedra[newrelic]newrelicPostgresqlpip install hedra[sql]aiopg, psycopg2-binary, sqlalchemyPrometheuspip install hedra[prometheus]prometheus-client, prometheus-client-apiRedispip install hedra[redis]redis, aioredisS3pip install hedra[aws]boto3Snowflakepip install hedra[snowflake]snowflake-connector-python, sqlalchemySQLite3pip install hedra[sql]sqlalchemyStatsDpip install hedra[statsd]aio_statsdTelegrafpip install hedra[statsd]aio_statsdTelegrafStatsDpip install hedra[statsd]aio_statsdTimescaleDBpip install hedra[sql]aiopg, psycopg2-binary, sqlalchemyXMLpip install hedra[xml]dicttoxmlResourcesHedra's official and full documentation is currently being written and will be linked here soon!LicenseThis software is licensed under the MIT License. See the LICENSE file in the top distribution directory for the full license text.ContributingHedra will be open to general contributions starting Fall, 2023 (once the distributed rewrite and general testing is complete). Until then, feel
free to use Hedra on your local machine and report any bugs or issues you find!Code of ConductHedra has adopted and follows theContributor Covenant code of conduct.
If you observe behavior that violates those rules please report to:NameEmailTwitterSean Corbettsean.corbett@umontana.edu@sc_codeum