Package: locust-grasshopper
GrasshopperA lightweight framework for performing load tests against an environment, primarily
against an API. Grasshopper gluesLocust,Pytest, some plugins (namelyLocust InfluxDBListener) and some custom code to provide a
package that makes authoring load tests simple with very little boilerplate needed.Here are some key functionalities that this project extends on Locust:checkscustom trendstiming thresholdsstreamlined metric reporting/tagging system(only influxDB is supported right now)InstallationThis package can be installed via pip:pip install locust-grasshopperExample Load TestYou can refer to the testtest_example.pyin theexampledirectory for a basic
skeleton of how to get a load test running. In the same directory, there is also an
exampleconftest.pythat will show you how to get basic parameterization working.This test can be invoked by runningpytest example/test_example.pyin the root of
this project.This test can also be invoked via a YAML scenario file:cdexample
pytestexample_scenarios.YAML--tags=example1In this example scenario file, you can see how grasshopper_args,
grasshopper_scenario_args, and tags are being set.(back to top)Creating a load testWhen creating a new load test, the primary grasshopper function you will be using
is calledGrasshopper.launch_test. This function can be imported like so:from grasshopper.lib.grasshopper import Grasshopperlaunch_testtakes in a wide variety of args:user_classes: User classes that the runner will run. These user classes must
extend BaseJourney, which is a grasshopper class
(from grasshopper.lib.journeys.base_journey import BaseJourney). This can be a
single class, a list of classes, or a dictionary where the key is the class and
the value is the locust weight to assign to that class.**complete_configuration: In order for the test to have the correct configuration, you
must pass in the kwargs provided by thecomplete_configurationfixture. See example
load test on how to do this properly.(back to top)Scenario ArgsIf you want to parameterize your journey class, you should use thescenario_argsdict. This is the proper way to pass in values from outside of
the journey for access by the journey code. Note that each journey gets acopyon start, so the journey itself can safely modify its own dictionary
once the test is running.scenario_argsexists for any journey that extends the grasshopperbase_journeyclass.scenario_argsalso grabs fromself.defaultson initialization. For example:fromlocustimportbetween,taskfromgrasshopper.lib.journeys.base_journeyimportBaseJourneyfromgrasshopper.lib.grasshopperimportGrasshopper# a journey class with an example taskclassExampleJourney(BaseJourney):# number of seconds to wait between each taskwait_time=between(min_wait=20,max_wait=30)# this defaults dictionary will be merged into scenario_args with lower precedence# when the journey is initializeddefaults={"foo":"bar",}@taskdefexample_task:logging.info(f'foo is `{self.scenario_args.get("foo")}`.')# aggregate all metrics for the below request under the name "get google"# if name is not specified, then the full url will be the name of the metricresponse=self.client.get('https://google.com',name='get google')# the pytest test which launches the journey classdeftest_run_example_journey(complete_configuration):# update scenario args before initializationExampleJourney.update_incoming_scenario_args(complete_configuration)# launch the journeylocust_env=Grasshopper.launch_test(ExampleJourney,**complete_configuration)returnlocust_env(back to top)Commonly used grasshopper pytest arguments--runtime: Number of seconds to run each test. Set to 120 by default.--users: Max number of users that are spawned. Set to 1 by default.--spawn_rate: Number of users to spawn per second. Set to 1 by default.--shape: The name of a shape to run for the test.
If you don't specify a shape or shape instance, then the shapeDefaultwill be used,
which just runs with the users, runtime & spawn_rate specified on the command line (or picks up defaults
of 1, 1, 120s). Seeutils/shapes.pyfor available shapes and information on how to add
your own shapes.--scenario_fileIf you want a yaml file where you pre-define some args, this is how
you specify that file path. For example,scenario_file=example/scenario_example.YAML.--scenario_nameIf--scenario_filewas specified, this is the scenario name that is
within that YAML file that corresponds to the scenario you wish to run. Defaults to None.--tagsSee below example:Loop through a collection of scenarios that match some  tag.--scenario_delayAdds a delay in seconds between scenarios. Defaults to 0.--influx_hostIf you want to report your performance test metrics to some influxdb,
you must specify a host.
E.g.1.1.1.1. Defaults to None.--influx_port: Port for yourinflux_hostin the case where it is non-default.--influx_user: Username for yourinflux_host, if you have one.--influx_pwd: Password for yourinflux_host, if you have one.--grafana_host: If your grafana is a separate URL from the influxdb, you can
specify it here. If you don't, then the grafana URL will be the same as the
influxdb URL when the grasshopper object generates grafana links.(back to top)Launching tests with a configurationAll in all, there are a few ways you can actually collect and pass params to a test:Run a test with its defaultscd examplepytest test_example.py ...Run a test with a specific scenariocd examplepytest test_example.py --scenario_file=example_scenarios.YAML --scenario_name=example_scenario_1 ...Loop through a collection of scenarios that match some tagcd examplepytest example_scenarios.YAML --tags=smoke ...As shown above, this case involves passing a.YAMLscenario file into pytest instead of a.pyfile.The--scenario_fileand--scenario_nameargs will be ignored in this caseThe--tagsarg supports AND/OR operators according to the opensourcetag-matcherpackage. More info on these operators can be foundhere.If no--tagsarg is specified, then ALL the scenarios in the.yamlfile will be run.If a value is given for--scenario_delay, the test will wait that many seconds between collected scenarios.All scenarios are implicitly tagged with the scenario name to support easily selecting one single
scenario(back to top)Configuring GrasshopperGrasshopper adds a variety of parameters relating to performance testing along with a
variety of ways to set these values.Recent changes (>= 1.1.1) include an expanded set of sources, almost full access to all
arguments via every source (some exceptions outlined below), and the addition of some
new values that will be used with integrations such as report portal & slack
(integrations are NYI). These changes are made in a backwards compatible manner,
meaning all existing grasshopper tests should still run without modification. The
original fixtures and sources for configuration are deprecated, but still produce the
same behavior.All of the usualpytest argumentsalso remain available.The rest of the sections on configuration assume you are usinglocust-grasshopper>=1.1.1.(back to top)SourcesCurrently, there are 5 different sources for configuration values and they are, in
precedence ordercommand line argumentsenvironment variablesscenario values from a scenario yaml filegrasshopper configuration fileglobal defaults (currently stored in code, not configurable by consumers)Obviously, the global defaults defined by Grasshopper are not really a source for
consumers to modify, but we mention it so values don't seem to appear "out of thin air".(back to top)CategoriesThe argument list is getting lengthy, so we've broken it down into categories. These
categories are entirely for humans: better readability, understanding and ease of use.
Once they are all fully loaded by Grasshopper, they will be stored in a singleGHConfigurationobject (dict). By definition, every argument is in only one category
and there is no overlap of keys between the categories. If the same key is supplied in
multiple categories, they will be merged with the precedence order as they appear in
the table.NameScopeDescription/UsageGrasshopperSessionVariables that rarely change, may span many test runs.Test RunSessionVariables that may change per test run, but are thesame for every scenario in the runScenarioSessionVariables that may change per scenario and are oftenscenario specific; Includes user defined variables that arenot declared as command line arguments by Grasshopper.However, you may use pytest's addoptions hook in yourconftest to define them.At least one of the sections must be present in the global configuration file and
eventually this will be the same in the configuration section of a scenario in a scenario
yaml file. Categories are not used when specifying environment variables or command line
options. We recommend that you use these categories in file sources, but if a variable
is in the wrong section, it won't actually affect the configuration loading process.(back to top)Using Configuration ValuesYour test(s) may access the complete merged set of key-value pairs via the session scoped
fixturecomplete_configuration. This returns aGHConfigurationobject (dict) which
is unique to the current scenario. This value will be re-calculated for each new scenario
executed.A few perhaps not obvious notes about configuration:use the environment variable convention of all uppercase key names (e.g.RUNTIME=10)
tospecifya key-value pair via an environment valueuse the lower case key toaccessa key from theGHConfigurationobject
(e.g.x = complete_configuration("runtime")) regardless of the original source(s)use--before the key name to specify it on the command line (e.g.--runtime=10)configure a grasshopper configuration file by creating a session scoped fixture loaded
by your conftest.py calledgrasshopper_config_file_pathwhich returns the full path to a
configuration yaml file.grasshopper supports thresholds specified asa json string - required for environment variable or commandline, but also accepted
from other sourcesa dict - when passing in via thescenario_argsmethod (more details on that below)
or via a journey class'sdefaultsattr.@pytest.fixture(scope="session")defgrasshopper_config_file_path():return"path/to/your/config/file"An example grasshopper configuration file:grasshopper:influx_host:1.1.1.1test_run:users:1.0spawn_rate:1.0runtime:600scenario:key1:'value1'key2:0(back to top)Additional Extensions to the configuration loading processIf you would like to include other environment variables that might be present in the
system, you can define a fixture calledextra_env_var_keyswhich returns a list of key
names to load from theos.environ. Keys that are missing in the environment will not
be included in theGHConfigurationobject.Any environment variables that use the prefixGH_will also be included in theGHConfigurationobject. TheGH_will be stripped before adding and any names that
become zero length after the strip will be discarded. This is a mechanism to include any
scenario arguments you might like to pass via an environment variable.In the unlikely case that you need to use a different prefix to designate scenario
variables, you can define a fixture calledenv_var_prefix_keywhich returns a prefix
string. The same rules apply about which values are included in the configuration.(back to top)ChecksChecks are an assertion that is recorded as a metric.
They are useful both to ensure your test is working correctly
(e.g. are you getting a valid id back from some post that you sent)
and to evaluate if the load is causing intermittent failures
(e.g. sometimes a percentage of workflow runs don't complete correctly the load increases).
At the end of the test, checks are aggregated by their name across all journeys that
ran and then reported to the console. They are also forwarded to the DB
in the "checks" table. Here is an example of using a check:fromgrasshopper.lib.util.utilsimportcheck...response=self.client.get('https://google.com',name='get google')check("get google responded with a 200",response.status_code==200,env=self.environment,)It is worth noting that it is NOT necessary to add checks on the http codes. All the
HTTP return codes are tracked automatically by grasshopper and will be sent to the DB.
If you aren't using a DB then you might want the checks console output.(back to top)Custom TrendsCustom trends are useful when you want to time something that spans multiple HTTP
calls. They are reported to the specified database just like any other HTTP request,
but with the "CUSTOM" HTTP verb as opposed to "GET", "POST", etc. Here is an example
of using a custom trend:fromlocustimportbetween,taskfromgrasshopper.lib.util.utilsimportcustom_trend...@task@custom_trend("my_custom_trend")defgoogle_get_journey(self)foriinrange(len(10)):response=self.client.get('https://google.com',name='get google',context={'foo1':'bar1'})(back to top)ThresholdsThresholds are time-based, and can be added to any trend, whether it be a custom
trend or a request response time. Thresholds default to the 0.9 percentile of timings.
Here is an example of using a threshold:# a journey class with an example thresholdfromlocustimportbetween,taskfromgrasshopper.lib.journeys.base_journeyimportBaseJourneyfromgrasshopper.lib.grasshopperimportGrasshopperclassExampleJourney(BaseJourney):# number of seconds to wait between each taskwait_time=between(min_wait=20,max_wait=30)@taskdefexample_task:self.client.get("https://google.com",name="get google")@task@custom_trend("my custom trend")defexample_task_custom_trend:time.sleep(10)# the pytest test which launches the journey class, thresholds could be# parameterized here as well.deftest_run_example_journey(complete_configuration):ExampleJourney.update_incoming_scenario_args(complete_configuration)ExampleJourney.update_incoming_scenario_args({"thresholds":{"get google":{"type":"get","limit":4000# 4 second HTTP response threshold},"my custom trend":{"type":"custom","limit":11000# 11 second custom trend threshold}}})locust_env=Grasshopper.launch_test(ExampleJourney,**complete_configuration)returnlocust_envThresholds can also be defined for individual YAML scenarios. Refer to thethresholdskey inexample/example_scenarios.YAMLfor how to use thresholds for YAML scenarios.After a test has concluded, trend/threshold data can be found inlocust_env.stats.trends.
This data is also reported to the console at the end of each test.(back to top)Time Series DB Reporting and TaggingAdditional design details about how a database listener works with grasshopper/locust can be
found in theDatabase Listener Design Documentation.When you specify a time series database URL param tolaunch_test, such asinflux_host, all metrics will be automatically reported to tables within thelocusttimeseries database via the specified URL. These tables include:locust_checks: check name, check passed, etc.locust_events: test started, test stopped, etc.locust_exceptions: error messageslocust_requests: HTTP requests and custom trendsTo run the influxdb/grafana locally, you can use the docker-compose file in the example directory:cdexample/observability_infrastructure
docker-composeup-dand then you can access the grafana UI atlocalhost. The default username/password isadmin/admin.
To then run a test which reports to this influxdb just add the--influx_host=localhosthandle.There are a few ways you can pass in extra tags which
will be reported to the time series DB:HTTP Request TaggingAll HTTP requests are automatically tagged with their name. If you want to pass in
extra tags for a particular HTTP request, you can pass them in
as a dictionary for thecontextparam when making a request. For example:self.client.get('https://google.com',name='get google',context={'foo':'bar'})The tags on this metric would then be:{'name': 'get google', 'foo': 'bar'}which
would get forwarded to the database if specified.Check TaggingWhen defining a check, you can pass in extra tags with thetagsparameter:fromgrasshopper.lib.util.utilsimportcheck...response=self.client.get('https://google.com',name='get google',context={'foo1':'bar1'})check("get google responded with a 200",response.status_code==200,env=self.environment,tags={'foo2':'bar2'})Custom Trend TaggingSince custom trends are decorators, they do not have access to
non-static class variables when defined. Therefore, you must use theextra_tag_keysparam, which is an array of keys that exist in the journey's
scenario_args. So for example, if a journey had the scenario args{"foo" : "bar"}and you wanted to tag
a custom trend based on the "foo" scenario arg key, you would do something like this:fromlocustimportbetween,taskfromgrasshopper.lib.util.utilsimportcustom_trend...@task@custom_trend("my_custom_trend",extra_tag_keys=["foo"])defgoogle_get_journey(self)foriinrange(len(10)):response=self.client.get('https://google.com',name='get google',context={'foo1':'bar1'})(back to top)Project RoadmapCustom TrendsChecksThresholdsTaggingInfluxDB metric reportingdocker-compose template for influxdb/grafanaPrometheusDB metric reportingSlack reportingReportPortal reportingSee the open issues for a full list of proposed features (and known issues).(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Make sure unit tests pass (pytest tests/unit)Add unit tests to keep coverage up, if necessaryCommit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)