Package: loadero-python
Loadero-PythonPython client for managingLoaderotests.Loadero-Python provides easy-to-use programmatic access to Loadero API. Allows
to manage tests, participants, asserts, runs and other Loadero resources, start
and stop tests and extract test run results. Example usage might be running
Loadero tests as a part of CI/CD.Table of ContentsInstallationUsageAPI AccessInitializationWorking with Existing ResourcesCreating a TestRunning a TestPollingStopping Test ExecutionGetting ResultsParticipant ResultsLog RetrievalExtracting Failed AssertsChecking metricsFiltering and PaginationFilteringPaginationStructureAPI clientResourcesResource ParamsConstantsResource APIContributingInstallationLoadero-Python is available onPyPIand can be installed by runningpipinstallloadero-pythonUsageAPI AccessBefore using the Loadero-Python client an API access token needs to be acquired.
This can be done in the projects settings page in Loadero web-app.
More information about the API can be found in theLoadero wikiNoteAccess tokens are project specific and cannot be used across multiple
projects. Make sure to specify the project ID in the request for an access
token.InitializationAfter acquiring the access token Loadero-Python needs to be initialized with it.
Loadero-Python uses a singleton objectAPIClientfromloadero_python.api_clientmodule to perform all requests to the API. SinceAPIClientis a singleton it needs to be initialized once like sofromloadero_python.api_clientimportAPIClientAPIClient(project_id=1234,access_token="your_access_token",)Further examples will not includeAPIClientinitialization. It is assumed that
the client has been initialized at an earlier step.Working with Existing ResourcesLoadero resources have a tree-like structure hence there can be child and parent
resources.project
   |
   |----tests----groups-----participants
   |      |
   |      |------asserts----assert_preconditions
   |
   |----files
   |
   |----runs-----resultsEvery parent resource can read all of its child resources.Projectclass fromloadero_python.resources.projectmodule provides an entry
point to access all resources.Projectclass can be imported withfromloadero_python.resources.projectimportProjectAll tests in a project can be read withtests,pagination,filters=Project().tests()Notice, that project ID was not specified in this example, this is because theAPIClienthas already been initialized with the project ID and corresponding
access token.testsis a list ofTestobjects fromloadero_python.resources.testmodulepaginationis aPaginationParamsobject fromloadero_python.resources.paginationmodule.filtersis a python dictionary of applied filters.A more detailed explanation ofpaginationandfiltersreturn values is
available in theFiltering and Paginationsection.Creating a TestWith an initializedAPIClientLoadero-Python can now manage resources in the
project. Test is a resource one of many in Loadero-Python. More information
about all the resources that Loadero-Python provides can be found in theStructuresection. This usage guide cannot demonstrate all of
Loadero-Python's functionality hence will cover only common use case scenarios
starting with creating a test.Test resource is contained within theloadero_python.resources.testmodule.
From itTest,TestParams, andScriptclasses need to be imported.fromloadero_python.resources.testimportTest,TestParams,ScriptAdditionally,TestModeandIncrementStrategyclassificator constant
enumerations need to be imported fromloadero_python.resources.classificator.
They will be used for test attribute definitions.fromloadero_python.resources.classificatorimportTestMode,IncrementStrategyTest attributes can be specified in two ways. Directly as arguments in params
initialization.test=Test(params=TestParams(name="my first loadero python test",start_interval=1,participant_timeout=10*60,# ten minutesmode=TestMode.TM_LOAD,increment_strategy=IncrementStrategy.IS_LINEAR,script=Script(content='print("hello test script")'),)).create()or with builder methods.test=Test(params=TestParams().with_name("my second loadero python test").with_start_interval(1).with_participant_timeout(10*60)# ten minutes.with_mode(TestMode.TM_PERFORMANCE).with_increment_strategy(IncrementStrategy.IS_RANDOM).with_script(Script(content='print("hello test script")'))).create()Resource create and update operations have required and optional attributes. If
a required attribute is missing the API call will fail. Loadero-Python checks if
all of the required attributes have been populated before making the API call
and raises an exception if one or more required attributes are missing.For test resources, the required attributes are:namestart_intervalparticipant_timeoutmodeincrement_strategyAfter the create operation completes thetestobject will have a few more of
its attributes populated. Any resource attributes can be simply printed.print(test)This will output a JSON object representation of the resource.{"id":1234,"name":"my first loadero python test","start_interval":1,"participant_timeout":600,"mode":"load","increment_strategy":"linear","script":"print(\"hello test script\")","created":"2022-08-25 15:33:04+00:00","updated":"2022-08-25 15:33:04+00:00","script_file_id":12345}Different output formats can be achieved by usingto_dictandto_dict_fullresource params methods. Both methods return a python dictionary representation
of the resource.to_dictwill return only the required attributes and optional
attributes if present.to_dict_fullwill return all attributes present.Noteto_dictwill raise an exception if one or more required attribute is
missing.importyamlprint(yaml.dump(test.params.to_dict_full()))created:"2022-08-2515:33:04+00:00"id:1234increment_strategy:linearmode:loadname:my first loadero python testparticipant_timeout:600script:print("hello test script")script_file_id:12345start_interval:1updated:"2022-08-2515:33:04+00:00"Running a TestTo run a test the only required attribute is test ID.For thetestobject from previous examples, thetest_idattribute has been
populated by create operation, so it can simply be run by calling thelaunchmethod.run=test.launch()If a test ID is known it can be run directly.run=Test(test_id=1234).launch()All tests in a project can be run withfortestinProject().tests()[0]:test.launch()PollingAfter a test has been launched, waiting for the test to finish can be achieved
with thepollmethod.run.poll()By default, thepollmethod will make an API call to check if the test
execution has finished every 15 seconds and will wait up to 12 hours. This
functionality can be customized with theintervalandtimeoutarguments.# will poll every 5 seconds and will wait up to 10 minutes.run.poll(interval=5.0,timeout=10*60.0)If test execution does not finish within the specified timeout, thepollmethod will raise an exception.Stopping Test ExecutionIf test execution needs to be prematurely stopped, it can be done with thestopmethod.run.stop()Notestoponly sends an API request that starts a Loadero procedure of
stopping the test. This process is NOT immediate. Even though thestopAPI
request completes relatively quickly, the test can remain running for a while
longer.NoteIf another process is polling the test execution, it will automatically
stop if the test is stopped.Getting ResultsAfter the test run finishes execution, therunobject already contains many
useful attributes that may be used in result analysis. The attributes are stored
on therun.paramsfield.run.paramsis anRunParamsobject fromloadero_python.resources.runmodule.print(run.params.success_rate)Participant Resultsrunobject describes a result overview of the whole test. To get a more
detailed result information about each test participant's results needs to be
read.results,_,_=run.Results()result=results[0]The ignored return values are pagination and filters. They are not relevant for
result retrieval, hence they are omitted. A more detailed explanation of these
values is available in theFiltering and Paginationsection.resultsis a list ofResultobjects fromloadero_python.resources.resultmodule. A single result corresponds to a single participant in the test.Resultjust like a regular resource object has aparamsfield of typeResultParamsthat contains its attributes. The result resource has the largest
amount of attributes, so this showcase will cover only common use cases.Log Retrievalimportrequestsresp=requests.get(result.params.log_paths.selenium)ifnotresp:print("failed to download selenium log")exit(1)withopen(f"selenium_log_of_result_{result.params.result_id}","w")asf:f.write(resp.text)result.params.log_paths.seleniumis an URL to an Selenium log. It first needs
to be downloaded using the HTTP libraryrequests. Then it can be written to a
file.Extracting Failed AssertsBefore extracting failed asserts.AssertStatusclassificator constant
enumeration needs to be imported.fromloadero_python.resources.classificatorimportAssertStatusfailed_asserts=[]forresult_assertinresult.params.asserts:ifresult_assert.status==AssertStatus.AS_FAILfailed_asserts.append(result_assert)Checking metricsLoadero tests collect various different metrics from CPU, RAM, and network usage
to video and audio quality indicators. Loadero organizes these metrics with
metric base paths - a path-like string that uniquely identifies metric data. For
example, CPU usage metric data is described by the metric base pathmachine/cpu/used.After test execution finishes Loadero processes the collected metric data by
applying aggregator functions.totalminimummaximumaveragestandard deviationrelative standard deviation1st percentile5th percentile25th percentile50th percentile75th percentile95th percentile99th percentileThe result is a single float value identified by a metric path. For example, the
maximal CPU usage is described by the metric path -machine/cpu/used/maxIn Loadero-Python metric base paths -MetricBasePathand metric paths -MetricPathare constant enumerations of all the available metric and metric
base paths. Contained within theloadero_python.resources.metric_pathmodule.To access a specific metricMetricBasePathenumeration needs to be imported.fromloadero_python.resources.metric_pathimportMetricBasePathThen a specific metric can be checked like thisifresult.params.metricsisNoneorresult.metrics.machineisNone:print("result has no machine metrics")exit(1)ifMetricBasePath.MACHINE_CPU_AVAILABLEnotinresult.params.metrics.machine:print("result has no machine cpu available metric")exit(1)if(result.params.metrics.machine[MetricBasePath.MACHINE_CPU_AVAILABLE].average<10.0):print("test is well configured. efficient usage of CPU resources")Thenot Nonechecks are required because some or all metrics for a result can
be missing. For example, non-WebRTC tests will not have anywebrtcmetrics.Filtering and PaginationRead-all operations have the option to limit the number of resources returned,
offset a limited read-all operation by some amount of resources and filter out
undesired resources.This is done by passing a query params argument when performing a read-all
operation.QueryParamsclass is contained inloadero_python.resources.resourcemodule
and can be imported withfromloadero_python.resources.resourceimportQueryParamsFilteringFilters are resource-specific and are defined in each resource module. For
example, test resource filters are defined in theTestFilterKeyconstant
enumeration in theloadero_python.resources.testmodule.TestFilterKeycan be imported withfromloadero_python.resources.testimportTestFilterKeyNow test read all operations can be filtered like thistests,_,filters=Project().tests(query_params=QueryParams().filter(TestFilterKey.PARTICIPANT_TIMEOUT_TO,10*60# ten minutes))The ignored value is pagination. It can be ignored because pagination will
contain useful page information when limit and offset query_params have been
applied.This will return tests whose participant timeout attribute is smaller than ten
minutes.filtersis a python dictionary with the applied filters.PaginationWhen performing a read-all operation that will return many resources it is good
practice to limit the number of resources returned and perform multiple smaller
reads. This can be achieved by limiting and offsetting the number of resources
returned.tests,pagination,_=Project().tests(query_params=QueryParams().limit(20).offset(10))This time the ignored value is filters. It can be ignored because no filters
were applied.Let's assume that the project has 28 tests numbered from 1 to 28, then this
read-all operation would return tests with numbers from 11 to 28. This happens
because the returned resources were offset by 10 and the next resource after the
10th is the 11th. Only 18 resources were returned because the remaining
resources after offset were smaller than the defined limit - 20.paginationis an instancePaginationParamsclass fromloadero_python.resources.resourcemodule. It contains information about the
applied limit and offset, plus additional information describing how many
resources remain to be read.StructureThe Loadero-Python library structure is similar to the Loadero API structure.
The main structural components are:API clientresources and operations with themAPI clientContained withinloadero_python.api_clientmodule is theAPIClientsingleton
object. It needs to be initialized once with the project ID and access token.
All requests to Loadero API are done with theAPIClientobject. It adds the
required headers to make valid API requests. Additionally, theAPIClientrate
limits all requests to be compliant with Loadero API's access limits. Rate
limiting can be opted out on initialization.APIClient(project_id=1234,access_token="your_access_token",rate_limit=False,)ResourcesEach resource has a separate module.Resource classModuleAssertPreconditionloadero_python.resources.assert_preconditionAssertloadero_python.resources.assert_resourceFileloadero_python.resources.fileGrouploadero_python.resources.groupParticipantloadero_python.resources.participantProjectloadero_python.resources.projectResultloadero_python.resources.resultRunParticipantloadero_python.resources.run_participantRunloadero_python.resources.runTestloadero_python.resources.testAll resource classes have a similar structure:Resource classes have an attributeparamsthat is used to store the data of
a single instance of the resource. Read more about resource paramshere.most resources implement common CRUD manipulation methods -create,read,update,delete,duplicate. Some resources do not have these methods
because they are impossible or not available via API access. For example,Projectresource has onlyreadmethod because API access prohibits updates
to this resource.resources that can have child resources have methods for reading all the child
resources.# reads all groups in testgroups,_,_=Test(test_id=123).groups()Resource ParamsEvery resource has a resource params class that stores the data of a single
resource instance.Resource params class for each resource is available in the resources module.Resource params classModuleAssertPreconditionParamsloadero_python.resources.assert_preconditionAssertParamsloadero_python.resources.assert_resourceFileParamsloadero_python.resources.fileGroupParamsloadero_python.resources.groupParticipantParamsloadero_python.resources.participantProjectParamsloadero_python.resources.projectResultParamsloadero_python.resources.resultRunParticipantParamsloadero_python.resources.run_participantRunParamsloadero_python.resources.runTestParamsloadero_python.resources.testResource params classes provide access to the resource attributes.# read a test and print its nameprint(Test(test_id=123).read().params.name)ConstantsLoadero-Python has two modules for constants.loadero_python.resources.classificatorfor classificator constant
enumerations.loadero_python.resources.metric_pathfor metric path and metric base path
constant enumerations.Resource APIEvery resource has its own API class also available in the resources module.Resource classModuleAssertPreconditionAPIloadero_python.resources.assert_preconditionAssertAPIloadero_python.resources.assert_resourceFileAPIloadero_python.resources.fileGroupAPIloadero_python.resources.groupParticipantAPIloadero_python.resources.participantProjectAPIloadero_python.resources.projectResultAPIloadero_python.resources.resultRunParticipantAPIloadero_python.resources.run_participantRunAPIloadero_python.resources.runTestAPIloadero_python.resources.testResource API class implements all the available API operations of that resource.
Resource API classes are internally used by all resources, but the class on its
own is not very useful.ContributingFound a bug? - Feel free to open anissue.Would like to request a feature? - Open an issue describing the request and the
reason for it or contact Loaderosupport.Want to contribute? - Openissuesis a good place where
to find stuff that needs to be worked on.