Package: blazetest-beta
BlazetestWhat is the problem with tests?When we deal with long-running and large number of tests, we may come across the common problem that it can take hours or days to finish.
Moreover, when using standalone servers, it may become costly. Monitoring those tests and getting detailed logs also can be extremely inconvenient.
Our solution is BlazeTest, CLI tool, which resolves the problem through the time and cost-effective solution of using cloud functions.What is BlazeTest?In order to solve above-mentioned problems, we developed a CLI tool, which will take care of all aspects of testing on the cloud,
such as collection, deployment, parallel execution of the tests, and detailed monitoring and logging collection.
BlazeTest helps to bring every challenging part of the testing process in one place through custom configuration and additional useful features, such as retrying failed tests.BlazeTest saves time and cost for the test execution and minimizes additional overhead of setting up testing pipelines.Benefits of BlazeTestParallel test execution.BlazeTest CLI introduces parallel test execution, enabling multiple tests to run simultaneously.
This fundamental feature effectively minimizes test execution time and optimizes resource utilization through deploying your tests to AWS Lambda.
Tests that used to run sequentially can now execute concurrently, dramatically reducing overall test suite
execution time and according costs.Flaky Test Mitigation.The BlazeTest CLI offers a robust flaky test mitigation mechanism.
In scenarios where tests exhibit intermittent failures, the tool automatically reruns the affected tests for a
configurable number of times. By isolating and handling flaky tests, the CLI ensures that the entire test run is
less susceptible to the adverse effects of flakiness, resulting in more consistent and reliable test outcomes.Enhanced Debugging Support.The BlazeTest CLI incorporates insightful logging and reporting capabilities,
facilitating easier identification of potential issues. In the event of test failures, comprehensive logs are
generated, containing valuable information that aids in debugging efforts. This accelerated debugging process
enables developers to swiftly address any problems, improving the overall development lifecycle.ArchitectureHere is a glance to the BlazeTest CLI from the architecture perspective:InstallationPre-RequisitesRequirementsPython3.7 or higher.OS should be Linux/AMD 64.Pulumiis installed. See below for details.AWS Policy required for setting up cloud infrastructure is set up.Cloud Platform Setup (Pulumi)BlazeTest uses Cloud platforms, such as AWS, for running the project tests of users.
Specifically, as a user, you should provide account credentials to the CLI or in theConfigfor it to be able to set up resources on your account and run
your project tests. How to pass those credentials will be discussed inCLI Reference.BlazeTest uses Pulumi to set up the cloud infrastructure. To install Pulumi use:curl -fsSL https://get.pulumi.com | shDocker pre-requisitesBlazeTest builds the project using Docker and installs all of your project dependencies in the Docker image.
It is done for ensuring the project integrity. BlazeTest uses its ownDocker imageto build the project and install all of the necessary dependencies to run tests.For building the project smoothly, ensure the following:The project should have a package manager. You can use one of the following to manage your dependencies:pip(requirements.txt).poetry(pyproject.toml).pipenv(Pipfile).BlazeTest builds the Docker image, context of which will be your project base. To include only necessary files and directories, you can use.dockerignorefile in the project base to exclude files not used by tests.InstallTo install BlazeTest, you can use any package manager. To install withpip, you can use:pip install blazetestFor using the beta version, please use:pip install blazetest-betaTo install withpoetry:poetry add blazetestConfigurationConfiguration is required for setting up the testing sessions with BlazeTest CLI in the way that you need.
It allows you to name the resources BlazeTest CLI creates in the cloud platform account.
Additionally, it can help you to select the tests that you need and execute them the way you want through PyTest collection and execution arguments.Configuration variablesIn order to configure the CLI, you should use the.tomlfile.By default, BlazeTest usesblazetest.tomlfile located at the project root.
However, any location can be used with specifying--config-pathoption while running CLI. Example:blazetest --config-path /path/to/toml/file.tomlConfiguration variables currently can be divided into4 categories:[general]configuration variables.[cloud]configuration variables: used for connecting to cloud account (for now, only AWS is supported) and create resources with given names.[framework]configuration variables: used for PyTest, PlayWright and more.[browsers]configuration variables: used for configuring the version of browsers.Variable nameSectionTypeRequiredDescriptionExamplelicense_key[general.license]stringYes, if not passed anywhere else(online activation) License key. Can be set with SPEEDYTEST_LICENSE environment variable or can be passed as --license-key CLI optionXXXXX-XXXXX-XXXXX-XXXXXlicense_file[general.license]stringYes, if not passed anywhere else(offline activation) File path or remote url license file/files/ActivationFile.skmtags[cloud.aws]key and value mappingsNoGlobal tags that are attached to all the cloud resources created by BlazeTest. Default is empty{tag1 = "tag1value", tag2 = "tag2value"}region[cloud.aws]stringYesAWS Region. Can be set with AWS_REGION environment variableeu-west-1ecr_repository_prefix[cloud.aws]stringNoAWS Elastic Container Registry (ECR) Repository name. It is where your project Docker image will be uploaded. Default is "blazetest-repo"blazetest-reporesource_prefix[cloud.aws]stringNoResource prefix. Used for naming the Lambda and CloudFormation. Default is "blazetest-stack"blazetest-stacks3_bucket_prefix[cloud.aws]stringNoAWS Simple Storage Service (S3) bucket name. Used for storing the test results. Default is "blazetest-s3"blazetest-s3lambda_function_memory_size[cloud.aws]integerNoAWS Lambda function memory size. Default is 40964096lambda_function_timeout[cloud.aws]integerNoAWS Lambda function timeout. Default is 900900collection_args[framework.pytest]array of stringsNoArguments passed to Pytest while collecting/selecting tests (not executing them). See PyTest docs for possible options.["tests/module1", "-m", "feature1"]execution_args[framework.pytest]array of stringNoArguments passed to Pytest while executing tests. See PyTest docs for possible options.["-s", "-vv", "--durations=10"]failure_retry_attempts[general.flake_detection]integerNoHow many times to retry the failed tests. Retrial happens after the initial test run ends. Default is 01fail_on_flake[general.flake_detection]booleanNoWhether to exit with fail or success when the flaky tests are found. By default, BlazeTest will fail if the flaky test is foundtrueremove_flakes[general.flake_detection]booleanNoIf true, BlazeTest will consider flaky tests as failed and will not include them in the JUnit XML report as flaky. Default is falsefalseexit_on_flake_detection[general.flake_detection]booleanNoIf true, when the failed tests are retried, if the test passed, it will mark it as flaky and will not retry this test again. Otherwise, it will retry the testfailure_retry_attemptstime even if it has passed during the retrial. Default is true.true:::info
For PyTest arguments, please see the usage examples in the pytest documentation:https://docs.pytest.org/en/7.1.x/contents.html.
:::Difference between collection and execution arguments for PyTestIn the workflow of BlazeTest CLI, concepts of collection and execution can be explained in a simple way:Collection is selection of the tests that we need.There are several options that PyTest allows us to collect the tests, such as the directory the tests are located and the markers that test use.Execution is actually running the tests.There is a variety of options you can run your tests with PyTest. For example, you can increase verbosity with-voption, or capture the output with-soption whilerunning the tests.ExampleHere you can see the exampleblazetest.tomlfile:[general.license]
license_key = "ASDSA-ASDSA-ASDSA-ASDSA"

[general.flake_detection]
failure_retry_attempts = 3
fail_on_flake = false
remove_flakes = false
exit_on_flake_detection = false

[cloud.aws]
region = "eu-west-1"
ecr_repository_prefix = "my-example-repository"
resource_prefix = "example-stack-name"
s3_bucket_prefix = "example-bucket"
tags = {key1 = "value1", key2 = "value2"}

[framework.pytest]
collection_args = ["tests/module1", "-m", "feature1"] 
execution_args = ["-s", "-vv"]

[browsers.chrome]
version = "107.0"! You don't need to indicate--junitxmlflag as it will be defined automatically.CLI Command Reference! Use double quotes for argument values with spaces. Example: --project "demo project"BlazeTest has 3 main commands:blazetest run: main command, which executes the main logic.blazetest rerun: command used for rerunning the session.blazetest sessions: shows all runs and reruns history.Blazetest Runblazetestrun[OPTIONS]Available options:KeyRequiredDescriptionExample-k, --license-key-k or -l(online activation) License key. Can be set with SPEEDYTEST_LICENSE environment variable or can be included in project configuration-k XXXXX-XXXXX-XXXXX-XXXXX-l, --license-file-k or -l(offline activation) File path or remote url license file-l /files/ActivationFile.skm-ak, --aws-access-key-idYesAWS Access Key ID. Can be set with AWS_ACCESS_KEY_ID environment variable-awskey AKIAIOSFODNN7EXAMPLE-as, --aws-secret-access-keyYesAWS Secret Access Key. Can be set with AWS_SECRET_ACCESS_KEY environment variable-awssecret wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY-config, --config-pathNoPath to the configuration file (should be .toml file), default is blazetest.toml in the folder CLI is executed. If not found, raises an error-config /config/blazetest.toml-lo, --logsNoEnables logs and shows them in the stdout. Default is enabled. When disabled, saved to blazetest.log file in the folder CLI is executed-awskey XXX -awssecret XXX --logs=disabled-lk, --lokiNoLoki API Key. If provided, logs are sent to the Loki Log aggregation system of Railflow-lk XXXX [108 characters long string]-d, --diagNoExecutes one trial test to make sure everything works. The tests are not collected, instead dummy test is sent to Lambda-awskey XXX -awssecret XXX -d-t, --tagsNoTags specified for the AWS Lambda function. The tags will be attached to the created Lambda function instance-t tag1=tag1value,tag2=tag2value-de, --debugNoEnables debugging output-awskey XXX -awssecret XXX -deblazetestrun-k<licensekey>-ak<aws-access-key-id>-as<aws-secret-access-key>-config<directoryname>/<configurationfilename>-lk<lokiapikey>-t<tag1>=<tagvalue1>,<tag2>=<tagvalue2>Blazetest Rerunblazetestrerun[OPTIONS]Available options:KeyRequiredDescriptionExample-u, --uuidYesSession UUID, which you want to rerun. Can be found by commandblazetest sessions-u a2bb567ablazetestrerun--uuida2bb567a-lk<lokiapikey>Blazetest Sessionsblazetestsessions[OPTIONS]Available options:KeyRequiredDescriptionExample-i, --includeNoWhich type of runs to include in the output of the history of sessions. Can be: "all", "runs", "reruns". Default is "all"-i runs-t, --tagsNoOption to filter the sessions by tags. If not specified, all results are shown.-t name=railflow,runtime=pythonblazetestsessions--includeruns--tagsname=railflow,runtime=javascriptCLI ExamplesRunning tests using license key, logs disabled, sending them to Loki, tags specifiedblazetestrun-kABCDE-12345-FGHIJ-67890-akAWS-ACCESS-KEY-asAWS-SECRET-ACCESS-KEY-lo=disabled-lkLOKI-API-KEY-ttag1=blazetest,tag2=railflowRunning -d option to test if everything worksblazetestrun-l/home/user/ActivationFile20201020.skm-akAWS-ACCESS-KEY-asAWS-SECRET-ACCESS-KEY-dRunning with debugging enabledblazetest-kABCDE-12345-FGHIJ-67890-akAWS-ACCESS-KEY-asAWS-SECRET-ACCESS-KEY-de