Package: behavex
BehaveXBehaveX is a test wrapper on top of Python Behave that provides additional capabilities to improve testing pipelines.BehaveX can be used to build testing pipelines from scratch using the sameBehaveframework principles, or to expand the capabilities of Behave-based projects.Features provided by BehaveXPerform parallel test executionsExecute tests using multiple processes, either by feature or by scenario.Get additional test execution reportsGenerate friendly HTML reports and JSON reports that can be exported and integrated with third-party toolsProvide additional evidence as part of execution reportsInclude any testing evidence by pasting it to a predefined folder path associated with each scenario. This evidence will then be automatically included as part of the HTML reportGenerate test logs per scenarioAny logs generated during test execution using the logging library will automatically be compiled into an individual log report for each scenarioMute test scenarios in build serversBy just adding the @MUTE tag to test scenarios, they will be executed, but they will not be part of the JUnit reportsGenerate metrics in HTML report for the executed test suiteAutomation Rate, Pass Rate and Steps executions & durationExecute dry runs and see the full list of scenarios into the HTML reportThis is enhanced implementation of Behave's dry run feature, allowing you to see the full list of scenarios in the HTML report without actually executing the testsRe-execute failing test scenariosBy just adding the @AUTORETRY tag to test scenarios, so when the first execution fails the scenario is immediately re-executedAdditionally, you can provide the wrapper with a list of previously failing scenarios, which will also be re-executed automaticallyInstalling BehaveXExecute the following command to install BehaveX with pip:pip install behavexExecuting BehaveXThe execution is performed in the same way as you do when executing Behave from command line, but using the "behavex" command.Examples:Run scenarios tagged asTAG_1but notTAG_2:behavex -t @TAG_1 -t ~@TAG_2Run scenarios tagged asTAG_1orTAG_2:behavex -t @TAG_1,@TAG_2Run scenarios tagged asTAG_1, using4parallel processes:behavex -t @TAG_1 --parallel-processes 4 --parallel-scheme scenarioRun scenarios located at "features/features_folder_1" and "features/features_folder_2" folders, using2parallel processesbehavex features/features_folder_1 features/features_folder_2 --parallel-processes 2Run scenarios from "features_folder_1/sample_feature.feature" feature file, using2parallel processesbehavex features_folder_1/sample_feature.feature --parallel-processes 2Run scenarios tagged asTAG_1from "features_folder_1/sample_feature.feature" feature file, using2parallel processesbehavex features_folder_1/sample_feature.feature -t @TAG_1 --parallel-processes 2Run scenarios located at "features/feature_1" and "features/feature_2" folders, using2parallel processesbehavex features/feature_1 features/feature_2 --parallel-processes 2Run scenarios tagged asTAG_1, using5parallel processes executing a feature on each process:behavex -t @TAG_1 --parallel-processes 5 --parallel-scheme featurePerform a dry run of the scenarios tagged asTAG_1, and generate the HTML report:behavex -t @TAG_1 --dry-runRun scenarios tagged asTAG_1, generating the execution evidence into the "exec_evidence" folder (instead of the default "output" folder):behavex -t @TAG_1 -o execution_evidenceConstraintsBehaveX is currently implemented on top of Behavev1.2.6, and not all Behave arguments are yet supported.The parallel execution implementation is based on concurrent Behave processes. Therefore, any code in thebefore_allandafter_allhooks in theenvironment.pymodule will be executed in each parallel process. The same applies to thebefore_featureandafter_featurehooks when the parallel execution is set by scenario.Additional CommentsThe JUnit reports have been replaced by the ones generated by the test wrapper, just to support muting tests scenarios on build serversSupported Behave argumentsno_colorcolordefineexcludeincludeno_snippetsno_capturenamecaptureno_capture_stderrcapture_stderrno_logcapturelogcapturelogging_levelsummaryquietstoptagstags-helpIMPORTANT: It worth to mention that some arguments do not apply when executing tests with more than one parallel process, such asstop,color, etc.Also, there might be more arguments that can be supported, it is just a matter of extending the wrapper implementation to use these.Specific arguments from BehaveXoutput-folder(-o or --output-folder)Specifies the output folder where execution reports will be generated (JUnit, HTML and JSon)dry-run(-d or --dry-run)Overwrites the existing Behave dry-run implementationPerforms a dry-run by listing the scenarios as part of the output reportsparallel-processes(--parallel-processes)Specifies the number of parallel Behave processesparallel-scheme(--parallel-scheme)Performs the parallel test execution by [scenario|feature]You can take a look at the provided examples (above in this documentation) to see how to use these arguments.Parallel test executionsThe implementation for running tests in parallel is based on concurrent executions of Behave instances in multiple processes.As mentioned as part of the wrapper constraints, this approach implies that whatever you have in the Python Behave hooks inenvironment.pymodule, it will be re-executed on every parallel process.BehaveX will be in charge of managing each parallel process, and consolidate all the information into the execution reportsParallel test executions can be performed byfeatureor byscenario.Examples:behavex --parallel-processes 3behavex -t @<TAG> --parallel-processes 3behavex -t @<TAG> --parallel-processes 2 --parallel-scheme scenariobehavex -t @<TAG> --parallel-processes 5 --parallel-scheme featureWhen the parallel-scheme is set byfeature, all tests within each feature will be run sequentially.Test execution reportsHTML reportThis is a friendly test execution report that contains information related to test scenarios, execution status, execution evidence and metrics. A filters bar is also provided to filter scenarios by name, tag or status.It should be available by default at the following path:<output_folder>/report.htmlJSON reportContains information about test scenarios and execution status.It should be available by default at the following path:<output_folder>/report.jsonThe report is provided to simplify the integration with third party tools, by providing all test execution data in a format that can be easily parsed.JUnit reportThe wrapper overwrites the existing Behave JUnit reports, just to enable dealing with parallel executions and muted test scenariosBy default, there will be one JUnit file per feature, unless the parallel execution is performed by scenario, in which there will be one JUnit file per scenario.Reports are available by default at the following path:<output_folder>/behave/*.xmlAttaching additional execution evidence to test reportIt is considered a good practice to provide as much as evidence as possible in test executions reports to properly identify the root cause of issues.Any evidence file you generate when executing a test scenario, it can be stored into a folder path that the wrapper provides for each scenario.The evidence folder path is automatically generated and stored into the"context.evidence_path"context variable. This variable is automatically updated by the wrapper before executing each scenario, and all the files you copy into that path will be accessible from the HTML report linked to the executed scenarioTest logs per scenarioThe HTML report provides test execution logs per scenario. Everything that is being logged using thelogginglibrary will be written into a test execution log file linked to the test scenario.MetricsAutomation RatePass RateSteps execution counter and average execution timeAll metrics are provided as part of the HTML reportDry runsThe wrapper overwrites the exiting Behave dry run implementation just to be able to provide the outputs into the wrapper reports.The HTML report generated as part of the dry run can be used to share the scenarios specifications with any stakeholder.Example:behavex -t @TAG --dry-runMuting test scenariosSometimes it is necessary that failing test scenarios continue being executed in all build server plans, but having them muted until the test or product fix is provided.Tests can be muted by adding the @MUTE tag to each test scenario. This will cause the wrapper to run the test but the execution will not be notified in the JUnit reports. However, you will see the execution information in the HTML report.What to do with failing scenarios?@AUTORETRY tagThis tag can be used for flaky scenarios or when the testing infrastructure is not stable at all.The @AUTORETRY tag can be applied to any scenario or feature, and it is used to automatically re-execute the test scenario when it fails.Rerun all failed scenariosWhenever you perform an automated test execution and there are failing scenarios, thefailing_scenarios.txtfile will be created into the execution output folder.
This file allows you to run all failing scenarios again.This can be done by executing the following command:behavex -rf ./<OUTPUT_FOLDER>/failing_scenarios.txtorbehavex --rerun-failures ./<OUTPUT_FOLDER>/failing_scenarios.txtTo avoid the re-execution to overwrite the previous test report, we suggest to provide a different output folder, using the-oor--output-folderargument.It is important to mention that this argument doesn't work yet with parallel test executionsShow Your SupportIf you find this project helpful or interesting, we would appreciate it if you could give it a star(:star:). It's a simple way to show your support and let us know that you find value in our work.By starring this repository, you help us gain visibility among other developers and contributors. It also serves as motivation for us to continue improving and maintaining this project.Thank you in advance for your support! We truly appreciate it.