Package: pytest-history
pytest-historyPytest plugin to keep history of your pytest runsOverviewpytest-historyenables the tracking of test statuses and other metadata across multiple test runs, providing
additional insights into test behavior.Initially, this plugin was developed specifically to identify potentially flaky tests (approximately 200) within a test
suite containing over 1000 tests, where various tests exhibited inconsistent behavior by failing on alternate runs.PurposeTracking Test History:Capturing and storing historical test results, encompassing pass, fail, and other pertinent
metadata.Identifying Flaky Tests:Enabling the identification of flaky tests by scrutinizing historical data, detecting
irregularities or recurring patterns in test outcomes.Facilitating Debugging:Offering developers and testers insights into test stability, thereby assisting in
debugging efforts and enhancing overall test reliability.UsageInstall the plugin usingpip install pytest-history.Utilize the historical data stored in the.test-results.dbSQLite database using either thepytest-historyCLI or an SQLite client.Configuration Options1. INI File SettingUsers can configure the database file location by adding the following setting in their INI file:# pytest.ini[pytest]pytest-history=/path/to/history.db# pyproject.toml[tool.pytest.ini_options]pytest-history="history.db"2. Environment VariableAnother option to configure the database file location is through an environment variable:Setting the environment variable in a Unix-based system:exportPYTEST_HISTORY_DB=/path/to/history.dbSetting the environment variable in a Windows system:set PYTEST_HISTORY_DB=C:\path\to\history.db3. Command-line ParameterUsers can also specify the database file location via a command-line parameter:pytest--history-db/path/to/history.dbtests/The database file path specified should be writable by the user running pytest commands.
When multiple configuration methods are used simultaneously, the command-line parameter takes precedence over the environment variable, and both take precedence over the INI file setting.CLIThepytest-historycommand utilizes a SQLite database (default:.test-results.db) to provide analysis and information about past test runs and their results. It offers two main subcommands:listandflakes.Subcommandslistpytest-history listoffers insights into historical data and provides two additional subcommands:resultsandruns.subcommandresultspytest-history list results <id>: Lists historic test results for a specific test run identified by its ID.subcommandrunspytest-history list runs: Lists historic test runs.flakespytest-history flakes: Lists all flaky tests identified in the test suite.Usagepytest-history[options]<subcommand>[suboptions]options-h,--help:Showthehelpmessageandexit.--dbDB:Specifythedatabasetobeusedforanalyzingdata(default:.test-results.db).ExamplesList historic test runspytest-historylistrunsList historic test results for a specific runpytest-historylistresults<id>List all flaky testspytest-historyflakesExample QueriesExample: To find flaky tests between two distinct test runs, execute the following SQL query:SELECTt1.testcase,t1.test_run,t2.test_run,t1.outcome,t2.outcomeFROM"test.results"t1JOIN"test.results"t2ont1.testcase=t2.testcaseAND(t1.test_run=1ANDt2.test_run=2)WHERE(t1.outcome='passed'ANDt2.outcome='failed')OR(t1.outcome='failed'ANDt2.outcome='passed')GROUPBYt1.testcaseORDERBYt1.testcase;