Package: pytest-play
pytest-playpytest-playis a codeless, generic, pluggable and extensibleautomation tool,
not necessarilytest automationonly, based on the fantasticpytesttest framework
that let you define and executeYAMLfiles containing scripts or test scenarios
through actions and assertions that can be implemented and managed even bynon technical users:automation (not necessarily test automation). You can build a set of actions on a single file (e.g,
call a JSON based API endpoint, perform an action if a condition matches) or a test automation
project with many test scenarios.For example you can create always fresh test data on demand supporting
manual testing activities, build a live simulator and so oncodeless, or better almost codeless. If you have to write assertions against action results or some
conditional expressions you need a very basic knowledge of Python or Javascript expressions
with a smooth learning curve (something likevariables['foo']== 'bar')generic. It is not yet again another automation tool for browser automation only, API only, etc.
You can drive a browser, perform some API calls, make database queries and/or make assertions
using the same tool for different technologiesSo there are several free or not free testing frameworks or automation tools and many times
they address just one single area testing needs and they are not extensible: API testing only,
UI testing only and so on. It could be fine if you are testing a web
only application like a CMS but if you are dealing with a reactive IoT application you might something more,
make cross actions or cross checks against different systems or build something of more complex uponpytest-playpowerful. It is not yet again another test automation tool, it only extends thepytestframework
with another paradigm and inherits a lot of good stuff (test data decoupled by test implementation
that let you write once and executed many times the same scenario thanks to native parametrization
support, reporting, integration with test management tools, many useful command line options, browsers and
remote Selenium grids integration, etc)pluggable and extensible. Let’s say you need to interact with a system not yet supported by apytest-playplugin, you can write by your own or pay someone for you. In addition there is a scaffolding tool that
let you implement your own command:https://github.com/davidemoro/cookiecutter-play-plugineasy to use. Why YAML? Easy to read, easy to write, simple and standard syntax, easy to be validated and
no parentheses hell. Despite there are no recording tools (not yet) for browser interaction or API calls, the
documentation based on very common patterns let you copy, paste and edit command by command with no painfree software. It’s an open source project based on the large and friendlypytestcommunityeasy to install. The only prerequisite is Docker thanks to thedavidemoro/pytest-playDocker Hub container.
Or better, with docker, no installation is required: you just need to type the following commanddocker run-i--rm-v$(pwd):/srcdavidemoro/pytest-playinside your project folder
Seehttps://hub.docker.com/r/davidemoro/pytest-playSee at the bottom of the page the third party plugins that extendspytest-play:Third party pytest-play pluginsHow it worksDepending on your needs and skills you can choose to use pytest-play programmatically
writing some Python code or following a Python-less approach.As said before withpytest-playyou will be able to create codeless scripts or test scenarios
with no or very little Python knowledge: a filetest_XXX.yml(e.g.,test_something.yml,
wheretest_and.ymlmatter) will be automatically recognized and executed without having
to touch any*.pymodule.You can run a single scenario withpytest test_XXX.ymlor running the entire suite filtering
by name or keyword markers.Despitepytest-playwas born with native support for JSON format,pytest-play>=2.0 versions will support
YAML only for improved usability.Python-less (pure YAML)Here you can see the contents of apytest-playproject without any Python files inside
containing a login scenario:$ tree
.
├── env-ALPHA.yml    (OPTIONAL)
└── test_login.ymland you might have some global variables in a settings file specific for a target environment:$ cat env-ALPHA.yml
pytest-play:
  base_url: https://www.yoursite.comThe test scenario with action, assertions and optional metadata
(play_seleniumexternal plugin needed):$ cat test_login.yml
---
markers:
  - login
test_data:
  - username: siteadmin
    password: siteadmin
  - username: editor
    password: editor
  - username: reader
    password: reader
---
- comment: visit base url
  type: get
  provider: selenium
  url: "$base_url"
- comment: click on login link
  locator:
    type: id
    value: personaltools-login
  type: clickElement
  provider: selenium
- comment: provide a username
  locator:
    type: id
    value: __ac_name
  text: "$username"
  type: setElementText
  provider: selenium
- comment: provide a password
  locator:
    type: id
    value: __ac_password
  text: "$password"
  type: setElementText
  provider: selenium
- comment: click on login submit button
  locator:
    type: css
    value: ".pattern-modal-buttons > input[name=submit]"
  type: clickElement
  provider: selenium
- comment: wait for page loaded
  locator:
    type: css
    value: ".icon-user"
  type: waitForElementVisible
  provider: seleniumThe first optional YAML document contains some metadata with keywords akamarkersso you can filter tests to be executed invoking pytest with marker expressions,
decoupled test data, etc.The sametest_login.ymlscenario will be executed 3 times with different
decoupled test datatest_datadefined inside its first optional YAML
document (the block between the 2---lines).So write once and execute many times with different test data!You can see a hello world example here:https://github.com/davidemoro/pytest-play-plone-exampleAs told before the metadata document is optional so you might have 1 or 2
documents in your YAML file. You can find more info aboutMetadata format.Here you can see the same example without the metadata section for sake of
completeness:---
- comment: visit base url
  type: get
  provider: selenium
  url: "http://YOURSITE"
- comment: click on login link
  type: clickElement
  provider: selenium
  locator:
    type: id
    value: personaltools-login
- comment: provide a username
  type: setElementText
  provider: selenium
  locator:
    type: id
    value: __ac_name
  text: "YOURUSERNAME"
- comment: provide a password
  type: setElementText
  provider: selenium
  locator:
    type: id
    value: __ac_password
  text: "YOURPASSWORD"
- comment: click on login submit button
  type: clickElement
  provider: selenium
  locator:
    type: css
    value: ".pattern-modal-buttons > input[name=submit]"
- comment: wait for page loaded
  type: waitForElementVisible
  provider: selenium
  locator:
    type: css
    value: ".icon-user"ProgrammaticallyYou can invoke pytest-play programmatically too.You can define a testtest_login.pylike this:def test_login(play):
    data = play.get_file_contents(
        'my', 'path', 'etc', 'login.yml')
    play.execute_raw(data, extra_variables={})Or this programmatical approach might be used if you are
implementing BDD based tests usingpytest-bdd.Core commandspytest-playprovides some core commands that let you:write simple Python assertions, expressions and variablesreuse steps including other test scenario scriptsprovide a default command template for some particular providers
(eg: add by default HTTP authentication headers for all requests)a generic wait until machinery. Useful for waiting for an
observable asynchronous event will complete its flow before
proceeding with the following commands that depends on the previous
step completionYou can write restricted Python expressions and assertions based on theRestrictedPythonpackage.RestrictedPythonis a tool that helps to define a subset of the Python
language which allows to provide a program input into a trusted environment.
RestrictedPython is not a sandbox system or a secured environment, but it helps
to define a trusted environment and execute untrusted code inside of it.See:https://github.com/zopefoundation/RestrictedPythonHow to reuse stepsYou can split your commands and reuse them using theincludecommand avoiding
duplication:- provider: include
  type: include
  path: "/some-path/included-scenario.yml"You can create a variable for the base folder where your test scripts live.Default commandsSome commands require many verbose options you don’t want to repeat (eg: authentication headers forplay_requests).Instead of replicating all the headers information you can initialize apytest-playwith the provider name as
key and as a value the default command you want to omit (this example neets the external pluginplay_selenium):- provider: python
  type: store_variable
  name: bearer
  expression: "'BEARER'"
- provider: python
  type: store_variable
  name: play_requests
  expression: "{'parameters': {'headers': {'Authorization': '$bearer'}}}"
- provider: play_requests
  type: GET
  comment: this is an authenticated request!
  url: "$base_url"Store variablesYou can store apytest-playvariables:- provider: python
  type: store_variable
  expression: "1+1"
  name: fooMake a Python assertionYou can make an assertion based on a Python expression:- provider: python
  type: assert
  expression: variables['foo'] == 2SleepSleep for a given amount of seconds:- provider: python
  type: sleep
  seconds: 2Exec a Python expresssionYou can execute a Python expression:- provider: python
  type: exec
  expression: "1+1"While condition and loopingIf you need to loop over a series of commands or wait something you can use
thewhilecommand. It will execute the sequence of sub commands, if any,
while the resulting expression condition is true. Assuming you have acountdownvariable containing a integer10, the block of commands whill be executed 10 times:---
- provider: python
  type: while
  expression: variables['countdown'] >= 0
  timeout: 2.3
  poll: 0.1
  sub_commands:
  - provider: python
    type: store_variable
    name: countdown
    expression: variables['countdown'] - 1Thewhilecommand supersedes the other legacy commandswait_until(stops when the condition becomes true) orwait_until_not.
commands.Conditional commands (Python)You can skip any command evaluating a Python based skip condition
like the following:- provider: include
  type: include
  path: "/some-path/assertions.yml"
  skip_condition: variables['cassandra_assertions'] is TrueHow to assert commands elapsed timeThe engine updates apytest-playvariable called_elapsedfor each executed command. So you can write something that:---
- type: GET
  provider: play_requests
  url: https://api.chucknorris.io/jokes/categories
  expression: "'dev' in response.json()"
- type: assert
  provider: python
  expression: "variables['_elapsed'] > 0"Generate a JUnit XML reportUse the--junit-xmlcommand line option, e.g.:--junit-xml results.xmlYou’ll get for each test case errors, commands executed insystem-output(do not use-sor--capture=nootherwise you won’t
see commands insystem-output) and execution timing metrics (global, per test case and per single command thanks to_elapsedproperty tracked on every executed command shown insystem-output).Here you can see a standardresults.xmlfile:<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.360"><testcase classname="test_assertion.yml" file="test_assertion.yml" name="test_assertion.yml" time="0.326"><system-out>{&apos;expression&apos;: &apos;1 == 1&apos;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0003077983856201172}
{&apos;expression&apos;: &apos;0 == 0&apos;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0002529621124267578}
</system-out></testcase></testsuite>Generate a custom JUnit XML report with custom properties and execution times metricsYou can track execution time metrics for monitoring and measure
what is important to you. For example you can track using a machine interpretable format:response times (e.g., how much time is needed for returning aPOSTjson payload)time that occurs between the invocation of an API and a reactive web application update or some asynchronous data appearing on an event storetime that occurs between a user input on browser and results updated (e.g., a live search)time that occurs between a login button and the page loaded an usable (e.g., how much time is needed after a browser action to click on a target button)Track response time metric in JUnit XML reportFor example, atest_categories.ymlfile executed with
the command line option--junit-xmlreport.xml(requiresplay_requestsplugin):test_data:
  - category: dev
  - category: movie
  - category: food
---
- type: GET
  provider: play_requests
  url: https://api.chucknorris.io/jokes/categories
  expression: "'$category' in response.json()"
- provider: metrics
  type: record_elapsed
  name: categories_time
- type: assert
  provider: python
  expression: "variables['categories_time'] < 2.5"
  comment: you can make an assertion against the categories_timewill generate an extendedreport.xmlfile with custom properties like that:<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="3" time="2.031"><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml0" time="0.968"><properties><property name="categories_time" value="0.5829994678497314"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;dev&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.5829994678497314}
{&apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed&apos;, &apos;_elapsed&apos;: 3.3855438232421875e-05}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2.5&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0006382465362548828}
</system-out></testcase><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml1" time="0.481"><properties><property name="categories_time" value="0.4184422492980957"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;movie&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.4184422492980957}
{&apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed&apos;, &apos;_elapsed&apos;: 2.09808349609375e-05}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2.5&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.000553131103515625}
</system-out></testcase><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml2" time="0.534"><properties><property name="categories_time" value="0.463592529296875"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;food&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.463592529296875}
{&apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed&apos;, &apos;_elapsed&apos;: 2.09808349609375e-05}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2.5&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.00054931640625}
</system-out></testcase></testsuite>and the custom propertycategories_timewill be tracked for each
test case execution, for example:<properties>
    <property name="categories_time" value="0.5829994678497314"/>
</properties>Advanced metrics in JUnit XML reportIn this example we want to measures how long it takes a page to become interactive
(page responding to user interactions) and evaluate update time for a live search feature.
Let’s see thetest_search.ymlexample (requiresplay_selenium):---
- provider: selenium
  type: get
  url: https://www.plone-demo.info/
- provider: metrics
  type: record_elapsed_start
  name: load_time
- provider: selenium
  type: setElementText
  text: plone 5
  locator:
    type: id
    value: searchGadget
- provider: metrics
  type: record_elapsed_stop
  name: load_time
- provider: metrics
  type: record_elapsed_start
  name: live_search_time
- provider: selenium
  type: waitForElementVisible
  locator:
    type: css
    value: li[data-url$="https://www.plone-demo.info/front-page"]
- provider: metrics
  type: record_elapsed_stop
  name: live_search_timeIf you execute this scenario with the--junit-xmlresults.xmloption you’ll get aresults.xmlfile similar to this one:<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="13.650"><testcase classname="test_search.yml" file="test_search.yml" name="test_search.yml" time="13.580"><properties><property name="load_time" value="1.1175920963287354"/><property name="live_search_time" value="1.0871295928955078"/></properties><system-out>{&apos;provider&apos;: &apos;selenium&apos;, &apos;type&apos;: &apos;get&apos;, &apos;url&apos;: &apos;https://www.plone-demo.info/&apos;, &apos;_elapsed&apos;: 9.593282461166382}
{&apos;name&apos;: &apos;load_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed_start&apos;, &apos;_elapsed&apos;: 1.1682510375976562e-05}
{&apos;locator&apos;: {&apos;type&apos;: &apos;id&apos;, &apos;value&apos;: &apos;searchGadget&apos;}, &apos;provider&apos;: &apos;selenium&apos;, &apos;text&apos;: &apos;plone 5&apos;, &apos;type&apos;: &apos;setElementText&apos;, &apos;_elapsed&apos;: 1.1019845008850098}
{&apos;name&apos;: &apos;load_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed_stop&apos;, &apos;_elapsed&apos;: 1.9788742065429688e-05}
{&apos;name&apos;: &apos;live_search_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed_start&apos;, &apos;_elapsed&apos;: 1.0013580322265625e-05}
{&apos;locator&apos;: {&apos;type&apos;: &apos;css&apos;, &apos;value&apos;: &apos;li[data-url$=&quot;https://www.plone-demo.info/front-page&quot;]&apos;}, &apos;provider&apos;: &apos;selenium&apos;, &apos;type&apos;: &apos;waitForElementVisible&apos;, &apos;_elapsed&apos;: 1.060795545578003}
{&apos;name&apos;: &apos;live_search_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_elapsed_stop&apos;, &apos;_elapsed&apos;: 2.3603439331054688e-05}
</system-out></testcase></testsuite>and in this case you’ll find out that the key metricload_timewas1.11seconds and thelive_search_timewas1.09seconds as
you can see here:<properties>
    <property name="load_time" value="1.1175920963287354"/>
    <property name="live_search_time" value="1.0871295928955078"/>
</properties>So thanks to JUnit XML reporting you can track response times (not only browser based timings)
using a machine readable format to be ingested by third party systems with an acceptable approximation
if you cannot track timings directly on the systems under test.Track any property in JUnit XML reports using expressionsLet’s see atest_categories.yml(play_seleniumrequired):test_data:
  - category: dev
  - category: movie
  - category: food
---
- type: GET
  provider: play_requests
  url: https://api.chucknorris.io/jokes/categories
  expression: "'$category' in response.json()"
- provider: metrics
  type: record_property
  name: categories_time
  expression: "variables['_elapsed']*1000"
- type: assert
  provider: python
  expression: "variables['categories_time'] < 2500"
  comment: you can make an assertion against the categories_timegenerates some custom properties (categories_timein milliseconds using a python expression)
using the--junit-xmlresults.xmlcli option:<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skipped="0" tests="3" time="2.312"><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml0" time="1.034"><properties><property name="categories_time" value="610.3124618530273"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;dev&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.6103124618530273}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;exec&apos;, &apos;_elapsed&apos;: 0.0006859302520751953}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_property&apos;, &apos;_elapsed&apos;: 0.006484270095825195}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2500&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0005526542663574219}
</system-out></testcase><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml1" time="0.550"><properties><property name="categories_time" value="443.72105598449707"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;movie&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.44372105598449707}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;exec&apos;, &apos;_elapsed&apos;: 0.0009415149688720703}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_property&apos;, &apos;_elapsed&apos;: 0.01613616943359375}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2500&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0011241436004638672}
</system-out></testcase><testcase classname="test_categories.yml" file="test_categories.yml" name="test_categories.yml2" time="0.676"><properties><property name="categories_time" value="576.5485763549805"/></properties><system-out>{&apos;expression&apos;: &quot;&apos;food&apos; in response.json()&quot;, &apos;provider&apos;: &apos;play_requests&apos;, &apos;type&apos;: &apos;GET&apos;, &apos;url&apos;: &apos;https://api.chucknorris.io/jokes/categories&apos;, &apos;_elapsed&apos;: 0.5765485763549805}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;exec&apos;, &apos;_elapsed&apos;: 0.0006375312805175781}
{&apos;expression&apos;: &quot;variables[&apos;_elapsed&apos;]*1000&quot;, &apos;name&apos;: &apos;categories_time&apos;, &apos;provider&apos;: &apos;metrics&apos;, &apos;type&apos;: &apos;record_property&apos;, &apos;_elapsed&apos;: 0.006584644317626953}
{&apos;comment&apos;: &apos;you can make an assertion against the categories_time&apos;, &apos;expression&apos;: &quot;variables[&apos;categories_time&apos;] &lt; 2500&quot;, &apos;provider&apos;: &apos;python&apos;, &apos;type&apos;: &apos;assert&apos;, &apos;_elapsed&apos;: 0.0005452632904052734}
</system-out></testcase></testsuite>obtaining the metrics you want to track for each execution, for example:<properties><property name="categories_time" value="610.3124618530273"/></properties>so you might track the category as well for each test execution
or whatever you want.Monitoring test metrics with statsd/graphiteIf you like the measure everything approach you can track and monitor interesting
custom test metrics from an end user perspective during normal test executions or
heavy load/stress tests thanks to thestatsd/graphiteintegration.Measuring important key metrics is important for many reasons:compare performance between different versions under same conditions using past
tracked stats for the same metric (no more say the systemseems slowertoday)predict the system behaviour with many items on frontend (e.g., evaluate
the browser dealing with thousands and thousands of items managed by an infinite
scroll plugin)predict the system behaviour under loadYou can installstatsd/graphitein minutes using Docker:https://graphite.readthedocs.io/en/latest/install.htmlBasically you can track onstatsd/graphiteeverynumericmetric using
the same commands used for tracking metrics on JUnit XML reports as we will see.In addition, but not required, installing the third party plugin calledpytest-statsd.
you can track onstatsd/graphite:execution timesnumber of executed tests per status (pass, fail, error, etc)Prerequisites (you need to install the optional statsd client not installed by
default)::pip install pytest-play[statsd]Usage (cli options compatible withpytest-statsd):--stats-d [--stats-prefix play --stats-host http://myserver.com --stats-port 3000]where:--stats-d, enablestatsd--stats-prefix(optional), if you plan on having multiple projects sending
results to the same server.
For example if you provideplayas prefix you’ll get a time metric under
thestats.timers.play.YOURMETRIC.meankey (or instead of.meanyou can use.upper,upper_90, etc)--stats-host, by defaultlocalhost--stats-port, by default8125Now you can track timing metrics using therecord_elapsedorrecord_elapsed_start/record_elapsed_stopcommands seen before (pytest-play will
send for you time values tostatsdconverted tomillisecondsas requested bystatsd).If you want to track custom metrics using therecord_propertycommand you have to provide
an additional parameter calledmetric_type. For example:- provider: metrics
  type: record_property
  name: categories_time
  expression: "variables['_elapsed']*1000"
  metric_type: timing
- provider: metrics
  type: record_property
  name: fridge_temperature
  expression: "4"
  metric_type: gaugeSome additional information regarding therecord_propertycommand:if you don’t provide themetric_typeoption inrecord_propertycommands values
will not be transmitted tostatsd(eventually they will be tracked on JUnit XML report
if--junit-xmloption was provided)if you provide an allowedmetric_typevalue (timingorgauge) non numeric values
will be considered as an error (ValueErrorexception raised)non allowedmetric_typevalues will be considered as an errorif you providetimingasmetric_type, it’s up to you providing a numeric value
expressed inmillisecondsMonitor HTTP response timesMonitor API response time (seehttps://github.com/pytest-dev/pytest-play/tree/features/examples/statsd_graphite_monitoring):Browser metricsMonitor browser metrics using Selenium from an end user perspective (seehttps://github.com/pytest-dev/pytest-play/tree/features/examples/statsd_graphite_monitoring_selenium):from page load to page usablelive search responsivenessRecord metrics programmaticallyIf you don’t want to usepytest-playbut you need to record test metrics
you can usepytest-playas a library::def test_programmatically(play):
    play.execute_command({
        'provider': 'metrics',
        'type': 'record_property',
        'name': 'oil_temperature',
        'expression': '60',
        'metric_type': 'gauge'})Performance tests with pytest-play and bzt/Taurus (BlazeMeter)You can reuse all your pytest-play scenario and turn them to
performance tests using bzt/Taurus (so it is compatible withBlazeMetertoo and all its goodies).Add a bzt/Taurus YAML file with notest_prefix like that (full example here inbzt_performance):settings:
  artifacts-dir: /tmp/%Y-%m-%d_%H-%M-%S.%f

execution:
- executor: pytest
  scenario: pytest-run
  iterations: 1

scenarios:
  pytest-run:
    # additional-args: --stats-d --stats-prefix play
    script: scripts/

services:
- module: shellexec
  prepare:
  - pip3 install -r https://raw.githubusercontent.com/davidemoro/pytest-play-docker/master/requirements.txtand run the following command:docker run --rm -it -v $(pwd):/src --user root --entrypoint "bzt" davidemoro/pytest-play bzt.ymlYou will see bzt up and running playing our scenarios:You can uncommentadditional-argsto pass otherpytestcommand line options (e.g., enablestatsdfor key user metrics monitoring or any other cli option).More info about bzt/Taurus here:http://gettaurus.org/Dynamic expressions in payloads without declaring variablesIf you have to send a certain payload to a REST endpoint or a MQTT message
containing a dynamic value you can store a variable withstore_variableand use$variable_namein your payload when needed.
Storing variables is cool if you will reuse later that value but if just have to
generate a dynamic value, let’s say a timestamp in milliseconds,
you can use the{! EXPRESSION !}format.For example (play_mqttplugin required):---
- comment: python expressions in mqtt payload (without declaring variables)
  provider: mqtt
  type: publish
  host: "$mqtt_host"
  port: "$mqtt_port"
  endpoint: "$mqtt_endpoint/$device_serial_number"
  payload: '{
        "measure_id":   [124],
        "obj_id_L":     [0],
        "measureType":  ["float"],
        "start_time":   {! int(datetime.datetime.utcnow().timestamp()*1000) !},
        "bin_value":    [1]
    }'where instead of the expression:{! int(datetime.datetime.utcnow().timestamp()*1000) !},will be printed:1553007973702Browser based commandsThepytest-playcore no more includes browser based commands. Moved toplay_seleniumexternal plugin.pytest-play is pluggable and extensiblepytest-playhas a pluggable architecture and you can extend it.For example you might want to support your own commands, support non UI
commands like making raw POST/GET/etc calls, simulate IoT devices
activities, provide easy interaction with complex UI widgets like
calendar widgets, send commands to a device using the serial port implementing
a binary protocol and so on.How to register a new command providerLet’s suppose you want to extend pytest-play with the following command:command = {'type': 'print', 'provider': 'newprovider', 'message': 'Hello, World!'}You just have to implement a command provider:from pytest_play.providers import BaseProvider

class NewProvider(BaseProvider):

    def this_is_not_a_command(self):
        """ Commands should be command_ prefixed """

    def command_print(self, command):
        print(command['message'])

    def command_yetAnotherCommand(self, command):
        print(command)and register your new provider in yoursetup.pyadding an entrypoint:entry_points={
    'playcommands': [
        'print = your_package.providers:NewProvider',
    ],
},You can define new providers also for non UI commands. For example publish MQTT
messages simulating IoT device activities for integration tests.If you want you can generate a new command provider thanks to:https://github.com/davidemoro/cookiecutter-play-pluginMetadata formatYou can also add some scenario metadata placing another YAML document on top of the scenario
defined on thetest_XXX.ymlwith the following format:---
markers:
  - marker1
  - marker2
test_data:
  - username: foo
  - username: bar
---
# omitted scenario steps in this example...Option details:markers, you can decorate your scenario with one or more markers. You can use them
in pytest command line for filtering scenarios to be executed thanks to marker
expressions like-m"marker1 and not slow"test_data, enables parametrization of your decoupletd test data and let you execute
the same scenario many times. For example
the example above will be executed twice (one time with “foo” username and another time
with “bar”)New options will be added in the next feature (e.g., skip scenarios, xfail, xpass, etc).Exampleshttps://github.com/pytest-dev/pytest-play/tree/master/exampleshttps://github.com/davidemoro/pytest-play-docker/tree/master/testshttps://github.com/davidemoro/pytest-play-plone-exampleArticles and talksArticles:Hello pytest-play!API/REST testing like Chuck Norris with pytest play using YAMLpytest-play automated docker hub publishing workflowTest automation framework thoughts and examples with Python, pytest and JenkinsTesting metrics thoughts and examples: how to turn lights on and off through MQTT with pytest-playTalks:Serena Martinetti @ Pycon9 - Florence: Integration tests ready to use with pytest-playDavide Moro @ STF2019 - Milan: Automazione e monitoraggio metriche di test in ambito IoT con pytest-playThird party pytest-play pluginsplay_selenium,pytest-playplugin driving browsers using Selenium/Splinter
under the hood. Selenium grid compatible and implicit auto wait actions
for more robust scenarios with less controls.play_requests,pytest-playplugin driving the famous Pythonrequestslibrary for makingHTTPcalls.play_sql,pytest-playsupport for SQL expressions and assertionsplay_cassandra,pytest-playsupport for Cassandra expressions and assertionsplay_dynamodb,pytest-playsupport for AWS DynamoDB queries and assertionsplay_websocket,pytest-playsupport for websocketsplay_mqtt,pytest-playplugin for MQTT support. Thanks toplay_mqttyou can test the integration between a mocked IoT device that sends
commands on MQTT and a reactive web application with UI checks.You can also build a simulator that generates messages for you.Feel free to add your own public plugins with a pull request!Twitterpytest-playtweets happens here:@davidemoroChangelog2.3.1 (2019-06-12)Bugfix:fix compatibility with pytest 4.6. Rif #86Documentation:update media section (articles and talks)2.3.0 (2019-04-05)Features and improvements:wait_untilandwait_until_notnow accept commands with nosub_commandspropertyimplement newwhilecommand in python provider (while expression is true)2.2.2 (2019-03-29)Minor changes:remove internal property parameter on engineBugfix:add compatibility withpytest-repeat’s--countcommand line optionDocumentation:mention how to generate dynamic values using{! expr !}expressions
(e.g., dynamic payloads in REST or MQTT without having to store variables
when not needed)2.2.1 (2019-03-19)Minor changes:addintandfloatbuiltins available in Python expressionsmake python expressions more flexible for future improvements (internal change that doesn’t
affect compatibility)Bugfix:fix--setup-planinvokationDocumentation:add more examples (bzt/Taurus and performance tests using pytest-play)2.2.0 (2019-03-01)statsdintegration (optional requirement) for advanced test metrics using statsd/graphite.
If you install pytest play with the optional statsd support withpytest-play[statsd]you will get the additional dependencystatsdclient and you can use the same cli
options defined by thepytest-statsdplugin (e.g.,--stats-d[--stats-prefixmyproject--stats-hosthttp://myserver.com--stats-port3000]).Note well: despite the above cli options are the same defined by thepytest-statsdplugin,
at this time of writingpytest-statsdis not apytest-playdependency
so you won’t get stats about number of failures, passing, etc but only stats tracked bypytest-play. If you need them you can installpytest-statsd(it plays well withpytest-play)2.1.0 (2019-02-22)Features:support junit xml generation file withsystem-outelement for
each test case execution (pytest--junit-xmloption).system-outwill tracked by default in junit report unless you use
the--capture=noor its alias-strack_elapsedtime for each executed command--junit-xmlreport
ifsystem-outis enabledtrackpytestcustom properties in--junit-xmlreport for monitoring
and measure what is important to you. For example you can track as key metric
the time of the time occurred between the end of the previous action and
the completion of the following. Basically you can track under theproperty_nameload_loginkey the time occurred between the click on the submit button
and the end of the current command (e.g., click on the menu or text input
being able to receive text) using a machine interpretable format.Theproperty_namevalue elapsed time will be available as standardpytest-playvariable so that you can make additional assertionsafter every command execution apytest-playvariable will be added/updated
reporting the elapsed time (accessible usingvariables['_elapsed']).So be aware that the_elapsedvariable name should be considered as a special
variable and so you should not use this name for storing variablesimprove debug in case of failed assertions or errored commands. Logged variables
dump in standard logs andsystem-outreporting if availableimprove debuggability in case of assertion errors (log failing expression)added a newmetricsprovider that let you track custom metrics in conjunction
with--junit-xmloption. You can track in a machine readable format response
times, dynamic custom expressions, time that occurs between different commands
(e.g., measure the time needed after a login to interact with the page, time before
an asynchronous update happens and so on). Under themetricsprovider you’ll
find therecord_property,record_elapsed,record_elapsed_startandrecord_elapsed_stopcommandsDocumentation:minor documentation changesadd more examples2.0.2 (2019-02-06)Documentation:more examplesfix documentation bug on README (example based on selenium with missingprovider: selenium)2.0.1 (2019-01-30)Documentation:Mention davidemoro/pytest-play docker container in README.
You can use pytest-play with a docker command like that nowdocker run-i--rm-v$(pwd):/srcdavidemoro/pytest-playBugfix:Fix error locking pipenv due to pytest-play requirement
constraint not existing (RestrictedPython>=4.0.b2 -> RestrictedPython>=4.0b2)2.0.0 (2019-01-25)Breaking changes:Renamed fixture fromplay_jsontoplay(#5)Drop json support, adopt yaml only format for scenarios (#5)Drop.inifile for metadata, if you need them you can add
a YAML document on top of the scenario.ymlfile. You no more
need multiple files for decorating your scenarios now (#65)play.executeno more accepts raw data string), consumes a list of commands.
Introducedplay.execute_rawaccepting raw data string.play.execute_commandaccepts a Python dictionary only now (not a string)Selenium provider removed frompytest-playcore, implemented on a
separate packageplay_selenium. Starting from now you have to add
to your selenium commandsprovider: seleniumengine’sparametrizer_classattribute no more available (
useparametrizer.Parametrizerby default now)Bug fix:Fix invalid markup on PyPI (#55)Fix invalid escape sequences (#62).Documentation and trivial changes:Add examples folder1.4.2 (2018-05-17)Configuration change on Github. Use the same branching policy adopted by
pytest (master becomes main branch, see #56)Fixed skipped test and added new tests (deselect scenarios with keyword
and marker expressions)Fix #58: you no more get a TypeError if you try to launch pytest-play
in autodiscovery modeFix #55: restructured text lint on README.rst (bad visualization on pypi)Updated README (articles and talks links)Added aDeprecationWarningforplay_jsonfixture.
pytest-play will be based on yaml instead of json in version >=2.0.0.
Seehttps://github.com/pytest-dev/pytest-play/issues/51.4.1 (2018-04-06)Documentation improvementsAdd bzt/Taurus/BlazeMeter compatibility1.4.0 (2018-04-05)Small documentation improvementsNowtest_XXX.jsonfiles are automatically collected and executedYou can run a test scenario using the pytest CLIpytest test_YYY.jsonIntroduced json test scenario ini file with markers definition. For a giventest_YYY.jsonscenario you can add atest_YYY.iniini file:[pytest]
markers =
    marker1
    marker2and filter scenarios using marker expressionspytest-mmarker1Enabled parametrization of arguments for a plain json scenario in scenario ini file:[pytest]
test_data =
   {"username": "foo"}
   {"username": "bar"}and your json scenario will be executed twicepytest-playloads some variables based on the contents of the optionalpytest-playsection in yourpytest-variablesfile now. So if your variables file contains the following
values:pytest-play:
  foo: bar
  date_format: YYYYMMDDyou will be able to use expressions$foo,$date_format,variables['foo']orvariables['date_format']1.3.2 (2018-02-05)Addsortedin python expressions1.3.1 (2018-01-31)Add more testsDocumentation updateplay_json fixture no more assumes that you
have some pytest-variables settings.
No more mandatoryfix include scenario bug that occurs only
on Windows (slash vs backslash and
JSON decoding issues)1.3.0 (2018-01-22)documentation improvementssupports teardown callbacks1.2.0 (2018-01-22)implement python based commands inpytest-playand
deprecatesplay_python.
So this feature is a drop-in replacement for theplay-pythonplugin.You should no more installplay_pythonsince now.update documentationdeprecate selenium commands (they will be implemented
on a separate plugin and dropped inpytest-play>= 2.0.0). All your previous scripts
will work fine, this warning is just for people
directly importing the provider for some reason.implement skip conditions. You can omit the execution of
any command evaluating a Python based skip condition1.1.0 (2018-01-16)Documentation updated (add new pytest play plugins)Support default payloads for command providers. Useful
for HTTP authentication headers, common database settings1.0.0 (2018-01-10)execute command accepts kwargs nowexecute command returns the command value nowcomplete refactor ofincludeprovider (no
backwards compatibility)addplay_json.get_file_contentsand removeddata_getterfixture (no backwards compatibility)0.3.1 (2018-01-04)play engine now logs commands to be executed and errors0.3.0 (2018-01-04)you are able to update variables when executing commandsyou can extendpytest-playwith new pluggable commands coming
from third party packages thanks to setuptools entrypoints0.2.0 (2018-01-02)no more open browser by default
pytest-play is a generic test engine and it could be used for non UI tests too.So there is no need to open the browser for non UI tests (eg: API tests)0.1.0 (2017-12-22)implement reusable steps (include scenario)minor documentation changes0.0.1 (2017-12-20)First release