Package: pytest-orchestration
Pytest-OrchestrationPytest-Orchestration is configuration driven test orchestration plugin, designed with the intention of being used for performance tests, however it might fit other purposes as well.Installationpip install pytest-orchestrationConfiguration/DescriptionThe configuration describing the orchestrated tests has the form of a .json file.
Where these files live is defined in the pytest.ini under "orchestration_descriptions". Note that the name of the file should match thetest_namekey in the file as well.[pytest]
orchestration_descriptions = orchestration/descriptions/Below is an example config with comments:{
  "test_name": "my_orchestration_test",                 // Name of the test (no need to have a "test" prefix or suffix)
  "total_hours": 3,                                     // Total number of hours for the test to run
  "unref_setup_fixtures": ["device_with_collectd"],     // A list of setup fixtures that won't be references in actual test events
  "events": [                                           // A list of all "events" making up the test
    {
      "name": "start_playback",                         // Mandatory: The name of the event as it is implemented, see Events section below
      "interval_sec": 300,                              // Optional: The number of seconds between eache execution of this event. If it is not set, then the event will only be triggerd according to "at_startup" and "at_teardown"
      "at_startup": true,                               // Optional(Defaults to true): Specifies if the event should be executed once on startup, if false it will wait interval_sec befor being executed.
      "at_teardown": false,                              // Optional(Defaults to false): Specifies if a event should be executed on teardown, one last time after the time "total_hours" have been reached.
      "params":{                                        // Optional: Params is a dict of simple key: values that would be used in the implemented events
        "video_path": "/home/orch-tests/video-files/high_res_clip.mp4"
      }
    },
    {
      "name": "start_streamer",
      "at_startup": true
    },
    {
      "name": "collect_system_report",
      "at_startup": false,
      "at_teardown": true
    }
  ]
}EventsEvents are what make up the orchestrated test. From the configuration example above we have 3 event;start_playback,start_streamerandcollect_system_report. Each of these events needs to have an corresponding function. Where these functions live is specified in pytest.ini underorchestration_sourcesand can be a comma(',') separated list of files.[pytest]
orchestration_sources = tests/event_plugin.py, somethingelse.pyThese events implementations can use any existing pytest fixture as parameter, as well as any parameter defined in the configuration/description, see "video_path" in configuration example above.See an actual event implementation ofstart_playbackfrom configuration example above.
It's input parameters are:"video_player" - An already existing fixture we are reusing in our event"video_path" - An input defined from our configuration/description file"kill_switch" - The provided kill_switch fixture, we send it in you play method since it will start a subprocess which orchestration can not stop on it's own, so through the kill_switch we signal it and it has to take care of stopping itself."result_reporter" - We pass in the result_reporter so the play method can add its info/error whatever that orchestration then will monitor and execute on_result() when there is something new reporteddef start_playback(video_player, video_path, kill_switch, result_reporter):
    video_player.play(video_path, kill_switch, result_reporter)ReporterReporter is a class provided that the orchestration is using to collect reports/result/info from all its orchestrated events. An implemented event can use theresult_reporter()fixture to get the instance. The reporter has 2 main methods worth noticing;monitor()andon_result(). The monitor method will be running in the background of theorchestrationand monitor the Queue object it contains for new info, if it finds something thenon_result()will be called which will pop the info and log it. It is possible to implement your own Reporter class by inheriting from provided ResultReporter class and implementing your ownon_result()method. If you implement this you also need to override theresult_reporter(report_queue)fixture and make it return your reporter instead.FixturesThe plugin comes with a few fixture helpers, these are:kill_switch() - Gives you amultiprocessing.Event()which will be set when tests finishes, so your event can know when test has ended and tear itself down properly.result_reporter() - Facilitates a way to collect and report what is important from your events, see Reporter section.report_queue() - Fixture which returns a thread safemultiprocessing.Manager().Queue()object which is injected into result_reporter, could be overridden as well.UsageTo run a orchestration test you just need to specify --run-orch=<orch_name> where orch_name should be the name of the description configuration file excluding its extension.pytest --test-orch=my_orchestration_testThe option--load-orchis also available, specifying this will load all events. This is mostly meant for testing.