Package: ops-scenario
ScenarioScenario is a state-transition testing SDK for Operator Framework charms.Where the Harness enables you to procedurally mock pieces of the state the charm needs to function, Scenario tests allow
you to declaratively define the state all at once, and use it as a sort of context against which you can fire a single
event on the charm and execute its logic.This puts scenario tests somewhere in between unit and integration tests: some say 'functional', some say 'contract', I prefer 'state-transition'.Scenario tests nudge you into thinking of a charm as an input->output function. The input is the
union of anEvent(why am I, charm, being executed), aState(am I leader? what is my relation data? what is my
config?...) and the charm's executionContext(what relations can I have? what containers can I have?...). The output is anotherState: the state after the charm has had a chance to interact with the
mocked juju model and affect the initial state back.For example: a charm currently inunknownstatus is executed with astartevent, and based on whether it has leadership or not (according to its input state), it will decide to setactiveorblockedstatus (which will be reflected in the output state).Scenario-testing a charm, then, means verifying that:the charm does not raise uncaught exceptions while handling the eventthe output state (or the diff with the input state) is as expected.Core concepts as a metaphorI like metaphors, so here we go:There is a theatre stage.You pick an actor (a Charm) to put on the stage. Not just any actor: an improv one.You arrange the stage with content that the actor will have to interact with. This consists of selecting:An initial situation (State) in which the actor is, e.g. is the actor the main role or an NPC (is_leader), or what
other actors are there around it, what is written in those pebble-shaped books on the table?Something that has just happened (anEvent) and to which the actor has to react (e.g. one of the NPCs leaves the
stage (relation-departed), or the content of one of the books changes).How the actor will react to the event will have an impact on the context: e.g. the actor might knock over a table (a
container), or write something down into one of the books.Core concepts not as a metaphorScenario tests are about running assertions on atomic state transitions treating the charm being tested like a black
box. An initial state goes in, an event occurs (say,'start') and a new state comes out. Scenario tests are about
validating the transition, that is, consistency-checking the delta between the two states, and verifying the charm
author's expectations.Comparing scenario tests withHarnesstests:Harness exposes an imperative API: the user is expected to call methods on the Harness driving it to the desired
state, then verify its validity by calling charm methods or inspecting the raw data. In contrast, Scenario is declarative. You fully specify an initial state, an execution context and an event, then you run the charm and inspect the results.Harness instantiates the charm once, then allows you to fire multiple events on the charm, which is breeding ground
for subtle bugs. Scenario tests are centered around testing single state transitions, that is, one event at a time.
This ensures that the execution environment is as clean as possible (for a unit test).Harness maintains a model of the juju Model, which is a maintenance burden and adds complexity. Scenario mocks at the
level of hook tools and stores all mocking data in a monolithic data structure (the State), which makes it more
lightweight and portable.Writing scenario testsA scenario test consists of three broad steps:Arrange:declare the contextdeclare the input stateselect an event to fireAct:run the context (i.e. obtain the output state, given the input state and the event)Assert:verify that the output state (or the delta with the input state) is how you expect it to beverify that the charm has seen a certain sequence of statuses, events, andjuju-logcallsoptionally, you can use a context manager to get a hold of the charm instance and run assertions on internal APIs and the internal state of the charm and operator framework.The most basic scenario is one in which all is defaulted and barely any data is
available. The charm has no config, no relations, no leadership, and its status isunknown.With that, we can write the simplest possible scenario test:fromscenarioimportState,Context,Eventfromops.charmimportCharmBasefromops.modelimportUnknownStatusclassMyCharm(CharmBase):passdeftest_scenario_base():ctx=Context(MyCharm,meta={"name":"foo"})out=ctx.run(Event("start"),State())assertout.unit_status==UnknownStatus()Now let's start making it more complicated. Our charm sets a special state if it has leadership on 'start':importpytestfromscenarioimportState,Contextfromops.charmimportCharmBasefromops.modelimportActiveStatusclassMyCharm(CharmBase):def__init__(self,...):self.framework.observe(self.on.start,self._on_start)def_on_start(self,_):ifself.unit.is_leader():self.unit.status=ActiveStatus('I rule')else:self.unit.status=ActiveStatus('I am ruled')@pytest.mark.parametrize('leader',(True,False))deftest_status_leader(leader):ctx=Context(MyCharm,meta={"name":"foo"})out=ctx.run('start',State(leader=leader))assertout.unit_status==ActiveStatus('I rule'ifleaderelse'I am ruled')By defining the right state we can programmatically define what answers will the charm get to all the questions it can
ask the juju model: am I leader? What are my relations? What is the remote unit I'm talking to? etc...StatusesOne of the simplest types of black-box testing available to charmers is to execute the charm and verify that the charm
sets the expected unit/application status. We have seen a simple example above including leadership. But what if the
charm transitions through a sequence of statuses?fromops.modelimportMaintenanceStatus,ActiveStatus,WaitingStatus,BlockedStatus# charm code:def_on_event(self,_event):self.unit.status=MaintenanceStatus('determining who the ruler is...')try:ifself._call_that_takes_a_few_seconds_and_only_passes_on_leadership:self.unit.status=ActiveStatus('I rule')else:self.unit.status=WaitingStatus('checking this is right...')self._check_that_takes_some_more_time()self.unit.status=ActiveStatus('I am ruled')except:self.unit.status=BlockedStatus('something went wrong')More broadly, often we want to test 'side effects' of executing a charm, such as what events have been emitted, what
statuses it went through, etc... Before we get there, we have to explain what theContextrepresents, and its
relationship with theState.Context and StateConsider the following tests. Suppose we want to verify that while handling a given toplevel juju event:a specific chain of (custom) events was emitted on the charmthe charmjuju-logged these specific stringsthe charm went through this sequence of app/unit statuses (e.g.maintenance, thenwaiting, thenactive)These types of test have a place in Scenario, but that is not State: the contents of the juju log or the status history
are side effects of executing a charm, but are not persisted in a charm-accessible "state" in any meaningful way.
In other words: those data streams are, from the charm's perspective, write-only.As such, they do not belong inscenario.Statebut inscenario.Context: the object representing the charm's execution
context.Status historyYou can verify that the charm has followed the expected path by checking the unit/app status history like so:fromcharmimportMyCharmfromops.modelimportMaintenanceStatus,ActiveStatus,WaitingStatus,UnknownStatusfromscenarioimportState,Contextdeftest_statuses():ctx=Context(MyCharm,meta={"name":"foo"})ctx.run('start',State(leader=False))assertctx.unit_status_history==[UnknownStatus(),MaintenanceStatus('determining who the ruler is...'),WaitingStatus('checking this is right...'),ActiveStatus("I am ruled"),]# similarly you can check the app status history:assertctx.app_status_history==[UnknownStatus(),...]Note that the current status is not in theunit status history.Also note that, unless you initialize the State with a preexisting status, the first status in the history will always
beunknown. That is because, so far as scenario is concerned, each event is "the first event this charm has ever
seen".If you want to simulate a situation in which the charm already has seen some event, and is in a status other than
Unknown (the default status every charm is born with), you will have to pass the 'initial status' to State.fromops.modelimportActiveStatusfromscenarioimportState,Status# ...ctx.run('start',State(unit_status=ActiveStatus('foo')))assertctx.unit_status_history==[ActiveStatus('foo'),# now the first status is active: 'foo'!# ...]Workload version historyUsing a similar api to*_status_history, you can assert that the charm has set one or more workload versions during a
hook execution:fromscenarioimportContext# ...ctx:Contextassertctx.workload_version_history==['1','1.2','1.5']# ...Emitted eventsIf your charm deals with deferred events, custom events, and charm libs that in turn emit their own custom events, it
can be hard to examine the resulting control flow. In these situations it can be useful to verify that, as a result of a
given Juju event triggering (say, 'start'), a specific chain of events is emitted on the charm. The
resulting state, black-box as it is, gives little insight into how exactly it was obtained.fromscenarioimportContextfromops.charmimportStartEventdeftest_foo():ctx=Context(...)ctx.run('start',...)assertlen(ctx.emitted_events)==1assertisinstance(ctx.emitted_events[0],StartEvent)You can configure what events will be captured by passing the following arguments toContext:capture_deferred_events: If you want to include re-emitted deferred events.capture_framework_events: If you want to include framework events (pre-commit,commit, andcollect-status).For example:fromscenarioimportContext,Event,Statedeftest_emitted_full():ctx=Context(MyCharm,capture_deferred_events=True,capture_framework_events=True,)ctx.run("start",State(deferred=[Event("update-status").deferred(MyCharm._foo)]))assertlen(ctx.emitted_events)==5assert[e.handle.kindforeinctx.emitted_events]==["update_status","start","collect_unit_status","pre_commit","commit",]Low-level access: using directlycapture_eventsIf you need more control over what events are captured (or you're not into pytest), you can use directly the context
manager that powers theemitted_eventsfixture:scenario.capture_events.
This context manager allows you to intercept any events emitted by the framework.Usage:fromops.charmimportStartEvent,UpdateStatusEventfromscenarioimportState,Context,DeferredEvent,capture_eventswithcapture_events()asemitted:ctx=Context(...)state_out=ctx.run("update-status",State(deferred=[DeferredEvent("start",...)]))# deferred events get reemitted firstassertisinstance(emitted[0],StartEvent)# the main juju event gets emitted nextassertisinstance(emitted[1],UpdateStatusEvent)# possibly followed by a tail of all custom events that the main juju event triggered in turn# assert isinstance(emitted[2], MyFooEvent)# ...You can filter events by type like so:fromops.charmimportStartEvent,RelationEventfromscenarioimportcapture_eventswithcapture_events(StartEvent,RelationEvent)asemitted:# capture all `start` and `*-relation-*` events.passConfiguration:Passing no event types, like:capture_events(), is equivalent tocapture_events(EventBase).By default,framework events(PreCommit,Commit) are not considered for inclusion in the output list even if
they match the instance check. You can toggle that by passing:capture_events(include_framework=True).By default,deferred eventsare included in the listing if they match the instance check. You can toggle that by
passing:capture_events(include_deferred=False).RelationsYou can write scenario tests to verify the shape of relation data:fromops.charmimportCharmBasefromscenarioimportRelation,State,Context# This charm copies over remote app data to local unit dataclassMyCharm(CharmBase):...def_on_event(self,e):rel=e.relationassertrel.app.name=='remote'assertrel.data[self.unit]['abc']=='foo'rel.data[self.unit]['abc']=rel.data[e.app]['cde']deftest_relation_data():state_in=State(relations=[Relation(endpoint="foo",interface="bar",remote_app_name="remote",local_unit_data={"abc":"foo"},remote_app_data={"cde":"baz!"},),])ctx=Context(MyCharm,meta={"name":"foo"})state_out=ctx.run('start',state_in)assertstate_out.relations[0].local_unit_data=={"abc":"baz!"}# you can do this to check that there are no other differences:assertstate_out.relations==[Relation(endpoint="foo",interface="bar",remote_app_name="remote",local_unit_data={"abc":"baz!"},remote_app_data={"cde":"baz!"},),]# which is very idiomatic and superbly explicit. Noice.The only mandatory argument toRelation(and other relation types, see below) isendpoint. Theinterfacewill be
derived from the charm'smetadata.yaml. When fully defaulted, a relation is 'empty'. There are no remote units, the
remote application is called'remote'and only has a single unitremote/0, and nobody has written any data to the
databags yet.That is typically the state of a relation when the first unit joins it.When you useRelation, you are specifying a regular (conventional) relation. But that is not the only type of
relation. There are also peer relations and subordinate relations. While in the background the data model is the same,
the data access rules and the consistency constraints on them are very different. For example, it does not make sense
for a peer relation to have a different 'remote app' than its 'local app', because it's the same application.PeerRelationTo declare a peer relation, you should usescenario.state.PeerRelation. The core difference with regular relations is
that peer relations do not have a "remote app" (it's this app, in fact). So unlikeRelation, aPeerRelationdoes not
haveremote_app_nameorremote_app_dataarguments. Also, it talks in terms ofpeers:Relation.remote_units_datamaps toPeerRelation.peers_datafromscenario.stateimportPeerRelationrelation=PeerRelation(endpoint="peers",peers_data={1:{},2:{},42:{'foo':'bar'}},)be mindful when usingPeerRelationnot to include"this unit"'s ID inpeers_dataorpeers_ids, as that would
be flagged by the Consistency Checker:fromscenarioimportState,PeerRelation,Contextstate_in=State(relations=[PeerRelation(endpoint="peers",peers_data={1:{},2:{},42:{'foo':'bar'}},)])Context(...,unit_id=1).run("start",state_in)# invalid: this unit's id cannot be the ID of a peer.SubordinateRelationTo declare a subordinate relation, you should usescenario.state.SubordinateRelation. The core difference with regular
relations is that subordinate relations always have exactly one remote unit (there is always exactly one remote unit
that this unit can see).
Because of that,SubordinateRelation, compared toRelation, always talks in terms ofremote:Relation.remote_units_databecomesSubordinateRelation.remote_unit_datataking a singleDict[str:str]. The remote unit ID can be provided as a separate argument.Relation.remote_unit_idsbecomesSubordinateRelation.remote_unit_id(a single ID instead of a list of IDs)Relation.remote_units_databecomesSubordinateRelation.remote_unit_data(a single databag instead of a mapping from unit IDs to databags)fromscenario.stateimportSubordinateRelationrelation=SubordinateRelation(endpoint="peers",remote_unit_data={"foo":"bar"},remote_app_name="zookeeper",remote_unit_id=42)relation.remote_unit_name# "zookeeper/42"Triggering Relation EventsIf you want to trigger relation events, the easiest way to do so is get a hold of the Relation instance and grab the
event from one of its aptly-named properties:fromscenarioimportRelationrelation=Relation(endpoint="foo",interface="bar")changed_event=relation.changed_eventjoined_event=relation.joined_event# ...This is in fact syntactic sugar for:fromscenarioimportRelation,Eventrelation=Relation(endpoint="foo",interface="bar")changed_event=Event('foo-relation-changed',relation=relation)The reason for this construction is that the event is associated with some relation-specific metadata, that Scenario
needs to set up the process that will runops.mainwith the right environment variables.Working with relation IDsEvery time you instantiateRelation(or peer, or subordinate), the new instance will be given a uniquerelation_id.
To inspect the ID the next relation instance will have, you can callstate.next_relation_id.fromscenarioimportRelationfromscenario.stateimportnext_relation_idnext_id=next_relation_id(update=False)rel=Relation('foo')assertrel.relation_id==next_idThis can be handy when usingreplaceto create new relations, to avoid relation ID conflicts:fromscenarioimportRelationfromscenario.stateimportnext_relation_idrel=Relation('foo')rel2=rel.replace(local_app_data={"foo":"bar"},relation_id=next_relation_id())assertrel2.relation_id==rel.relation_id+1If you don't do this, and pass both relations into aState, you will trigger a consistency checker error.Additional event parametersAll relation events have some additional metadata that does not belong in the Relation object, such as, for a
relation-joined event, the name of the (remote) unit that is joining the relation. That is what determines whatops.model.Unityou get when you getRelationJoinedEvent().unitin an event handler.In order to supply this parameter, you will have tocallthe event object and pass asremote_unit_idthe id of the
remote unit that the event is about. The reason that this parameter is not supplied toRelationbut to relation
events, is that the relation already ties 'this app' to some 'remote app' (cfr. theRelation.remote_app_nameattr),
but not to a specific unit. What remote unit this event is about is not aStateconcern but anEventone.Theremote_unit_idwill default to the first ID found in the relation'sremote_units_data, but if the test you are
writing is close to that domain, you should probably override it and pass it manually.fromscenarioimportRelation,Eventrelation=Relation(endpoint="foo",interface="bar")remote_unit_2_is_joining_event=relation.joined_event(remote_unit_id=2)# which is syntactic sugar for:remote_unit_2_is_joining_event=Event('foo-relation-changed',relation=relation,relation_remote_unit_id=2)NetworksSimplifying a bit the Juju "spaces" model, each integration endpoint a charm defines in its metadata is associated with a network. Regardless of whether there is a living relation over that endpoint, that is.If your charm has a relation"foo"(defined in metadata.yaml), then the charm will be able at runtime to doself.model.get_binding("foo").network.
The network you'll get by doing so is heavily defaulted (seestate.Network.default) and good for most use-cases because the charm should typically not be concerned about what IP it gets.On top of the relation-provided network bindings, a charm can also define someextra-bindingsin its metadata.yaml and access them at runtime. Note that this is a deprecated feature that should not be relied upon. For completeness, we support it in Scenario.If you want to, you can override any of these relation or extra-binding associated networks with a custom one by passing it toState.networks.fromscenarioimportState,Networkstate=State(networks={'foo':Network.default(private_address='4.4.4.4')})Wherefoocan either be the name of anextra-bindings-defined binding, or a relation endpoint.ContainersWhen testing a kubernetes charm, you can mock container interactions. When using the null state (State()), there will
be no containers. So if the charm were toself.unit.containers, it would get back an empty dict.To give the charm access to some containers, you need to pass them to the input state, like so:State(containers=[...])An example of a state including some containers:fromscenario.stateimportContainer,Statestate=State(containers=[Container(name="foo",can_connect=True),Container(name="bar",can_connect=False)])In this case,self.unit.get_container('foo').can_connect()would returnTrue, while for 'bar' it would giveFalse.Container filesystem setupYou can configure a container to have some files in it:frompathlibimportPathfromscenario.stateimportContainer,State,Mountlocal_file=Path('/path/to/local/real/file.txt')container=Container(name="foo",can_connect=True,mounts={'local':Mount('/local/share/config.yaml',local_file)})state=State(containers=[container])In this case, if the charm were to:def_on_start(self,_):foo=self.unit.get_container('foo')content=foo.pull('/local/share/config.yaml').read()thencontentwould be the contents of our locally-suppliedfile.txt. You can usetempdirfor nicely wrapping
data and passing it to the charm via the container.container.pushworks similarly, so you can write a test like:importtempfilefromops.charmimportCharmBasefromscenarioimportState,Container,Mount,ContextclassMyCharm(CharmBase):def__init__(self,*args):super().__init__(*args)self.framework.observe(self.on.foo_pebble_ready,self._on_pebble_ready)def_on_pebble_ready(self,_):foo=self.unit.get_container('foo')foo.push('/local/share/config.yaml',"TEST",make_dirs=True)deftest_pebble_push():withtempfile.NamedTemporaryFile()aslocal_file:container=Container(name='foo',can_connect=True,mounts={'local':Mount('/local/share/config.yaml',local_file.name)})state_in=State(containers=[container])Context(MyCharm,meta={"name":"foo","containers":{"foo":{}}}).run(container.pebble_ready_event(),state_in,)assertlocal_file.read().decode()=="TEST"container.pebble_ready_eventis syntactic sugar for:Event("foo-pebble-ready", container=container). The reason we
need to associate the container with the event is that the Framework uses an envvar to determine which container the
pebble-ready event is about (it does not use the event name). Scenario needs that information, similarly, for injecting
that envvar into the charm's runtime.Container filesystem post-mortemIf the charm writes files to a container (to a location you didn't Mount as a temporary folder you have access to), you will be able to inspect them using theget_filesystemapi.fromops.charmimportCharmBasefromscenarioimportState,Container,Mount,ContextclassMyCharm(CharmBase):def__init__(self,*args):super().__init__(*args)self.framework.observe(self.on.foo_pebble_ready,self._on_pebble_ready)def_on_pebble_ready(self,_):foo=self.unit.get_container('foo')foo.push('/local/share/config.yaml',"TEST",make_dirs=True)deftest_pebble_push():container=Container(name='foo',can_connect=True)state_in=State(containers=[container])ctx=Context(MyCharm,meta={"name":"foo","containers":{"foo":{}}})ctx.run("start",state_in)# this is the root of the simulated container filesystem. Any mounts will be symlinks in it.container_root_fs=container.get_filesystem(ctx)cfg_file=container_root_fs/'local'/'share'/'config.yaml'assertcfg_file.read_text()=="TEST"Container.execmockscontainer.execis a tad more complicated, but if you get to this low a level of simulation, you probably will have far
worse issues to deal with. You need to specify, for each possible command the charm might run on the container, what the
result of that would be: its return code, what will be written to stdout/stderr.fromops.charmimportCharmBasefromscenarioimportState,Container,ExecOutput,ContextLS_LL=""".rw-rw-r--  228 ubuntu ubuntu 18 jan 12:05 -- charmcraft.yaml.rw-rw-r--  497 ubuntu ubuntu 18 jan 12:05 -- config.yaml.rw-rw-r--  900 ubuntu ubuntu 18 jan 12:05 -- CONTRIBUTING.mddrwxrwxr-x    - ubuntu ubuntu 18 jan 12:06 -- lib"""classMyCharm(CharmBase):def_on_start(self,_):foo=self.unit.get_container('foo')proc=foo.exec(['ls','-ll'])stdout,_=proc.wait_output()assertstdout==LS_LLdeftest_pebble_exec():container=Container(name='foo',exec_mock={('ls','-ll'):# this is the command we're mockingExecOutput(return_code=0,# this data structure contains all we need to mock the call.stdout=LS_LL)})state_in=State(containers=[container])state_out=Context(MyCharm,meta={"name":"foo","containers":{"foo":{}}},).run(container.pebble_ready_event,state_in,)StorageIf your charm definesstoragein its metadata, you can usescenario.state.Storageto instruct Scenario to make (mocked) filesystem storage available to the charm at runtime.Using the sameget_filesystemAPI asContainer, you can access the tempdir used by Scenario to mock the filesystem root before and after the scenario runs.fromscenarioimportStorage,Context,State# some charm with a 'foo' filesystem-type storage defined in metadata.yamlctx=Context(MyCharm)storage=Storage("foo")# setup storage with some content(storage.get_filesystem(ctx)/"myfile.txt").write_text("helloworld")withctx.manager("update-status",State(storage=[storage]))asmgr:foo=mgr.charm.model.storages["foo"][0]loc=foo.locationpath=loc/"myfile.txt"assertpath.exists()assertpath.read_text()=="helloworld"myfile=loc/"path.py"myfile.write_text("helloworlds")# post-mortem: inspect fs contents.assert(storage.get_filesystem(ctx)/"path.py").read_text()=="helloworlds"Note that State only wants to know aboutattachedstorages. A storage which is not attached to the charm can simply be omitted from State and the charm will be none the wiser.Storage-addIf a charm requests adding more storage instances while handling some event, you can inspect that from theContext.requested_storageAPI.# in MyCharm._on_foo:# the charm requests two new "foo" storage instances to be provisionedself.model.storages.request("foo",2)From test code, you can inspect that:fromscenarioimportContext,Statectx=Context(MyCharm)ctx.run('some-event-that-will-cause_on_foo-to-be-called',State())# the charm has requested two 'foo' storages to be provisionedassertctx.requested_storages['foo']==2Requesting storages has no other consequence in Scenario. In real life, this request will trigger Juju to provision the storage and execute the charm again withfoo-storage-attached.
So a natural follow-up Scenario test suite for this case would be:fromscenarioimportContext,State,Storagectx=Context(MyCharm)foo_0=Storage('foo')# the charm is notified that one of the storages it has requested is readyctx.run(foo_0.attached_event,State(storage=[foo_0]))foo_1=Storage('foo')# the charm is notified that the other storage is also readyctx.run(foo_1.attached_event,State(storage=[foo_0,foo_1]))PortsSinceops 2.6.0, charms can invoke theopen-port,close-port, andopened-portshook tools to manage the ports opened on the host vm/container. Using theState.opened_portsapi, you can:simulate a charm run with a port opened by some previous executionfromscenarioimportState,Port,Contextctx=Context(MyCharm)ctx.run("start",State(opened_ports=[Port("tcp",42)]))assert that a charm has calledopen-portorclose-port:fromscenarioimportState,Port,Contextctx=Context(MyCharm)state1=ctx.run("start",State())assertstate1.opened_ports==[Port("tcp",42)]state2=ctx.run("stop",state1)assertstate2.opened_ports==[]SecretsScenario has secrets. Here's how you use them.fromscenarioimportState,Secretstate=State(secrets=[Secret(id='foo',contents={0:{'key':'public'}})])The only mandatory arguments to Secret are its secret ID (which should be unique) and its 'contents': that is, a mapping
from revision numbers (integers) to astr:strdict representing the payload of the revision.There are three cases:the secret is owned by this app but not this unit, in which case this charm can only manage it if we are the leaderthe secret is owned by this unit, in which case this charm can always manage it (leader or not)(default) the secret is not owned by this app nor unit, which means we can't manage it but only view itThus by default, the secret is not owned bythis charm, but, implicitly, by some unknown 'other charm', and that other charm has granted us view rights.The presence of the secret inState.secretsentails that we have access to it, either as owners or as grantees. Therefore, if we're not owners, we must be grantees. Absence of a Secret from the known secrets list means we are not entitled to obtaining it in any way. The charm, indeed, shouldn't even know it exists.[note]
If this charm does not own the secret, but also it was not granted view rights by the (remote) owner, you model this in Scenario bynot adding it to State.secrets! The presence of aSecretinState.secretsmeans, in other words, that the charm has view rights (otherwise, why would we put it there?). If the charm owns the secret, or is leader, it willalsohave manage rights on top of view ones.
[/note]To specify a secret owned by this unit (or app):fromscenarioimportState,Secretstate=State(secrets=[Secret(id='foo',contents={0:{'key':'private'}},owner='unit',# or 'app'remote_grants={0:{"remote"}}# the secret owner has granted access to the "remote" app over some relation with ID 0)])To specify a secret owned by some other application and give this unit (or app) access to it:fromscenarioimportState,Secretstate=State(secrets=[Secret(id='foo',contents={0:{'key':'public'}},# owner=None, which is the defaultrevision=0,# the revision that this unit (or app) is currently tracking)])ActionsAn action is a special sort of event, even thoughopshandles them almost identically.
In most cases, you'll want to inspect the 'results' of an action, or whether it has failed or
logged something while executing. Many actions don't have a direct effect on the output state.
For this reason, the output state is less prominent in the return type ofContext.run_action.How to test actions with scenario:Actions without parametersfromscenarioimportContext,State,ActionOutputfromcharmimportMyCharmdeftest_backup_action():ctx=Context(MyCharm)# If you didn't declare do_backup in the charm's `actions.yaml`,# the `ConsistencyChecker` will slap you on the wrist and refuse to proceed.out:ActionOutput=ctx.run_action("do_backup_action",State())# you can assert action results, logs, failure using the ActionOutput interfaceassertout.logs==['baz','qux']ifout.success:# if the action did not fail, we can read the results:assertout.results=={'foo':'bar'}else:# if the action fails, we can read a failure messageassertout.failure=='boo-hoo'Parametrized ActionsIf the action takes parameters, you'll need to instantiate anAction.fromscenarioimportAction,Context,State,ActionOutputfromcharmimportMyCharmdeftest_backup_action():# define an actionaction=Action('do_backup',params={'a':'b'})ctx=Context(MyCharm)# if the parameters (or their type) don't match what declared in actions.yaml,# the `ConsistencyChecker` will slap you on the other wrist.out:ActionOutput=ctx.run_action(action,State())# ...Deferred eventsScenario allows you to accurately simulate the Operator Framework's event queue. The event queue is responsible for
keeping track of the deferred events. On the input side, you can verify that if the charm triggers with this and that
event in its queue (they would be there because they had been deferred in the previous run), then the output state is
valid.fromscenarioimportState,deferred,ContextclassMyCharm(...):...def_on_update_status(self,e):e.defer()def_on_start(self,e):e.defer()deftest_start_on_deferred_update_status(MyCharm):"""Test charm execution if a 'start' is dispatched when in the previous run an update-status had been deferred."""state_in=State(deferred=[deferred('update_status',handler=MyCharm._on_update_status)])state_out=Context(MyCharm).run('start',state_in)assertlen(state_out.deferred)==1assertstate_out.deferred[0].name=='start'You can also generate the 'deferred' data structure (called a DeferredEvent) from the corresponding Event (and the
handler):fromscenarioimportEvent,RelationclassMyCharm(...):...deferred_start=Event('start').deferred(MyCharm._on_start)deferred_install=Event('install').deferred(MyCharm._on_start)relation events:foo_relation=Relation('foo')deferred_relation_changed_evt=foo_relation.changed_event.deferred(handler=MyCharm._on_foo_relation_changed)On the output side, you can verify that an event that you expect to have been deferred during this trigger, has indeed
been deferred.fromscenarioimportState,ContextclassMyCharm(...):...def_on_start(self,e):e.defer()deftest_defer(MyCharm):out=Context(MyCharm).run('start',State())assertlen(out.deferred)==1assertout.deferred[0].name=='start'Deferring relation eventsIf you want to test relation event deferrals, some extra care needs to be taken. RelationEvents hold references to the
Relation instance they are about. So do they in Scenario. You can use the deferred helper to generate the data
structure:fromscenarioimportState,Relation,deferredclassMyCharm(...):...def_on_foo_relation_changed(self,e):e.defer()deftest_start_on_deferred_update_status(MyCharm):foo_relation=Relation('foo')State(relations=[foo_relation],deferred=[deferred('foo_relation_changed',handler=MyCharm._on_foo_relation_changed,relation=foo_relation)])but you can also use a shortcut from the relation event itself, as mentioned above:fromscenarioimportRelationclassMyCharm(...):...foo_relation=Relation('foo')foo_relation.changed_event.deferred(handler=MyCharm._on_foo_relation_changed)Fine-tuningThe deferred helper Scenario provides will not support out of the box all custom event subclasses, or events emitted by
charm libraries or objects other than the main charm class.For general-purpose usage, you will need to instantiate DeferredEvent directly.fromscenarioimportDeferredEventmy_deferred_event=DeferredEvent(handle_path='MyCharm/MyCharmLib/on/database_ready[1]',owner='MyCharmLib',# the object observing the event. Could also be MyCharm.observer='_on_database_ready')StoredStateScenario can simulate StoredState. You can define it on the input side as:fromops.charmimportCharmBasefromops.frameworkimportStoredStateasOps_StoredState,FrameworkfromscenarioimportState,StoredStateclassMyCharmType(CharmBase):my_stored_state=Ops_StoredState()def__init__(self,framework:Framework):super().__init__(framework)assertself.my_stored_state.foo=='bar'# this will pass!state=State(stored_state=[StoredState(owner_path="MyCharmType",name="my_stored_state",content={'foo':'bar','baz':{42:42},})])And the charm's runtime will seeself.stored_State.fooand.bazas expected. Also, you can run assertions on it on
the output side the same as any other bit of state.ResourcesIf your charm requires access to resources, you can make them available to it throughState.resources.
From the perspective of a 'real' deployed charm, if your charmhasresources defined inmetadata.yaml, theymustbe made available to the charm. That is a Juju-enforced constraint: you can't deploy a charm without attaching all resources it needs to it.
However, when testing, this constraint is unnecessarily strict (and it would also mean the great majority of all existing tests would break) since a charm will only notice that a resource is not available when it explicitly asks for it, which not many charms do.So, the only consistency-level check we enforce in Scenario when it comes to resource is that if a resource is provided in State, it needs to have been declared in metadata.fromscenarioimportState,Contextctx=Context(MyCharm,meta={'name':'juliette',"resources":{"foo":{"type":"oci-image"}}})withctx.manager("start",State(resources={'foo':'/path/to/resource.tar'}))asmgr:# if the charm, at runtime, were to call self.model.resources.fetch("foo"), it would get '/path/to/resource.tar' back.path=mgr.charm.model.resources.fetch('foo')assertpath=='/path/to/resource.tar'Emitting custom eventsWhile the main use case of Scenario is to emit juju events, i.e. the built-instart,install,*-relation-changed,
etc..., it can be sometimes handy to directly trigger custom events defined on arbitrary Objects in your hierarchy.Suppose your charm uses a charm library providing aningress_providedevent.
The 'proper' way to emit it is to run the event that causes that custom event to be emitted by the library, whatever
that may be, for example afoo-relation-changed.However, that may mean that you have to set up all sorts of State and mocks so that the right preconditions are met and
the event is emitted at all.If for whatever reason you don't want to do that and you attempt to run that event directly you will get an error:fromscenarioimportContext,StateContext(...).run("ingress_provided",State())# raises scenario.ops_main_mock.NoObserverErrorThis happens because the framework, by default, searches for an event source namedingress_providedincharm.on, but
since the event is defined on another Object, it will fail to find it.You can prefix the event name with the path leading to its owner to tell Scenario where to find the event source:fromscenarioimportContext,StateContext(...).run("my_charm_lib.on.foo",State())This will instruct Scenario to emitmy_charm.my_charm_lib.on.foo.(always omit the 'root', i.e. the charm framework key, from the path)Live charm introspectionScenario is a black-box, state-transition testing framework. It makes it trivial to assert that a status went from A to
B, but not to assert that, in the context of this charm execution, with this state, a certain charm-internal method was called and returned a
given piece of data, or would return this and thatifit had been called.Scenario offers a cheekily-named context manager for this use case specifically:fromopsimportCharmBase,StoredStatefromcharms.bar.lib_name.v1.charm_libimportCharmLibfromscenarioimportContext,StateclassMyCharm(CharmBase):META={"name":"mycharm"}_stored=StoredState()def__init__(self,framework):super().__init__(framework)self._stored.set_default(a="a")self.my_charm_lib=CharmLib()framework.observe(self.on.start,self._on_start)def_on_start(self,event):self._stored.a="b"deftest_live_charm_introspection(mycharm):ctx=Context(mycharm,meta=mycharm.META)# If you want to do this with actions, you can use `Context.action_manager` instead.withctx.manager("start",State())asmanager:# this is your charm instance, after ops has set it upcharm:MyCharm=manager.charm# we can check attributes on nested Objects or the charm itselfassertcharm.my_charm_lib.foo=="foo"# such as stored stateassertcharm._stored.a=="a"# this will tell ops.main to proceed with normal execution and emit the "start" event on the charmstate_out=manager.run()# after that is done, we are handed back control, and we can again do some introspectionassertcharm.my_charm_lib.foo=="bar"# and check that the charm's internal state is as we expectassertcharm._stored.a=="b"# state_out is, as in regular scenario tests, a State object you can assert on:assertstate_out.unit_status==...Note that you can't callmanager.run()multiple times: the manager is a context that ensures thatops.main'pauses' right
before emitting the event to hand you some introspection hooks, but for the rest this is a regular scenario test: you
can't emit multiple events in a single charm execution.The virtual charm rootBefore executing the charm, Scenario copies the charm's/src, any libs, the metadata, config, and actionsyamls to a temporary directory. The
charm will see that tempdir as its 'root'. This allows us to keep things simple when dealing with metadata that can be
either inferred from the charm type being passed toContextor be passed to it as an argument, thereby overriding
the inferred one. This also allows you to test charms defined on the fly, as in:fromops.charmimportCharmBasefromscenarioimportState,ContextclassMyCharmType(CharmBase):passctx=Context(charm_type=MyCharmType,meta={'name':'my-charm-name'})ctx.run('start',State())A consequence of this fact is that you have no direct control over the tempdir that we are creating to put the metadata
you are passing to.run()(becauseopsexpects it to be a file...). That is, unless you pass your own:fromops.charmimportCharmBasefromscenarioimportState,ContextimporttempfileclassMyCharmType(CharmBase):passtd=tempfile.TemporaryDirectory()state=Context(charm_type=MyCharmType,meta={'name':'my-charm-name'},charm_root=td.name).run('start',State())Do this, and you will be able to set up said directory as you like before the charm is run, as well as verify its
contents after the charm has run. Do keep in mind that any metadata files you create in it will be overwritten by Scenario, and therefore
ignored, if you pass any metadata keys toContext. Omitmetain the call
above, and Scenario will instead attempt to readmetadata.yamlfrom the
temporary directory.ImmutabilityAll of the data structures instate, e.g.State, Relation, Container, etc... are immutable (implemented as frozen
dataclasses).This means that all components of the state that goes into acontext.run()call are not mutated by the call, and the
state that you obtain in return is a different instance, and all parts of it have been (deep)copied.
This ensures that you can do delta-based comparison of states without worrying about them being mutated by scenario.If you want to modify any of these data structures, you will need to either reinstantiate it from scratch, or use
thereplaceapi.fromscenarioimportRelationrelation=Relation('foo',remote_app_data={"1":"2"})# make a copy of relation, but with remote_app_data set to {"3", "4"}relation2=relation.replace(remote_app_data={"3","4"})Consistency checksA Scenario, that is, the combination of an event, a state, and a charm, is consistent if it's plausible in JujuLand. For
example, Juju can't emit afoo-relation-changedevent on your charm unless your charm has declared afoorelation
endpoint in itsmetadata.yaml. If that happens, that's a juju bug. Scenario however assumes that Juju is bug-free,
therefore, so far as we're concerned, that can't happen, and therefore we help you verify that the scenarios you create
are consistent and raise an exception if that isn't so.That happens automatically behind the scenes whenever you trigger an event;scenario.consistency_checker.check_consistencyis called and verifies that the scenario makes sense.Caveats:False positives: not all checks are implemented yet; more will come.False negatives: it is possible that a scenario you know to be consistent is seen as inconsistent. That is probably a
bug in the consistency checker itself, please report it.Inherent limitations: if you have a custom event whose name conflicts with a builtin one, the consistency constraints
of the builtin one will apply. For example: if you decide to name your custom eventbar-pebble-ready, but you are
working on a machine charm or don't have either way abarcontainer in yourmetadata.yaml, Scenario will flag that
as inconsistent.Bypassing the checkerIf you have a clear false negative, are explicitly testing 'edge', inconsistent situations, or for whatever reason the
checker is in your way, you can set theSCENARIO_SKIP_CONSISTENCY_CHECKSenvvar and skip it altogether. Hopefully you
don't need that.Jhack integrationsUp untilv5.6.0,scenarioshipped with a cli tool calledsnapshot, used to interact with a live charm's state.
The functionalityhas been moved over tojhack,
to allow us to keep working on it independently, and to streamline
the profile ofscenarioitself as it becomes more broadly adopted and ready for widespread usage.