Repository: canonical/operator

Issue:
`get_pebble` not implemented in _TestingModelBackend so unit tests for pebble charms not possible
I'm working on converting a charm to use the new pebble approach as a proof of concept, but it seems all tests are failing with the following:
```
self = <test_charm.TestGunicornK8sCharm testMethod=test_validate_yaml>

    def setUp(self):
        """Setup the harness object."""
>       self.harness = testing.Harness(GunicornK8sCharm)

tests/unit/test_charm.py:33:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
build/venv/ops/testing.py:87: in __init__
    self._model = model.Model(self._meta, self._backend)
build/venv/ops/model.py:50: in __init__
    self._unit = self.get_unit(self._backend.unit_name)
build/venv/ops/model.py:115: in get_unit
    return self._cache.get(Unit, unit_name)
build/venv/ops/model.py:171: in get
    entity = entity_type(*args, meta=self._meta, backend=self._backend, cache=self)
build/venv/ops/model.py:272: in __init__
    self._containers = ContainerMapping(meta.containers, backend)
build/venv/ops/model.py:1080: in __init__
    self._containers = {name: Container(name, backend) for name in names}
build/venv/ops/model.py:1080: in <dictcomp>
    self._containers = {name: Container(name, backend) for name in names}
build/venv/ops/model.py:1030: in __init__
    pebble_client = backend.get_pebble(socket_path)
build/venv/ops/testing.py:667: in wrapped
    return orig_method(self, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ops.testing._TestingModelBackend object at 0x7f53bcbc3c40>, container_name = '/charm/containers/gunicorn/pebble.socket'

    def get_pebble(self, container_name):
>       raise NotImplementedError(self.get_pebble)
E       NotImplementedError: <bound method _record_calls.<locals>.decorator.<locals>.wrapped of <ops.testing._TestingModelBackend object at 0x7f53bcbc3c40>>

build/venv/ops/testing.py:838: NotImplementedError
```

Discussion:
jameinel: Note that this currently fails the test suite because of things like:
======================================================================
FAIL: test_quote_backslashes (test.test_infra.InfrastructureTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jameinel/dev/operator/test/test_infra.py", line 107, in test_quote_backslashes
    self.fail("Spurious backslashes found, please fix these quotings:\n" + "\n".join(msgs))
AssertionError: Spurious backslashes found, please fix these quotings:
ops/testing.py:985:                    raise RuntimeError("500 Internal Server Error: layer \"{}\" must define"
ops/testing.py:986:                                       "'override' for service \"{}\"".format(label, name))
ops/testing.py:988:                    raise RuntimeError("500 Internal Server Error: layer \"{}\" has invalid "
ops/testing.py:989:                                       "'override' value on service \"{}\"".format(label, name))


The issue in our test suite is that it doesn't like strings that mix '' and "". However the output from pebble does exactly this. I can change it in the harness, or I can wrap the whole thing in ''' or """. But I'm out for now.

jnsgruk: Hey @jameinel - this is neat! Thanks! :sunglasses: 

Have committed an example that seems to work nice, interested to know if this was roughly how you'd expect this to be used? https://github.com/jnsgruk/hello-operator/blob/3456ef5877bb26d42fd940aefc679bedd137106c/tests/test_charm.py#L38

jameinel: So I would actually expect that you wouldn't mock out workload.add_layer,
but actually do the event, and then at the end of it you can do:

# Check that the layer was correctly installed
self.assertEqual(harness.get_container_pebble_plan('httpbin').to_dict(),
expected)
# and that the service was started
self.assertTrue(harness.unit.container.get_container('httpbin').is_running())
^ this might be something we want to streamline a bit more.



On Tue, Apr 20, 2021 at 5:25 AM Jon Seager ***@***.***> wrote:

> Hey @jameinel <https://github.com/jameinel> - this is neat! Thanks! 😎
>
> Have committed an example that seems to work nice, interested to know if
> this was roughly how you'd expect this to be used?
> https://github.com/jnsgruk/hello-operator/blob/3456ef5877bb26d42fd940aefc679bedd137106c/tests/test_charm.py#L38
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/canonical/operator/pull/515#issuecomment-823126303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABRQ7OQUWNV5V67MXWO5ULTJVCCBANCNFSM43G4MK4A>
> .
>

jnsgruk: Ah yes, I'd already updated this to stop mocking but hadn't yet added the is_running check. Thanks!

Patch:
diff --git a/ops/pebble.py b/ops/pebble.py
index e498b3c..a34d901 100644
--- a/ops/pebble.py
+++ b/ops/pebble.py
@@ -426,9 +426,18 @@ class Plan:
         """
         return self._services
 
+    def to_dict(self) -> typing.Dict[str, typing.Any]:
+        """Convert this plan to its dict representation."""
+        as_dicts = {name: service.to_dict() for name, service in self._services.items()}
+        if not as_dicts:
+            return {}
+        return {
+            'services': as_dicts,
+        }
+
     def to_yaml(self) -> str:
         """Return this plan's YAML representation."""
-        return self._raw
+        return yaml.safe_dump(self.to_dict())
 
     __str__ = to_yaml
 
@@ -438,8 +447,18 @@ class Layer:
 
     The format of this is not documented, but is captured in code here:
     https://github.com/canonical/pebble/blob/master/internal/plan/plan.go
+
+    Attributes:
+        summary: A summary of the purpose of this layer
+        description: A long form description of this layer
+        services: A mapping of name: :class:`Service` defined by this layer
     """
 
+    # This is how you do type annotations, but it is not supported by Python 3.5
+    # summary: str
+    # description: str
+    # services: typing.Mapping[str, 'Service']
+
     def __init__(self, raw: typing.Union[str, typing.Dict] = None):
         if isinstance(raw, str):
             d = yaml.safe_load(raw) or {}
@@ -454,7 +473,7 @@ class Layer:
         """Convert this layer to its YAML representation."""
         return yaml.safe_dump(self.to_dict())
 
-    def to_dict(self) -> typing.Dict:
+    def to_dict(self) -> typing.Dict[str, typing.Any]:
         """Convert this layer to its dict representation."""
         fields = [
             ('summary', self.summary),


Test patch:
diff --git a/ops/testing.py b/ops/testing.py
index d7e6c2f..6a4048e 100755
--- a/ops/testing.py
+++ b/ops/testing.py
@@ -1,4 +1,4 @@
-# Copyright 2020 Canonical Ltd.
+# Copyright 2021 Canonical Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,6 +14,7 @@
 
 """Infrastructure to build unittests for Charms using the Operator Framework."""
 
+import datetime
 import inspect
 import pathlib
 import random
@@ -26,6 +27,7 @@ from ops import (
     charm,
     framework,
     model,
+    pebble,
     storage,
 )
 from ops._private import yaml
@@ -469,7 +471,7 @@ class Harness:
             relation_id: The relation whose content we want to look at.
             app_or_unit: The name of the application or unit whose data we want to read
         Return:
-            a dict containing the relation data for `app_or_unit` or None.
+            A dict containing the relation data for `app_or_unit` or None.
 
         Raises:
             KeyError: if relation_id doesn't exist
@@ -484,6 +486,25 @@ class Harness:
         """
         return self._backend._pod_spec
 
+    def get_container_pebble_plan(
+            self, container_name: str
+    ) -> pebble.Plan:
+        """Return the current Plan that pebble is executing for the given container.
+
+        Args:
+            container_name: The simple name of the associated container
+        Return:
+            The pebble.Plan for this container. You can use :meth:`pebble.Plan.to_yaml` to get
+            a string form for the content. Will raise KeyError if no pebble client exists
+            for that container name. (should only happen if container is not present in
+            metadata.yaml)
+        """
+        socket_path = '/charm/containers/{}/pebble.socket'.format(container_name)
+        client = self._backend._pebble_clients.get(socket_path)
+        if client is None:
+            raise KeyError('no known pebble client for container "{}"'.format(container_name))
+        return client.get_plan()
+
     def get_workload_version(self) -> str:
         """Read the workload version that was set by the unit."""
         return self._backend._workload_version
@@ -671,6 +692,27 @@ def _record_calls(cls):
     return cls
 
 
+def _copy_docstrings(source_cls):
+    """Copy the docstrings from source_cls to target_cls.
+
+    Use this as:
+      @_copy_docstrings(source_class)
+      class TargetClass:
+
+    And for any public method that exists on both classes, it will copy the
+    __doc__ for that method.
+    """
+    def decorator(target_cls):
+        for meth_name, orig_method in target_cls.__dict__.items():
+            if meth_name.startswith('_'):
+                continue
+            source_method = source_cls.__dict__.get(meth_name)
+            if source_method is not None and source_method.__doc__:
+                target_cls.__dict__[meth_name].__doc__ = source_method.__doc__
+        return target_cls
+    return decorator
+
+
 class _ResourceEntry:
     """Tracks the contents of a Resource."""
 
@@ -678,6 +720,7 @@ class _ResourceEntry:
         self.name = resource_name
 
 
+@_copy_docstrings(model._ModelBackend)
 @_record_calls
 class _TestingModelBackend:
     """This conforms to the interface for ModelBackend but provides canned data.
@@ -708,6 +751,9 @@ class _TestingModelBackend:
         self._unit_status = {'status': 'maintenance', 'message': ''}
         self._workload_version = None
         self._resource_dir = None
+        # {socket_path : _TestingPebbleClient}
+        # socket_path = '/charm/containers/{container_name}/pebble.socket'
+        self._pebble_clients = {}  # type: {str: _TestingPebbleClient}
 
     def _cleanup(self):
         if self._resource_dir is not None:
@@ -834,5 +880,232 @@ class _TestingModelBackend:
     def add_metrics(self, metrics, labels=None):
         raise NotImplementedError(self.add_metrics)
 
-    def get_pebble(self, container_name):
-        raise NotImplementedError(self.get_pebble)
+    def juju_log(self, level, msg):
+        raise NotImplementedError(self.juju_log)
+
+    def get_pebble(self, socket_path: str):
+        client = self._pebble_clients.get(socket_path, None)
+        if client is None:
+            client = _TestingPebbleClient(self)
+            self._pebble_clients[socket_path] = client
+        return client
+
+
+@_copy_docstrings(pebble.Client)
+class _TestingPebbleClient:
+    """This conforms to the interface for pebble.Client but provides canned data.
+
+    DO NOT use this class directly, it is used by `Harness`_ to run interactions with Pebble.
+    `Harness`_ is responsible for maintaining the internal consistency of the values here,
+    as the only public methods of this type are for implementing Client.
+    """
+
+    def __init__(self, backend: _TestingModelBackend):
+        self._backend = _TestingModelBackend
+        self._layers = {}
+        # Has a service been started/stopped?
+        self._service_status = {}
+
+    def get_system_info(self) -> pebble.SystemInfo:
+        raise NotImplementedError(self.get_system_info)
+
+    def get_warnings(
+            self, select: pebble.WarningState = pebble.WarningState.PENDING,
+    ) -> typing.List['pebble.Warning']:
+        raise NotImplementedError(self.get_warnings)
+
+    def ack_warnings(self, timestamp: datetime.datetime) -> int:
+        raise NotImplementedError(self.ack_warnings)
+
+    def get_changes(
+            self, select: pebble.ChangeState = pebble.ChangeState.IN_PROGRESS, service: str = None,
+    ) -> typing.List[pebble.Change]:
+        raise NotImplementedError(self.get_changes)
+
+    def get_change(self, change_id: pebble.ChangeID) -> pebble.Change:
+        raise NotImplementedError(self.get_change)
+
+    def abort_change(self, change_id: pebble.ChangeID) -> pebble.Change:
+        raise NotImplementedError(self.abort_change)
+
+    def autostart_services(self, timeout: float = 30.0, delay: float = 0.1) -> pebble.ChangeID:
+        for name, service in self._render_services().items():
+            # TODO: jam 2021-04-20 This feels awkward that Service.startup might be a string or
+            #  might be an enum. Probably should make Service.startup a property rather than an
+            #  attribute.
+            if service.startup == '':
+                startup = pebble.ServiceStartup.DISABLED
+            else:
+                startup = pebble.ServiceStartup(service.startup)
+            if startup == pebble.ServiceStartup.ENABLED:
+                self._service_status[name] = pebble.ServiceStatus.ACTIVE
+
+    def start_services(
+            self, services: typing.List[str], timeout: float = 30.0, delay: float = 0.1,
+    ) -> pebble.ChangeID:
+        # A common mistake is to pass just the name of a service, rather than a list of services,
+        # so trap that so it is caught quickly.
+        if isinstance(services, str):
+            raise TypeError('start_services should take a list of names, not just "{}"'.format(
+                services))
+
+        # Note: jam 2021-04-20 We don't implement ChangeID, but the default caller of this is
+        # Container.start() which currently ignores the return value
+        known_services = self._render_services()
+        # Names appear to be validated before any are activated, so do two passes
+        for name in services:
+            if name not in known_services:
+                # TODO: jam 2021-04-20 This needs a better error type
+                raise RuntimeError('400 Bad Request: service "{}" does not exist'.format(name))
+            current = self._service_status.get(name, pebble.ServiceStatus.INACTIVE)
+            if current == pebble.ServiceStatus.ACTIVE:
+                # TODO: jam 2021-04-20 I believe pebble actually validates all the service names
+                #  can be started before starting any, and gives a list of things that couldn't
+                #  be done, but this is good enough for now
+                raise pebble.ChangeError('''\
+cannot perform the following tasks:
+- Start service "{}" (service "{}" was previously started)
+'''.format(name, name), change=1234)  # the change id is not respected
+        for name in services:
+            # If you try to start a service which is started, you get a ChangeError:
+            # $ PYTHONPATH=. python3 ./test/pebble_cli.py start serv
+            # ChangeError: cannot perform the following tasks:
+            # - Start service "serv" (service "serv" was previously started)
+            self._service_status[name] = pebble.ServiceStatus.ACTIVE
+
+    def stop_services(
+            self, services: typing.List[str], timeout: float = 30.0, delay: float = 0.1,
+    ) -> pebble.ChangeID:
+        # handle a common mistake of passing just a name rather than a list of names
+        if isinstance(services, str):
+            raise TypeError('stop_services should take a list of names, not just "{}"'.format(
+                services))
+        # TODO: handle invalid names
+        # Note: jam 2021-04-20 We don't implement ChangeID, but the default caller of this is
+        # Container.stop() which currently ignores the return value
+        known_services = self._render_services()
+        for name in services:
+            if name not in known_services:
+                # TODO: jam 2021-04-20 This needs a better error type
+                #  400 Bad Request: service "bal" does not exist
+                raise RuntimeError('400 Bad Request: service "{}" does not exist'.format(name))
+            current = self._service_status.get(name, pebble.ServiceStatus.INACTIVE)
+            if current != pebble.ServiceStatus.ACTIVE:
+                # TODO: jam 2021-04-20 I believe pebble actually validates all the service names
+                #  can be started before starting any, and gives a list of things that couldn't
+                #  be done, but this is good enough for now
+                raise pebble.ChangeError('''\
+ChangeError: cannot perform the following tasks:
+- Stop service "{}" (service "{}" is not active)
+'''.format(name, name), change=1234)  # the change id is not respected
+        for name in services:
+            self._service_status[name] = pebble.ServiceStatus.INACTIVE
+
+    def wait_change(
+            self, change_id: pebble.ChangeID, timeout: float = 30.0, delay: float = 0.1,
+    ) -> pebble.Change:
+        raise NotImplementedError(self.wait_change)
+
+    def add_layer(
+            self, label: str, layer: typing.Union[str, dict, pebble.Layer], *,
+            combine: bool = False):
+        # I wish we could combine some of this helpful object corralling with the actual backend,
+        # rather than having to re-implement it. Maybe we could subclass
+        if not isinstance(label, str):
+            raise TypeError('label must be a str, not {}'.format(type(label).__name__))
+
+        if isinstance(layer, (str, dict)):
+            layer_obj = pebble.Layer(layer)
+        elif isinstance(layer, pebble.Layer):
+            layer_obj = layer
+        else:
+            raise TypeError('layer must be str, dict, or pebble.Layer, not {}'.format(
+                type(layer).__name__))
+        if label in self._layers:
+            # TODO: jam 2021-04-19 These should not be RuntimeErrors but should be proper error
+            #  types. https://github.com/canonical/operator/issues/514
+            if not combine:
+                raise RuntimeError('400 Bad Request: layer "{}" already exists'.format(label))
+            layer = self._layers[label]
+            for name, service in layer_obj.services.items():
+                # 'override' is actually single quoted in the real error, but
+                # it shouldn't be, hopefully that gets cleaned up.
+                if not service.override:
+                    raise RuntimeError('500 Internal Server Error: layer "{}" must define'
+                                       '"override" for service "{}"'.format(label, name))
+                if service.override not in ('merge', 'replace'):
+                    raise RuntimeError('500 Internal Server Error: layer "{}" has invalid '
+                                       '"override" value on service "{}"'.format(label, name))
+                if service.override != 'replace':
+                    raise RuntimeError(
+                        'override: "{}" unsupported for layer "{}" service "{}"'.format(
+                            service.override, label, name))
+                layer.services[name] = service
+        else:
+            self._layers[label] = layer_obj
+
+    def _render_services(self) -> typing.Mapping[str, pebble.Service]:
+        services = {}
+        for key in sorted(self._layers.keys()):
+            layer = self._layers[key]
+            for name, service in layer.services.items():
+                # TODO: (jam) 2021-04-07 have a way to merge existing services
+                services[name] = service
+        return services
+
+    def get_plan(self) -> pebble.Plan:
+        plan = pebble.Plan('{}')
+        services = self._render_services()
+        if not services:
+            return plan
+        for name in sorted(services.keys()):
+            plan.services[name] = services[name]
+        return plan
+
+    def get_services(self, names: typing.List[str] = None) -> typing.List[pebble.ServiceInfo]:
+        if isinstance(names, str):
+            raise TypeError('start_services should take a list of names, not just "{}"'.format(
+                names))
+        services = self._render_services()
+        infos = []
+        if names is None:
+            names = sorted(services.keys())
+        for name in sorted(names):
+            try:
+                service = services[name]
+            except KeyError:
+                # in pebble, it just returns "nothing matched" if there are 0 matches,
+                # but it ignores services it doesn't recognize
+                continue
+            status = self._service_status.get(name, pebble.ServiceStatus.INACTIVE)
+            if service.startup == '':
+                startup = pebble.ServiceStartup.DISABLED
+            else:
+                startup = pebble.ServiceStartup(service.startup)
+            info = pebble.ServiceInfo(name,
+                                      startup=startup,
+                                      current=pebble.ServiceStatus(status))
+            infos.append(info)
+        return infos
+
+    def pull(self, path: str, *, encoding: str = 'utf-8') -> typing.Union[typing.BinaryIO,
+                                                                          typing.TextIO]:
+        raise NotImplementedError(self.pull)
+
+    def push(
+            self, path: str, source: typing.Union[bytes, str, typing.BinaryIO, typing.TextIO], *,
+            encoding: str = 'utf-8', make_dirs: bool = False, permissions: int = None,
+            user_id: int = None, user: str = None, group_id: int = None, group: str = None):
+        raise NotImplementedError(self.push)
+
+    def list_files(self, path: str, *, pattern: str = None,
+                   itself: bool = False) -> typing.List[pebble.FileInfo]:
+        raise NotImplementedError(self.list_files)
+
+    def make_dir(
+            self, path: str, *, make_parents: bool = False, permissions: int = None,
+            user_id: int = None, user: str = None, group_id: int = None, group: str = None):
+        raise NotImplementedError(self.make_dir)
+
+    def remove_path(self, path: str, *, recursive: bool = False):
+        raise NotImplementedError(self.remove_path)
diff --git a/test/test_model.py b/test/test_model.py
index b1980dd..ed64add 100755
--- a/test/test_model.py
+++ b/test/test_model.py
@@ -27,6 +27,8 @@ import ops.pebble
 import ops.testing
 from ops.charm import RelationMeta, RelationRole
 
+from ops._private import yaml
+
 from test.test_helpers import fake_script, fake_script_calls
 
 
@@ -870,7 +872,7 @@ containers:
         plan = self.container.get_plan()
         self.assertEqual(self.pebble.requests, [('get_plan',)])
         self.assertIsInstance(plan, ops.pebble.Plan)
-        self.assertEqual(plan.to_yaml(), plan_yaml)
+        self.assertEqual(plan.to_yaml(), yaml.safe_dump(yaml.safe_load(plan_yaml)))
 
     @staticmethod
     def _make_service(name, startup, current):
@@ -1211,7 +1213,7 @@ class TestModelBindings(unittest.TestCase):
                     'interface-name': '',
                     'addresses': [
                         {
-                            'hostname':  '',
+                            'hostname': '',
                             'value': '10.1.89.35',
                             'cidr': ''
                         }
diff --git a/test/test_pebble.py b/test/test_pebble.py
index 48ac44b..ad6c93c 100644
--- a/test/test_pebble.py
+++ b/test/test_pebble.py
@@ -25,6 +25,7 @@ import sys
 
 import ops.pebble as pebble
 import test.fake_pebble as fake_pebble
+from ops._private import yaml
 
 
 # Ensure unittest diffs don't get truncated like "[17 chars]"
@@ -441,14 +442,22 @@ class TestPlan(unittest.TestCase):
             plan.services = {}
 
     def test_yaml(self):
+        # Starting with nothing, we get the empty result
         plan = pebble.Plan('')
-        self.assertEqual(plan.to_yaml(), '')
-        self.assertEqual(str(plan), '')
+        self.assertEqual(plan.to_yaml(), '{}\n')
+        self.assertEqual(str(plan), '{}\n')
 
-        yaml = 'services:\n foo:\n  override: replace\n  command: echo foo'
-        plan = pebble.Plan(yaml)
-        self.assertEqual(plan.to_yaml(), yaml)
-        self.assertEqual(str(plan), yaml)
+        # With a service, we return validated yaml content.
+        raw = '''\
+services:
+ foo:
+  override: replace
+  command: echo foo
+'''
+        plan = pebble.Plan(raw)
+        reformed = yaml.safe_dump(yaml.safe_load(raw))
+        self.assertEqual(plan.to_yaml(), reformed)
+        self.assertEqual(str(plan), reformed)
 
 
 class TestLayer(unittest.TestCase):
diff --git a/test/test_testing.py b/test/test_testing.py
index be87ed7..dae2844 100644
--- a/test/test_testing.py
+++ b/test/test_testing.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import importlib
+import inspect
 import pathlib
 import shutil
 import sys
@@ -21,6 +22,7 @@ import textwrap
 import unittest
 import yaml
 
+from ops import pebble
 from ops.charm import (
     CharmBase,
     RelationEvent,
@@ -34,8 +36,12 @@ from ops.model import (
     UnknownStatus,
     ModelError,
     RelationNotFoundError,
+    _ModelBackend,
+)
+from ops.testing import (
+    Harness,
+    _TestingPebbleClient,
 )
-from ops.testing import Harness
 
 
 class TestHarness(unittest.TestCase):
@@ -1375,6 +1381,60 @@ class TestHarness(unittest.TestCase):
             b_first = [a_first[2], a_first[3], a_first[0], a_first[1]]
             self.assertEqual(changes, b_first)
 
+    def test_get_pebble_container_plan(self):
+        harness = Harness(CharmBase, meta='''
+            name: test-app
+            containers:
+              foo:
+                resource: foo-image
+            ''')
+        self.addCleanup(harness.cleanup)
+        harness.begin()
+        initial_plan = harness.get_container_pebble_plan('foo')
+        self.assertEqual(initial_plan.to_yaml(), '{}\n')
+        container = harness.model.unit.get_container('foo')
+        container.pebble.add_layer('test-ab', '''\
+summary: test-layer
+description: a layer that we can use for testing
+services:
+  a:
+    command: /bin/echo hello from a
+  b:
+    command: /bin/echo hello from b
+''')
+        container.pebble.add_layer('test-c', '''\
+summary: test-for-c
+services:
+  c:
+    command: /bin/echo hello from c
+''')
+        plan = container.pebble.get_plan()
+        self.assertEqual(plan.to_yaml(), '''\
+services:
+  a:
+    command: /bin/echo hello from a
+  b:
+    command: /bin/echo hello from b
+  c:
+    command: /bin/echo hello from c
+''')
+        harness_plan = harness.get_container_pebble_plan('foo')
+        self.assertEqual(harness_plan.to_yaml(), plan.to_yaml())
+
+    def test_get_pebble_container_plan_unknown(self):
+        harness = Harness(CharmBase, meta='''
+            name: test-app
+            containers:
+              foo:
+                resource: foo-image
+            ''')
+        self.addCleanup(harness.cleanup)
+        harness.begin()
+        with self.assertRaises(KeyError):
+            harness.get_container_pebble_plan('unknown')
+        plan = harness.get_container_pebble_plan('foo')
+        self.assertEqual(plan.to_yaml(), "{}\n")
+
 
 class DBRelationChangedHelper(Object):
     def __init__(self, parent, key):
@@ -1496,8 +1556,30 @@ class RelationEventCharm(RecordingCharm):
                  data=dict(app=app_name, unit=unit_name, relation_id=event.relation.id)))
 
 
+def get_public_methods(obj):
+    """Get the public attributes of obj to compare to another object."""
+    public = set()
+    members = inspect.getmembers(obj)
+    for name, member in members:
+        if name.startswith('_'):
+            continue
+        if inspect.isfunction(member) or inspect.ismethod(member):
+            public.add(name)
+    return public
+
+
 class TestTestingModelBackend(unittest.TestCase):
 
+    def test_conforms_to_model_backend(self):
+        harness = Harness(CharmBase, meta='''
+            name: app
+            ''')
+        self.addCleanup(harness.cleanup)
+        backend = harness._backend
+        mb_methods = get_public_methods(_ModelBackend)
+        backend_methods = get_public_methods(backend)
+        self.assertEqual(mb_methods, backend_methods)
+
     def test_status_set_get_unit(self):
         harness = Harness(CharmBase, meta='''
             name: app
@@ -1609,3 +1691,524 @@ class TestTestingModelBackend(unittest.TestCase):
         self.assertEqual(backend.relation_remote_app_name(rel_id), 'postgresql')
 
         self.assertIs(backend.relation_remote_app_name(7), None)
+
+    def test_get_pebble_methods(self):
+        harness = Harness(CharmBase, meta='''
+            name: test-app
+            ''')
+        self.addCleanup(harness.cleanup)
+        backend = harness._backend
+
+        client = backend.get_pebble('/custom/socket/path')
+        self.assertIsInstance(client, _TestingPebbleClient)
+
+
+class TestTestingPebbleClient(unittest.TestCase):
+
+    def get_testing_client(self):
+        harness = Harness(CharmBase, meta='''
+            name: test-app
+            ''')
+        self.addCleanup(harness.cleanup)
+        backend = harness._backend
+
+        return backend.get_pebble('/custom/socket/path')
+
+    def test_methods_match_pebble_client(self):
+        client = self.get_testing_client()
+        self.assertIsNotNone(client)
+        pebble_client_methods = get_public_methods(pebble.Client)
+        testing_client_methods = get_public_methods(client)
+        self.assertEqual(pebble_client_methods, testing_client_methods)
+
+    def test_add_layer(self):
+        client = self.get_testing_client()
+        plan = client.get_plan()
+        self.assertIsInstance(plan, pebble.Plan)
+        self.assertEqual('{}\n', plan.to_yaml())
+        client.add_layer('foo', pebble.Layer('''\
+summary: Foo
+description: |
+  A longer description about Foo
+services:
+  serv:
+    summary: Serv
+    description: |
+      A description about Serv the amazing service.
+    startup: enabled
+    override: replace
+    command: '/bin/echo hello'
+    environment:
+      KEY: VALUE
+'''))
+        plan = client.get_plan()
+        # The YAML should be normalized
+        self.assertEqual('''\
+services:
+  serv:
+    command: /bin/echo hello
+    description: 'A description about Serv the amazing service.
+
+      '
+    environment:
+      KEY: VALUE
+    override: replace
+    startup: enabled
+    summary: Serv
+''', plan.to_yaml())
+
+    def test_add_layer_not_combined(self):
+        client = self.get_testing_client()
+        plan = client.get_plan()
+        self.assertIsInstance(plan, pebble.Plan)
+        self.assertEqual('{}\n', plan.to_yaml())
+        service = '''\
+summary: Foo
+description: |
+  A longer description about Foo
+services:
+  serv:
+    summary: Serv
+    description: |
+      A description about Serv the amazing service.
+    startup: enabled
+    override: replace
+    command: '/bin/echo hello'
+    environment:
+      KEY: VALUE
+'''
+        client.add_layer('foo', pebble.Layer(service))
+        # TODO: jam 2021-04-19 We should have a clearer error type for this case. The actual
+        #  pebble raises an HTTP exception. See https://github.com/canonical/operator/issues/514
+        #  that this should be cleaned up into a clearer error type, however, they should get an
+        #  error
+        with self.assertRaises(RuntimeError):
+            client.add_layer('foo', pebble.Layer(service))
+
+    def test_add_layer_three_services(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    override: replace
+    command: '/bin/echo foo'
+''')
+        client.add_layer('bar', '''\
+summary: bar
+services:
+  bar:
+    summary: The Great Bar
+    startup: enabled
+    override: replace
+    command: '/bin/echo bar'
+''')
+        client.add_layer('baz', '''\
+summary: baz
+services:
+  baz:
+    summary: Not Bar, but Baz
+    startup: enabled
+    override: replace
+    command: '/bin/echo baz'
+''')
+        plan = client.get_plan()
+        self.maxDiff = 1000
+        # Alphabetical services, and the YAML should be normalized
+        self.assertEqual('''\
+services:
+  bar:
+    command: /bin/echo bar
+    override: replace
+    startup: enabled
+    summary: The Great Bar
+  baz:
+    command: /bin/echo baz
+    override: replace
+    startup: enabled
+    summary: Not Bar, but Baz
+  foo:
+    command: /bin/echo foo
+    override: replace
+    startup: enabled
+    summary: Foo
+''', plan.to_yaml())
+
+    def test_add_layer_combine_no_override(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+command: '/bin/echo foo'
+''')
+        # TODO: jam 2021-04-19 Pebble currently raises a HTTP Error 500 Internal Service Error
+        #  if you don't supply an override directive. That needs to be fixed and this test
+        #  should be updated. https://github.com/canonical/operator/issues/514
+        with self.assertRaises(RuntimeError):
+            client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    command: '/bin/echo foo'
+''', combine=True)
+
+    def test_add_layer_combine_override_replace(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+  foo:
+    summary: Foo
+    command: '/bin/echo foo'
+''')
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    command: '/bin/echo foo new'
+    override: replace
+''', combine=True)
+        self.assertEqual('''\
+services:
+  bar:
+    command: /bin/echo bar
+    summary: Bar
+  foo:
+    command: /bin/echo foo new
+    override: replace
+''', client.get_plan().to_yaml())
+
+    def test_add_layer_combine_override_merge(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+  foo:
+    summary: Foo
+    command: '/bin/echo foo'
+''')
+        # TODO: jam 2021-04-19 override: merge should eventually be supported, but if it isn't
+        #  supported by the framework, we should fail rather than do the wrong thing
+        with self.assertRaises(RuntimeError):
+            client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    command: '/bin/echo foob'
+    override: merge
+''', combine=True)
+
+    def test_add_layer_combine_override_unknown(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+  foo:
+    summary: Foo
+    command: '/bin/echo foo'
+''')
+        with self.assertRaises(RuntimeError):
+            client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    command: '/bin/echo foob'
+    override: blah
+''', combine=True)
+
+    def test_get_services_none(self):
+        client = self.get_testing_client()
+        service_info = client.get_services()
+        self.assertEqual([], service_info)
+
+    def test_get_services_not_started(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        infos = client.get_services()
+        self.assertEqual(len(infos), 2)
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        # Default when not specified is DISABLED
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, bar_info.current)
+        self.assertFalse(bar_info.is_running())
+        foo_info = infos[1]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, foo_info.current)
+        self.assertFalse(foo_info.is_running())
+
+    def test_get_services_autostart(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        client.autostart_services()
+        infos = client.get_services()
+        self.assertEqual(len(infos), 2)
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        # Default when not specified is DISABLED
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, bar_info.current)
+        self.assertFalse(bar_info.is_running())
+        foo_info = infos[1]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.ACTIVE, foo_info.current)
+        self.assertTrue(foo_info.is_running())
+
+    def test_get_services_start_stop(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        client.start_services(['bar'])
+        infos = client.get_services()
+        self.assertEqual(len(infos), 2)
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        # Even though bar defaults to DISABLED, we explicitly started it
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.ACTIVE, bar_info.current)
+        # foo would be started by autostart, but we only called start_services
+        foo_info = infos[1]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, foo_info.current)
+        client.stop_services(['bar'])
+        infos = client.get_services()
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, bar_info.current)
+
+    def test_get_services_bad_request(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        # It is a common mistake to pass just a name vs a list of names, so catch it with a
+        # TypeError
+        with self.assertRaises(TypeError):
+            client.get_services('foo')
+
+    def test_get_services_subset(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        infos = client.get_services(['foo'])
+        self.assertEqual(len(infos), 1)
+        foo_info = infos[0]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, foo_info.current)
+
+    def test_get_services_unknown(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        # This doesn't seem to be an error at the moment.
+        # pebble_cli.py service just returns an empty list
+        # pebble service unknown says "No matching services" (but exits 0)
+        infos = client.get_services(['unknown'])
+        self.assertEqual(infos, [])
+
+    def test_invalid_start_service(self):
+        client = self.get_testing_client()
+        # TODO: jam 2021-04-20 This should become a better error
+        with self.assertRaises(RuntimeError):
+            client.start_services(['unknown'])
+
+    def test_start_service_str(self):
+        # Start service takes a list of names, but it is really easy to accidentally pass just a
+        # name
+        client = self.get_testing_client()
+        with self.assertRaises(TypeError):
+            client.start_services('unknown')
+
+    def test_stop_service_str(self):
+        # Start service takes a list of names, but it is really easy to accidentally pass just a
+        # name
+        client = self.get_testing_client()
+        with self.assertRaises(TypeError):
+            client.stop_services('unknown')
+
+    def test_mixed_start_service(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+''')
+        # TODO: jam 2021-04-20 better error type
+        with self.assertRaises(RuntimeError):
+            client.start_services(['foo', 'unknown'])
+        # foo should not be started
+        infos = client.get_services()
+        self.assertEqual(len(infos), 1)
+        foo_info = infos[0]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, foo_info.current)
+
+    def test_stop_services_unknown(self):
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+''')
+        client.autostart_services()
+        # TODO: jam 2021-04-20 better error type
+        with self.assertRaises(RuntimeError):
+            client.stop_services(['foo', 'unknown'])
+        # foo should still be running
+        infos = client.get_services()
+        self.assertEqual(len(infos), 1)
+        foo_info = infos[0]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.ACTIVE, foo_info.current)
+
+    def test_start_started_service(self):
+        # If you try to start a service which is started, you get a ChangeError:
+        # $ PYTHONPATH=. python3 ./test/pebble_cli.py start serv
+        # ChangeError: cannot perform the following tasks:
+        # - Start service "serv" (service "serv" was previously started)
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        client.autostart_services()
+        # Foo is now started, but Bar is not
+        with self.assertRaises(pebble.ChangeError):
+            client.start_services(['bar', 'foo'])
+        # bar could have been started, but won't be, because foo did not validate
+        infos = client.get_services()
+        self.assertEqual(len(infos), 2)
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        # Default when not specified is DISABLED
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, bar_info.current)
+        foo_info = infos[1]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.ACTIVE, foo_info.current)
+
+    def test_stop_stopped_service(self):
+        # If you try to stop a service which is stop, you get a ChangeError:
+        # $ PYTHONPATH=. python3 ./test/pebble_cli.py stop other serv
+        # ChangeError: cannot perform the following tasks:
+        # - Stop service "other" (service "other" is not active)
+        client = self.get_testing_client()
+        client.add_layer('foo', '''\
+summary: foo
+services:
+  foo:
+    summary: Foo
+    startup: enabled
+    command: '/bin/echo foo'
+  bar:
+    summary: Bar
+    command: '/bin/echo bar'
+''')
+        client.autostart_services()
+        # Foo is now started, but Bar is not
+        with self.assertRaises(pebble.ChangeError):
+            client.stop_services(['foo', 'bar'])
+        # foo could have been stopped, but won't be, because bar did not validate
+        infos = client.get_services()
+        self.assertEqual(len(infos), 2)
+        bar_info = infos[0]
+        self.assertEqual('bar', bar_info.name)
+        # Default when not specified is DISABLED
+        self.assertEqual(pebble.ServiceStartup.DISABLED, bar_info.startup)
+        self.assertEqual(pebble.ServiceStatus.INACTIVE, bar_info.current)
+        foo_info = infos[1]
+        self.assertEqual('foo', foo_info.name)
+        self.assertEqual(pebble.ServiceStartup.ENABLED, foo_info.startup)
+        self.assertEqual(pebble.ServiceStatus.ACTIVE, foo_info.current)