Project: Juju (Provisioning / Automation & Configuration)
File: architectural-overview.md

This document is targeted at new developers of Juju, and may be useful to experienced developers who need a refresher on some aspect of Juju's operation. It is deliberately light on detail, because the precise mechanisms of various components' operation are expected to change much faster than the general interactions between components. A Juju model is a distributed system comprising: A data store (mongodb) which describes the desired state of the world, in terms of running workloads or applications, and the relations between them; and of the units that comprise those applications, and the machines on which those units run. A bunch of agents*, each of which runs the same `jujud` binary, and which are variously responsible for causing reality to converge towards the idealised world- state encoded in the data store. Some number of clients* which talk over an API, implemented by the agents, to update the desired world-state (and thereby cause the agents to update the world to match). The `juju` binary is one of many possible clients; the `juju-dashboard` web application, and the `juju-deployer` python tool, are other examples. The whole system depends upon a substrate, or provider, which supplies the compute, storage, and network resources used by the workloads (and by Juju itself; but never forget that everything described in this document is merely supporting infrastructure geared towards the successful deployment and configuration of the workloads that solve actual problems for actual users). Here's the various high level parts of Juju system and how they interact: ``` +--+ ++ | | | | | Machine agent | | Unit agent | | +-+ | | +-+ | | | | | | | | | | | workers | | | | workers | | | | | | | | | | | +--+-+ | | +-+--+ | | | | | | | +--+ ++ | | | Juju API | | +-+ | | | | +--+ | | | | | Controller agent | | | ++ | +v-v-+ +-+ | | | | | | Juju API | | | | Client +--> apiserver +<--+ workers | | | | | Juju API | | | | | ++ | +++ +++ | | | | | | | | | | +--v--+ +v+ | | | | | | | | | state | | providers | | | | | | | | | +--+--+ +++ | | | | | +--+ | MongoDB protocol | cloud API | | +--v--+ +V+ | | | | | MongoDB | | cloud/substrate | | | | | +--+ +-+ ``` At the centre is a controller agent. It is responsible for maintaining the state for one or more Juju models and runs a server which provides the Juju API. Juju's state is kept in MongoDB. Juju's MongoDB may only be accessed by the controller agents. A controller agent runs a number of workers, many of which are specific to controller

Some workers in the controller agent use the Juju provider implementation to communicate with the underlying cloud substrate using the substrate's APIs. This is how cloud resources are created, managed and destroyed. Almost all workers will interact with Juju's state using Juju's API, even workers running within a controller agent. If a Juju deployment has high-availability enabled there will be multiple controller agents. An consumer of the Juju API may connect to any controller agent. In HA mode, there will be a MongoDB instance on each controller machine, with a MongoDB replicaset configured to synchronise data between the nodes. Each Juju deployed machine runs a machine agent. Each machine agent runs a number of workers. A controller agent is a machine agent with extra responsibilities. It runs all the workers which a normal machine runs as well as controller specific workers. A unit agent runs for each deployed unit of an application. It is mainly responsible for installing, running and maintaining charm code. It runs a different set of workers to a machine agent. There are a number of clients which interact with Juju using the Juju API. These include the `juju` command line tool and Juju Dashboard. There's a lot of detail to cover, but there's not much to say from an architectural standpoint. We use a mongodb replicaset to support HA; we use the `mgo` package from `labix.org` to implement multi-document transactions; we make use of the transaction log to detect changes to particular documents, and convert them into business-object- level events that get sent over the API to interested parties. The mongodb databases run on machines we refer to as controllers, and are only accessed by agents running on those machines; it's important to keep it locked down (and, honestly, to lock it down further and better than we currently have). There's some documentation on how to work with ; and plenty more on the and the details of their and from various perspectives; but there's not a lot more to say in this context. It is important to understand that the transaction-log watching is not an ideal solution, and we'll be retiring it at some point, in favour of an in-memory model of state and a pub-sub system for watchers; we know it's a scalability problem, but we're not devoting resources to it until it becomes more pressing. Code for dealing with mongodb is found primarily in the `state`, `state/watcher`, `replicaset`, and `worker/peergrouper` packages. Juju controllers expose an API endpoint over a websocket connection. The methods available over the API are broken down by client; there's a `Client` facade that exposes the methods used by clients, an `Agent` facade that exposes the methods common to all agents, and a wide range of worker-specific facades that individually deal with particular chunks of functionality implemented by one agent or another (for example, `Provisioner`, `Upgrader`, and `Uniter`, each used by the eponymous worker types). The API server is implemented in the `apiserver` top level package. Each API facade has it's own subpackage (e.g.

The code under `apiserver` is the only code that is allowed to import from the `state` package. Various facades share functionality; for example, the Life method is used by many worker facades. In these cases, the method is implemented on a separate type, which is embedded in the facade implementation. All APIs should be implemented such that they can be called in bulk, but not all of them are. The agent facades are (almost?) all implemented correctly, but the Client facade is almost exclusively not. As functionality evolves, and new versions of the client APIs are implemented, we must take care to implement them consistently -- this means both implementing bulk calls and splitting the monolithic Client facade into smaller application-specific facades, such that we can evolve interaction with (say) users without bumping global API versions across the board). The Juju API client is implemented under the `api` top level package. Client side API facade are implemented as subpackages underneath `api`. Agents all use the same `jujud` binary, and all follow roughly the same model. When starting up, they authenticate with an API server; possibly reset their password, if the one they used has been stored persistently somewhere and is thus vulnerable; determine their responsibilities; and run a set of tasks in parallel until one of those tasks returns an error indicating that the agent should either restart or terminate completely. Tasks that return any other error will be automatically restarted after a short delay; tasks that return nil are considered to be complete, and will not be restarted until the whole process is. When comparing the unit agent with the machine agent, the truth of the above may not be immediately apparent, because the responsibilities of the unit agent are so much less varied than those of the machine agent; but we have scheduled work to integrate the unit agent into the machine agent, rendering each unit agent a single worker task within its responsible machine agent. It's still better to consider a unit agent to be a simplistic and/or degenerate implementation of a machine agent than to attach too much importance to the differences. Machine agents all have at least one of two jobs: JobHostUnits and JobManageModel. Each of these jobs represents a number of tasks the agent needs to execute to fulfil its responsibilities; in addition, there are a number of tasks that are executed by every machine agent. The terms task and worker are generally used interchangeably in this document and in the source code; it's possible but not generally helpful to draw the distinction that a worker executes a task. All tasks are implemented by code in some subpackage of the `worker` package, and the `worker.Runner` type implements the retry behaviour described

It's useful to note that the Runner type is itself a worker, so we can and do nest Runners inside one another; the details of exactly how and where a given worker comes to be executed are generally elided in this document; but it's worth being aware of the fact that all the workers that use an API connection share a single one, mediated by a single Runner, such that when the API connection fails that single Runner can stop all its workers; shut itself down; be restarted by its parent worker; and set up a new API connection, which it then uses to start all its child workers. Please note that the lists of workers below should not be assumed to be exhaustive. Juju evolves, and the workers evolve with it. All agents run workers with the following responsibilities: Check for scheduled upgrades for their binaries, and replace themselves (implemented in `worker/upgrader`) Watch logging config, and reconfigure the local logger (`worker/logger`; yes, we know; it is not the stupidest name in the codebase) Watch and store the latest known addresses for the controllers (`worker/apiaddressupdater`) Machine agents additionally do the following: Run upgrade code in the new binaries once they're replaced themselves (implemented directly in the machine agent's `upgradeWorker` method) Handle SIGABRT and permanently stop the agent (`worker/terminationworker`) Handle the machine entity's death and permanently stop the agent (`worker/machiner`) Watch proxy config, and reconfigure the local machine (`worker/machineenvironmentworker`) Watch for contained LXC or KVM machines and provision/decommission them (`worker/provisioner`) All machine agents have JobHostUnits. These run the `worker/deployer` code which watches for units assigned to the machine, and deploys/recalls upstart configs for their respective unit agents as the units are assigned/removed. We expect the deployer implementation to change to just directly run the unit agents' workers in its own Runner. Machines with JobManageModel also run a number of other workers, which do the following. Run the API server used by all other workers (in this, and other, agents: `state/apiserver`) Provision/decommission provider instances in response to the creation/ destruction of machine entities (`worker/provisioner`, just like the container provisioners run in all machine agents anyway) Manipulate provider networks in response to units opening/closing ports, and users exposing/unexposing applications (`worker/firewaller`) Update network addresses and associated information for provider instances (`worker/instancepoller`) Respond to queued DB cleanup events (`worker/cleaner`) Maintain the MongoDB replica set (`worker/peergrouper`) Resume incomplete MongoDB transactions (`worker/resumer`) Many of these workers (more than strictly need to be) are wrapped as "singular" workers, which only run on the same machine as the current MongoDB replicaset master. When the master changes, the state connection is dropped, causing all those workers to also be stopped; when they're restarted, they won't run because they're no longer running on the master. Unit agents run all the common workers, and the `worker/uniter` task as well; this task is probably the single most forbiddingly complex part of Juju. (Side note: It's a unit-er because it deals with units, and we're bad at names; but it's also a unite-r because it's where all the various components of Juju come together to run actual workloads.) It's sufficiently large that it deserves its own top-level heading, below. At the highest level, the Uniter is a state machine. After a "little bit" of setup, it runs a tight loop in which it calls `Mode` functions one after another, with the next mode run determined by the result of its

All mode functions are implemented in `worker/uniter/modes.go`, which is actually pretty small: just a hair over 400 lines. It's deliberately implemented as conceptually single-threaded (just like almost everything else in Juju -- rampaging concurrency is the root of much evil, and so we save ourselves a huge number of headaches by hiding concurrency behind event channels and handling a single event at a time), but this property has degraded over time; in particular, the `RunListener` code can inject events at unhelpful times, and while the `hookLock` probably renders this safe it's still deeply suboptimal, because the fact of the concurrency requires that we be extremely careful with further modifications, lest they subtly break assumptions. We hope to address this by retiring the current implementation of `juju run`, but it's not entirely clear how to do this; in the meantime, Here Be Dragons. Leaving these woes aside, the mode functions make use of two fundamental components, which are glommed together until someone refactors it to make more sense. There's the `Filter`, which is responsible for communicating with the API server (and the rest of the outside world) such that relevant events can be delivered to the mode func via channels exposed on the filter; and then there's the `Uniter` itself, which exposes a number of methods that are expected to be called by the mode funcs. XXXX XXXX XXXX A Juju provider represents a different possible kind of substrate on which a Juju model can run, and (as far as possible) abstracts away the differences between them, by making them all conform to the Environ interface. The most important thing to understand about the various providers is that they're all implemented without reference to broader Juju concepts; they are squeezed into a shape that's convenient WRT allowing Juju to make use of them, but if we allow Juju-level concepts to infect the providers we will suffer greatly, because we will open a path by which changes to Juju end up causing changes to all the providers at once. However, we lack the ability to enforce this at present, because the package dependency flows in the wrong direction, thanks primarily (purely?) to the StateInfo method on Environ; and we jam all sorts of gymnastics into the state package to allow us to use Environs without doing so explicitly (see the state.Policy interface, and its many somewhat-inelegant uses). In other places, we have (quite reasonably) moved code out of the environs package (see both environs/config.Config, and instances.Instance). Environ implementations are expected to be goroutine-safe; we don't currently make much use of that property at the moment, but we will be coming to depend upon it as we move to eliminate the wasteful proliferation of Environ instances in the API server. It's important to note that an environ Config will generally contain sensitive information -- a user's authentication keys for a cloud provider -- and so we must always be careful to avoid spreading those around further than we need to. Basically, if an environ config gets off a controller, we've screwed up.