Apache Mesos. What’s That?
By: Daniel Gospodinow
Tags: Distributed Systems, Programming, Apache Mesos, Servers, Cluster Management

Apache Mesos

Overview

Apache Mesos is an open-source, distributed, highly available, and fault-tolerant cluster manager developed originally at the University of California, Berkeley.

Similar to Kubernetes, Apache Mesos provides an easy way to develop distributed applications.

Mesos provides CPU, memory, and disk resource abstractions for a given set of servers inside a data center. It can easily make data centers behave as if they were a single monstrous machine.

Mesos is an abstraction layer that abstracts both virtual and physical machines.

The basic idea of Mesos is to act as a facade in front of a group of machines. Such a group of machines that are managed by Mesos is called a Mesos cluster. So instead of reaching out to a particular server to deploy a workload, you can directly reach out to Mesos, and it will manage your workload for you.

Different distributed systems and applications — for example, Spark and Jenkins — can easily consume cluster resources through the interaction with Mesos.

As a note, in Mesos distributed applications are also called frameworks.

Simple Mesos conceptual diagram by the Mesos in Action book

Concepts

Mesos is built around concepts referred to as resource offers, two-tier scheduling, and resource isolation.

Resource offers are descriptions of the currently available CPU, memory, and storage resources of a given machine inside the Mesos cluster.

Two-tier scheduling is a scheduling mechanism that consists of scheduling on two different levels. The first level of scheduling is done by the Mesos controller’s allocation module, or scheduling algorithm, which provides resources to a given framework. Resources are offered to frameworks in the form of a percentage of the cluster’s available resources. The second level of scheduling is done by a framework’s internal scheduler, which schedules given user tasks. This concept is similar to the concept of green threads. Internal schedulers are implemented by the frameworks themselves.

The concept of green threads in Golang by Osamh Aloqaily

Very conveniently, the allocation module is a pluggable component. It can be replaced with a custom resource-sharing algorithm that better suits a specific scenario. The default scheduling algorithm of the allocation module is the Dominant Resource Fairness algorithm which aims to maximize the minimum dominant share across all users. It suits well most use cases.

“In a nutshell, DRF seeks to maximize the minimum dominant share across all users. For example, if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, DRF attempts to equalize user A’s share of CPUs with user B’s share of memory. In the single-resource case, DRF reduces to max-min fairness for that resource.” — Mesos in Action by Roger Ignazio

Resource isolation is the mechanism that provides virtual spaces to processes that protect them against malicious escape and breakout attempts into other processes on the same host or on the shared infrastructure. Using such isolation techniques, the physical infrastructure can ensure that multiple processes can operate on the same infrastructure without any negative impact on each other. This allows multiple applications to run alongside each other on a single machine. Mesos provides a way for the applications to run in an isolated environment by using either Docker containers or Linux cgroups.

Those characteristics make Mesos act exactly like an operating system — an operating system that’s designed to manage clusters of machines. Just as operating systems provide an isolated virtual space for processes to run in, Mesos provides isolation to its applications. Furthermore, just as an operating system has a scheduler that gives processes their necessary CPU usage time, Mesos has a scheduling algorithm that provides appropriate cluster resources to running applications. Lastly, just as an operating system sets up a common infrastructure and common APIs for processes to interact with, Mesos provides a common set of APIs that applications can utilize.

Architecture

Like most other cluster managers, Mesos is composed of controller and agent machines.

Agent machines are machines that are running a Mesos agent daemon. This agent daemon periodically measures the resources of the machine on which they’re currently running and sends those measurements to a Mesos controller in the form of a resource offer. Because of this, the Mesos controller knows exactly how many available resources the cluster has available. Later on, those available resources are given to different applications that need to consume cluster resources.

Controller machines are machines that run the allocation module. This component collects resource offerings from agent machines and decides which cluster resources go to which framework. When there are multiple Mesos controllers, there’s a process of leader election that promotes one of the controllers as a leader while all other controllers are marked as followers. Only the leader Mesos controller operates, while the followers are placed in a standby mode. If the leader Mesos controller fails and terminates for some reason, one of the followers becomes the leader and continues to operate in place of the failed leader. This illustrates Mesos’ high availability mode.

Frameworks, on the other hand, are the actual distributed applications that consume cluster resources. They are responsible for scheduling and executing tasks, and thus they’re made up of two components: a scheduler and an executor. The scheduler is responsible for accepting or rejecting resource offers. The resource offers are accepted or rejected based on the number and the size of the tasks that the framework has to perform. An executor, on the other hand, is a process that’s launched on a Mesos agent that is responsible for running the framework’s tasks on Mesos agents.

Mesos in action

Two-tier scheduling diagram by the Mesos in Action book

In the figure above, we can see illustrated all of the mentioned concepts: resource offers, two-tier scheduling, resource isolation, agent and controller machines, and frameworks.

Step one — First off, we see that the agent machine sends a resource offer to the controller machine.

Step two — Afterward, the controller machine’s resource allocation module selects a framework to which to offer those resources. The offered resources can either be accepted or rejected based on whether the selected framework has some jobs to execute or not.

Step three — Let’s say that the chosen framework doesn’t have any jobs to execute; therefore, the resource offer is rejected. After a resource offer is rejected, the Mesos controller allocation module offers those resources to another framework.

Step four — Now let’s say that the framework receives a job that has to be executed. In that case, the framework waits for a resource offer from the Mesos controller that satisfies the workload.

Step five — Upon receiving a suitable resource offer, the framework’s scheduler accepts the resource offer and schedules one or more tasks on an available Mesos agent. On the Mesos agent, these tasks run within a containerized environment to provide isolation between all other running tasks there.

When to use Mesos

The requirement for running at a large scale is mandatory for virtually every software company nowadays.

A Mesos cluster can help scale and run a data center more efficiently.

Mesos is best for large systems, and it is designed for maximum redundancy.

Mesos is recommended if you have existing workloads such as Hadoop, Kafka, etc. It gives you a framework that lets you interleave those workloads with each other.

It is a very stable platform, but it’s overly complex for small-scale systems under ten to twenty machines. That’s why Mesos is usually recommended and is best for companies that use multi-cloud and multi-region clusters, such as Airbnb, Netflix, Twitter, or Uber.

Preferably, Mesos should run only stateless applications that can easily shut down or boot up without a negative impact.