Runtime Fabric on VMs / Bare Metal Installation Prerequisites | MuleSoft Documentation
# Runtime Fabric on VMs / Bare Metal Installation Prerequisites
Before installing Runtime Fabric on VMs / Bare Metal, review the information in this topic to ensure your environment is ready for installation.
## Before You Begin
Most installation issues on Runtime Fabric on VMs / Bare Metal result from improper infrastructure provisioning or configuration.   
---  
Before you install Runtime Fabric on VMs / Bare Metal:
  * Refer to Shared Responsibility for Runtime Fabric on VMs / Bare Metal.
If your infrastructure does not meet the minimum hardware, operating system, and networking requirements, Runtime Fabric cannot operate successfully.
  * Runtime Fabric on VMs / Bare Metal requires tasks to be performed by the following members of your organization:
    * DevOps team to provision and manage the infrastructure
    * Network team to add ports to allowlists and configure proxy settings
    * Security team to review for compliance and obtain security certificates
Working with these teams in your organization ensures a successful installation and avoids performance and density issues.
  * Review the requirements in this topic and ensure that all prerequisites are met before installing Runtime Fabric on VMs / Bare Metal.
The installation process fails if the minimum requirements are not met.


See Anypoint Runtime Fabric on VMs / Bare Metal for a description of development and production configurations.
If needed, contact your MuleSoft representative for assistance.
## General Installation Requirements
Knowledge in the following areas is required to install Runtime Fabric on VMs / Bare Metal:
  * Linux system commands
  * Working in Linux with root access. Installation requires root access (permission to run the `sudo` command) on all servers in your environment
  * If running on Amazon Web Services (AWS), working with Terraform
  * Remote connection mechanisms, such as SSL and SSH


Ensure that you plan for:
  * Creating separate runtime fabrics for production and non-production applications
  * Creating separate runtime fabrics based upon environment network scope, such as DMZ versus private network
  * Ensuring that the runtime fabric default pod and services CIDR block do not interfere with the regions network


## Anypoint Platform Roles and Permissions
To successfully use Runtime Fabric, your Anypoint Platform account must have the following roles and permissions enabled:
  * To deploy and manage applications, you must have the ability to use Anypoint Runtime Manager. To deploy applications, you must also have the Exchange Contributors role enabled for your Anypoint Platform account.
  * To manage permissions for Anypoint Platform users, you must have the ability to use Anypoint Access Management.
  * To use Runtime Fabric, you must have the Organization Administrators role or the Manage Runtime Fabrics permission on the corresponding environments.
  * To delete Runtime Fabric instances, administrators need the Manage Runtime Fabrics permission at the organization level.


## Infrastructure Requirements
Perform the following actions:
  * Disable swap memory.
  * Ensure that chrony (system clocks) status is in sync.
  * Use of certain third-party software packages, such as antivirus or deep packet inspection (DPI) software, may impact certain functionalities of the Runtime Fabric appliance. Refer to How Antivirus and DPI Software Impacts Runtime Fabric Functionality for more information.
  * Either provision the required hardware or have enough quota with your infrastructure provider to create the required hardware:
    * x86 or x64 architecture. Only x86 and x64 architectures are supported.
    * Virtual machines
    * Disks with provisioned IOPS
    * Virtual networks
    * Firewall rules and security groups
    * Load balancers
  * If provisioning hardware with AWS or Azure, you must have the correct permissions to create the required resources.
Dedicated disks are required to ensure optimum performance. The Runtime Fabric on VMs / Bare Metal installer formats and mounts the disks needed during installation.   
---  
The default behavior of the AWS and Azure provisioning scripts creates a set of virtual machines and disks defined by the minimum requirements. It also creates a private network with the required network ports configured. This is optimal when evaluating Runtime Fabric on VMs / Bare Metal before integrating within your primary network. You should consult with your network administrator to determine if the default behavior is compatible. You might need to modify the provisioning scripts to accommodate your organization’s requirements.
  * Understand how to forward logs to your organization’s logging provider.
  * Obtain your organization’s SMTP provider information so that you can configure alerts.
  * Verify the following network items:
    * Outbound connections are routed to a proxy.
    * Outbound access is allowed using websockets (via port 443) to Anypoint Platform.
    * Your organization’s network and security policies, including:
      * The internal network IP address range in CIDR notation
      * The subnet range used for Runtime Fabric in CIDR notation
      * If virtual machines are allowed, public IP addresses
      * How to gain shell access to each VM
  * Based on the function of your organization’s Mule applications, determine if inbound traffic should be enabled for Runtime Fabric on VMs / Bare Metal.
If you plan to enable inbound traffic, evaluate your network’s traffic to determine whether you need to configure dedicated nodes for the inbound load balancer. For additional information, refer to Network Configuration and Internal Load Balancer.


## Operating System Requirements
Use the same operating system for each Runtime Fabric node. If you attempt to install Runtime Fabric on VMs / Bare Metal on different operating system versions or distributions, the installer fails.
Runtime Fabric on VMs / Bare Metal requires one of the following operating systems:
  * Red Hat (RHEL)
    * v7.4 - v7.9
    * v8.0 - v8.3
    * v8.4 (using Runtime Fabric Appliance version 1.1.1636064094-8b70d2d and later)
    * v8.6 (using Runtime Fabric Appliance version 1.1.1660253001-ab8f5e5 and later)
  * CentOS
    * v7.4 - v7.9
CentOS 8 has reached end-of-life. Consider using an alternative operating system.  
---  
  * Ubuntu v18.04 (using Runtime Fabric Appliance version 1.1.1625094374-7058b20 or later)
  * Ubuntu 20.04


There is a known control groups (cgroups) memory leak issue which can lead to pod creation failure or node malfunction in RHEL and CentOS 7. See the Knowledge Base Article for additional information. To avoid this issue, use RHEL or CentOS 8.x in your Runtime Fabric on VMs / Bare Metal installation.  
---  
## Network and Port Requirements
Runtime Fabric on VMs / Bare Metal requires specific network and port configuration for installation and operation.
Runtime Fabric on VMs / Bare Metal does not support cross-regional deployments. Network latency between each node can cause problems with cluster health.  
---  
### Bandwidth Requirements
A high bandwidth network is required, with a minimum of 1 GBps connectivity between each node operating Runtime Fabric on VMs / Bare Metal, and minimum of 100 MBps for outbound connectivity to the Internet.
For an example of the traffic volume between Runtime Fabric and the Anypoint control plane, consider the following:
  * The image for Mule Runtime 4.3.0 is 740 MB.
  * An Runtime Fabric appliance upgrade requires 1.5 GB. The appliance upgrade process takes approximately 30 minutes for a production size cluster (3 controllers and 3 workers).
There is a known control groups (cgroups) memory leak issue which can lead to pod creation failure or node malfunction in RHEL and CentOS 7. See the Knowledge Base Article for additional information. To avoid this issue, use RHEL or CentOS 8.x in your Runtime Fabric on VMs / Bare Metal installation.  
---  
## Network and Port Requirements
Runtime Fabric on VMs / Bare Metal requires specific network and port configuration for installation and operation.
Runtime Fabric on VMs / Bare Metal does not support cross-regional deployments. Network latency between each node can cause problems with cluster health.  
---  
### Bandwidth Requirements
A high bandwidth network is required, with a minimum of 1 GBps connectivity between each node operating Runtime Fabric on VMs / Bare Metal, and minimum of 100 MBps for outbound connectivity to the Internet.
For an example of the traffic volume between Runtime Fabric and the Anypoint control plane, consider the following:
  * The image for Mule Runtime 4.3.0 is 740 MB.
  * An Runtime Fabric appliance upgrade requires 1.5 GB. The appliance upgrade process takes approximately 30 minutes for a production size cluster (3 controllers and 3 workers).


### Required Port Settings
The following sections list the TCP and UDP network port requirements for Runtime Fabric on VMs / Bare Metal.
#### Ports Used During Installation
The following table lists the ports that must be accessible on all nodes. These ports are used for internal communication between nodes during installation. After completing the installation, you can safely disable these ports.
Table 1. Ports Required for Installation Port | Protocol | Description  
---|---|---  
Bandwidth checker  
61008-61010 | Used during installation  
61022-61024 | Installer agent ports  
#### Port Used by the Persistence Gateway
The Persistent Gateway requires a Postgres-compliant database to store persistent data across Mule application replicas. Ensure that your Kubernetes cluster has access to this database and port. See Persistence Gateway.
#### Network TCP and UDP Ports
The following table lists all ports that must be open, along with the respective protocols used and the request origin and destination:
Table 2. Ports Required for TCP and UDP Port | Layer 4 Protocol | Layer 5 Protocol | Source | Destination | Description  
---|---|---|---|---|---  
All nodes | Internet | Clock synchronization via Network Time Protocol  
API consumers | Controller nodes | Allow inbound API requests to ingress controllers  
AMQP over WebSockets | Controller nodes | Internet | Anypoint Platform management services  
All nodes | Internet | API Manager policy updates, API Analytics Ingestion, and Resource retrieval (application files, container images).  
443 (v1.8.50, or later) | Lumberjack | All nodes | Internet | Anypoint Monitoring, Anypoint Visualizer  
5044 (deprecated) | Lumberjack | All nodes | Internet | Anypoint Monitoring, Anypoint Visualizer **This port and hostname are deprecated in Anypoint Runtime Fabric, version 1.8.50 and later.** If you are using a previous version of Anypoint Runtime Fabric you must add this port and hostname to your allow list. If you are using a newer version, use the port and hostname specified above. This is applicable to endpoints in both the US and EU clouds.  
localhost | localhost | Internal cluster DNS  
2379, 2380, 4001, 7001 | All nodes | Controller nodes | etcd server communications  
3008, 3010, 3011, 3012 | HTTPS / gRPC | All nodes | All nodes | Anypoint Runtime Fabric internal services  
All nodes | Internal network | Anypoint Runtime Fabric internal Ops Center  
3022 - 3025 | All nodes | All nodes | Internal SSH control services  
All nodes | Controller nodes | Anypoint Runtime Fabric internal Ops Center  
All nodes | Controller nodes | Internal container registry  
HTTPS / gRPC | All nodes | All nodes | Anypoint Runtime Fabric internal services  
6443, 8080 | HTTPS / HTTP | All nodes | Controller nodes | Kubernetes API server  
localhost | localhost | Serf RPC for peer-to-peer connectivity  
All nodes | All nodes | Serf (health checking agent) / Peer-to-peer connectivity  
All nodes | All nodes | Cluster status API  
10248-10250 | All nodes | All nodes | Kubernetes components  
All nodes | All nodes | Kubernetes components  
30000-32767 | Application-specific | All nodes | All nodes | Internal services port range used to route requests to applications  
Internal network | Controller nodes | Anypoint Runtime Fabric Ops Center  
Controller nodes | All nodes | Ops Center internal API  
localhost | localhost | Internal cluster DNS  
All nodes | All nodes | Serf (health checking agent) / Peer-to-peer connectivity  
All nodes | All nodes | Overlay network  
30000-32767 | Application-specific | All nodes | All nodes | Internal services port range used to route requests to applications  
### Port IP Addresses and Hostnames to Add to the Allowlist
In your network configuration, you must add the hostnames and ports of Anypoint Platform components and services to allowlists to enable Anypoint Runtime Fabric to communicate with them. This is also required to download dependencies during installation and upgrades.
The following table lists the ports and hostnames to add to your allowlists to allow communication between Anypoint Runtime Fabric and Anypoint Platform.
The following table does not include ports and hostnames from API Manager that you may also need to allow. Refer to the API Manager documentation for a list of additional ports and hostnames.  
---  
Because the following endpoints use mutual TLS authentication to establish the connection, you must configure SSL passthrough to allow the following certificate:
  * US control plane
    * `transport-layer.prod.cloudhub.io`
    * `configuration-resolver.prod.cloudhub.io`
  * EU control plane
    * `transport-layer.prod-eu.msap.io`
    * `configuration-resolver.prod-eu.msap.io`


Port | Protocol | Hostnames | Description  
---|---|---|---  
AMQP over WebSockets | 
  * US control plane: `transport-layer.prod.cloudhub.io`
  * EU control plane: `transport-layer.prod-eu.msap.io`

| Runtime Fabric message broker for interaction with the control plane.  
443 (v1.8.50, or later) | TCP (Lumberjack) | 
  * US control plane: `dias-ingestor-router.us-east-1.prod.cloudhub.io`
  * EU control plane: `dias-ingestor-router.eu-central-1.prod-eu.msap.io`

| Anypoint Monitoring agent for Runtime Fabric.  
`anypoint.mulesoft.com` | Anypoint Platform for pulling assets.  
`kubernetes-charts.storage.googleapis.com` | Kubernetes base charts repository.  
`docker-images-prod.s3.amazonaws.com` | Kubernetes base charts repository.  
  * US control plane: `worker-cloud-helm-prod.s3.amazonaws.com`
  * EU control plane: `worker-cloud-helm-prod-eu-rt.s3.amazonaws.com` `worker-cloud-helm-prod-eu-rt.s3.eu-central-1.amazonaws.com`

| Runtime Fabric version repository. The Runtime Fabric installation uses software from this repository during installation and upgrades.  
  * US control plane: `exchange2-asset-manager-kprod.s3.amazonaws.com`
  * EU control plane: `exchange2-asset-manager-kprod-eu.s3.amazonaws.com` `exchange2-asset-manager-kprod-eu.s3.eu-central-1.amazonaws.com`

| Anypoint Exchange for application assets.  
  * US control plane: `exchange-files.anypoint.mulesoft.com`
  * EU control plane: `exchange-files.eu1.anypoint.mulesoft.com`

| Anypoint Exchange for application files.  
  * US control plane: `ecr.us-east-1.amazonaws.com`
  * EU control plane: `ecr.eu-central-1.amazonaws.com`

| Runtime Fabric Docker repository.  
  * US control plane: `rtf-runtime-registry.kprod.msap.io`.
  * EU control plane: `rtf-runtime-registry.kprod-eu.msap.io`

| Runtime Fabric Docker repository.  
  * US control plane: `api.ecr.us-east-1.amazonaws.com`
  * EU control plane: `api.ecr.eu-central-1.amazonaws.com`

| Runtime Fabric Docker repository.  
  * US control plane: `prod-us-east-1-starport-layer-bucket.s3.amazonaws.com` or `prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com`
  * EU control plane: `prod-eu-central-1-starport-layer-bucket.s3.amazonaws.com` or `prod-eu-central-1-starport-layer-bucket.s3.eu-central-1.amazonaws.com`

| Runtime Fabric Docker image delivery.  
  * US control plane: `runtime-fabric.s3.amazonaws.com` .
  * EU control plane: `runtime-fabric-eu.s3.eu-central-1.amazonaws.com`

| Runtime Fabric Docker repository.  
  * US control plane: `configuration-resolver.prod.cloudhub.io`
  * EU control plane: `configuration-resolver.prod-eu.msap.io`
| Runtime Fabric Docker repository.  
  * US control plane: `rtf-runtime-registry.kprod.msap.io`.
  * EU control plane: `rtf-runtime-registry.kprod-eu.msap.io`

| Runtime Fabric Docker repository.  
  * US control plane: `api.ecr.us-east-1.amazonaws.com`
  * EU control plane: `api.ecr.eu-central-1.amazonaws.com`

| Runtime Fabric Docker repository.  
  * US control plane: `prod-us-east-1-starport-layer-bucket.s3.amazonaws.com` or `prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com`
  * EU control plane: `prod-eu-central-1-starport-layer-bucket.s3.amazonaws.com` or `prod-eu-central-1-starport-layer-bucket.s3.eu-central-1.amazonaws.com`

| Runtime Fabric Docker image delivery.  
  * US control plane: `runtime-fabric.s3.amazonaws.com` .
  * EU control plane: `runtime-fabric-eu.s3.eu-central-1.amazonaws.com`

| Runtime Fabric Docker repository.  
  * US control plane: `configuration-resolver.prod.cloudhub.io`
  * EU control plane: `configuration-resolver.prod-eu.msap.io`

| Anypoint Configuration Resolver.  
### Required Network Settings
In addition to the previous port requirements, the following network settings are required to operate Runtime Fabric on VMs / Bare Metal:
  * A subnet with enough IP addresses allotable to add additional VMs to Runtime Fabric on VMs / Bare Metal
The pod CIDR block must not overlap with IP addresses that pods or servers use to communicate. If services within the cluster or services that you installed on nodes need to communicate with an IP range that overlaps the pod or service CIDR block, a conflict can occur. If a CIDR block is in use, but pods and services do not use those IP addresses to communicate, there is no conflict. If you deploy more than one cluster, each cluster can reuse the same IP range, because those addresses exist within the cluster nodes, and cluster-to-cluster communications is relayed on the external interfaces.   
---  
  * Shell/SSH access to each VM used to install Runtime Fabric on VMs / Bare Metal
  * Static private IPv4 addresses assigned to each VM, which persist between restarts
  * Kernel IP forwarding to enable internal Kubernetes load balancing. IPv4 forwarding is enabled on each VM during installation.
  * Bridge-netfilter to enable the host Linux kernel to translate packets to and from hosted containers. This kernel module is enabled on each VM during installation.
  * Access to an SMTP server to manage e-mail alerts triggered by Runtime Fabric on VMs / Bare Metal
  * An external load balancer to balance external requests to the internal load balancer running on each controller VM
  * Your environment’s HTTP proxy configuration if outgoing requests to the Internet must route through a proxy
  * A SOCKS5 proxy for Anypoint Monitoring and Visualizer support if outgoing requests to the Internet must route through a proxy


If you’re using VMWare and are unable to connect to your existing applications or deploy new applications refer to RTF Networking Issues with VMware in the MuleSoft Knowledge Base.   
---  
## Mule License
To deploy Mule applications and API proxies, you must have access to your organization’s Mule license file.
## Installation Requirements
During installation, the installation package is downloaded to a controller VM that serves as the leader of the installation.
Runtime Fabric on VMs / Bare Metal is configured to run as a cluster across multiple virtual machines. Each VM operates as one of the following roles:
  * Controller VM, which is dedicated to operate and run Runtime Fabric services. The internal load balancer runs within controller VMs.
  * Worker VM, which is dedicated to run Mule applications.


One controller VM acts as a leader during the installation. This VM downloads the installer and makes it accessible to each other VM on port 32009. Other VMs copy the installer files from the leader, perform the installation, and join the leader to form a cluster.
During the installation, a set of pre-flight checks run to verify the minimum hardware, operating system, and network requirements for Runtime Fabric. If these requirements are not met, the installer fails.
The installation process combines the following steps:
  * For AWS and Azure, provisions infrastructure per the system requirements
  * Installs Runtime Fabric across the VMs
  * Activates Runtime Fabric with your Anypoint organization
  * Installs your organization’s Mule Enterprise license


To complete the previous steps, specify environment variables for each VM at the beginning of installation. The leader requires additional variables to activate and add the Mule license. A script runs on each VM to perform the following actions:
  * Format and mount each dedicated disk.
  * Add mount entries for each disk to `/etc/fstab`.
  * Add iptable rules.
  * Enable required kernel modules.
  * Start the installation.


The controller VM that is the leader for installation performs the following actions:
  * Runs the activation script after installation.
  * Runs the Mule license insertion script after registration.


## Production Configuration Requirements
A minimum of 6 nodes are required for a production configuration of Runtime Fabric on VMs/ Bare Metal, as shown in the following diagram.
The following sections describe the minimum system requirements of controller and worker nodes in a production configuration.
Runtime Fabric on VMs / Bare Metal does not support running controller nodes using burstable VMs. When deploying an application on a worker node, you can allocate vCPU for bursting. See Resource Allocation and Performance on Anypoint Runtime Fabric for additional information.   
---  
### Controller Nodes
The minimum number of controller nodes required for a production environment is three. This requirement reflects the minimum requirements to maintain high-availability and ensure system performance.
An odd number of controller nodes is required for fault tolerance. Even numbers of controller nodes are not supported.
The maximum number of supported controller nodes is 5.
Each controller node must have the following resources:
  * A minimum of 2 dedicated cores.
  * A minimum of 8 GiB memory.
You must consider the amount of resources needed for the internal load balancer when sizing controller nodes. For additional information, refer to Internal Load Balancer resource allocation.
  * A minimum of 80 GiB dedicated disk for the operating system.
    * A minimum of 20 GiB for `/tmp` directory. The `/tmp` directory needs read, write, and executable permissions.
    * A minimum of 8 GiB for `/opt/anypoint/runtimefabric` directory.
    * A minimum of 1 GiB for `/var/log/` directory.
  * A minimum of 60 GiB dedicated disk with at least 3000 provisioned IOPS to run the etcd distributed database. This translates to a minimum of 12 Megabytes per second (MBps) disk read/write performance.
    * This disk is referred to as the `etcd` device.
  * A minimum of 250 GiB dedicated disk with 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance.
    * This disk is referred to as the `docker` device.


#### Scaling Considerations
Consider scaling the number of controller nodes in the following situations:
  * Fault tolerance is needed to mitigate the impact of controller node hardware failure. The minimum requirement of 3 controller nodes enables fault-tolerance of losing 1 controller. To improve fault-tolerance to lose 2 controllers, a total of 5 controllers should be configured.
  * Production traffic is being served with applications running on Runtime Fabric on VMs / Bare Metal.


### Worker Nodes
Runtime Fabric on VMs / Bare Metal requires at least three worker nodes to run Mule applications and API gateways.
The maximum number of worker nodes supported is 16.
Provision at least one extra worker node to preserve uptime in the event a node becomes unavailable (fault-tolerance) or during maintenance operations that require restarts, such as OS patching.
This enables the safe removal of a worker node from the cluster to apply upgrades, without impacting the availability of applications.
Runtime Fabric can reserve up to 0.5 cores per worker node for internal services. This is important when determining the amount of CPU cores and worker nodes needed. For example, your Runtime Fabric could have three worker nodes and two CPU cores for each worker node. If Runtime Fabric reserves 0.3 cores per worker node, a total of 0.9 cores, the number of available cores displayed in Runtime Manager is 5.1 vCPU instead of 6 vCPU.  
---  
Each worker node must have the following resources:
  * A minimum of 2 dedicated cores.
  * A minimum of 15 GiB memory.
You must consider how many Mules and tokenizers you plan to run on Runtime Fabric, and what they are licensed to deploy. Plan on approximately 0.5 cores per worker node for overhead.
  * A minimum of 80 GiB dedicated disk for the operating system.
    * A minimum of 20 GiB for `/tmp` directory. The `/tmp` directory needs read, write, and executable permissions.
    * A minimum of 8 GiB for `/opt/anypoint/runtimefabric` directory.
    * A minimum of 1 GiB for `/var/log/` directory.
  * A minimum of 250 GiB dedicated disk with at least 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance.
Having 250 GiB ensures there is enough space to run applications, cache docker images, provide temporary storage for running applications, and provide log storage.
### Additional Resource Requirements for Production Environments
When using Runtime Fabric on VMs / Bare Metal in a production environment:
  * An external load balancer must be configured to load balance external requests to the internal load balancer running on each controller VM, with a health check configured for TCP port 443. A TCP load balancer with a server pool consisting of each of the controller nodes is sufficient.
  * An internal load balancer that runs on at least 3 replicas to maintain availability.
  * Applications serving inbound requests must be scaled to a minimum of 2 replicas.


## Development Configuration Requirements
Runtime Fabric on VMs / Bare Metal requires a minimum of three nodes in a development environment as shown in the following diagram:
Runtime Fabric on VMs / Bare Metal does not support running controller nodes using burstable VMs. When deploying an application on a worker node, you can allocate vCPU for bursting. see Resource Allocation and Performance on Anypoint Runtime Fabric for additional information.   
---  
The following sections describe the minimum system requirements of each controller and worker nodes in a development configuration.
### Controller Nodes
The minimum number of controller nodes required for a non-production environment is one. Even numbers of controllers are not supported. Keep in mind that single-controller clusters won’t be recoverable in the event of corruption of the etcd database. For example the issue described in this knowledge article.
The maximum number of supported controller nodes is 5.
Each controller node must have the following resources:
  * A minimum of 2 dedicated cores.
  * A minimum of 8 GiB memory.
  * A minimum of 80 GiB dedicated disk for the operating system.
    * A minimum of 20 GiB for `/tmp` directory. The `/tmp` directory needs read, write, and executable permissions.
    * A minimum of 8 GiB for `/opt/anypoint/runtimefabric` directory.
    * A minimum of 1 GiB for `/var/log/` directory.
  * A minimum of 60 GiB dedicated disk with 3000 provisioned IOPS for etcd. This translates to a minimum of 12 Megabytes per second (MBps) disk read/write performance.
  * A minimum of 100 GiB dedicated disk with 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance.


### Worker Nodes
Runtime Fabric on VMs / Bare Metal requires at least two worker nodes to run Mule applications and API gateways. The maximum number of workers nodes supported is 16.
Provision at least one extra worker node to preserve uptime in the event a node becomes unavailable (fault-tolerance) or during maintenance operations that require restarts, such as OS patching.
This enables safe removal of a worker node from the cluster to apply upgrades, without impacting availability of applications.
Runtime Fabric can reserve up to 0.5 cores per worker node for internal services. This is important when determining the amount of CPU cores and worker nodes needed. For example, your Runtime Fabric could have three worker nodes and two CPU cores for each worker node. If Runtime Fabric reserves 0.3 cores per worker node, a total of 0.9 cores, the number of available cores displayed in Runtime Manager is 5.1 vCPU instead of 6 vCPU.  
---  
Each worker node must have the following resources:
  * A minimum of 2 dedicated cores.
  * A minimum of 15 GiB memory.
  * A minimum of 80 GiB dedicated disk for the operating system.
    * A minimum of 20 GiB for `/tmp` directory. The `/tmp` directory needs read, write, and executable permissions.
    * A minimum of 8 GiB for `/opt/anypoint/runtimefabric` directory.
    * A minimum of 1 GiB for `/var/log/` directory.
  * A minimum of 100 GiB dedicated disk with at least 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance.
Having 100 GiB ensures there is enough space to run applications, cache docker images, provide temporary storage for running applications, and provide log storage.