Setting Up Monitoring for Hybrid Deployments | MuleSoft Documentation
# Setting Up Monitoring for Hybrid Deployments
Install the Anypoint Monitoring agent on an on-premises server to monitor applications that are running on that server and managed in the cloud.
The Anypoint Monitoring agent sends monitoring data to the Anypoint Monitoring cloud endpoint. Anypoint Monitoring data is transferred using an open-source protocol for transferring log data over a highly efficient bidirectional binary channel. Data is securely transmitted using mutual TLS encryption.
To use Anypoint Monitoring across a server group or cluster, you must install Anypoint Monitoring on each individual server.
## Before You Begin
Before installing the Anypoint Monitoring agent, set up and run a Mule runtime engine instance on your server. For guidance, see Run Mule Runtime Engine On-Premises.
If you have an outbound firewall, open ports based on the `am.zip` installer version for the Anypoint Monitoring agent:
  * 443 for all installer versions
  * 8443 for 3.x installer versions


Outbound connections are made to the following endpoints:
  * US Control Plane
    * 3.x installers:
      * us1.ingest.mulesoft.com:8443
      * data-authenticator.anypoint.mulesoft.com:443
      * s3.amazonaws.com:443
    * 2.x installers:
      * dias-ingestor-router.us-east-1.prod.cloudhub.io:443
      * data-authenticator.anypoint.mulesoft.com:443
      * s3.amazonaws.com:443
  * US-GOV Control Plane
    * data-authenticator.gov.anypoint.mulesoft.com:443
    * dias-ingestor-router.us-gov-west-1.gprod.msap.io:443
  * EU Control Plane
    * 3.x installers:
      * eu1.ingest.mulesoft.com:8443
      * data-authenticator.anypoint.mulesoft.com:443
      * s3.eu-central-1.amazonaws.com:443
    * 2.x installers:
      * dias-ingestor-router.eu-central-1.prod-eu.msap.io:443
      * data-authenticator.eu1.anypoint.mulesoft.com:443
      * s3.eu-central-1.amazonaws.com:443


### Ports
Region | Name | Port  
---|---|---  
us1.ingest.mulesoft.com  
anypoint.mulesoft.com  
runtime-manager.anypoint.mulesoft.com  
analytics-ingest.anypoint.mulesoft.com  
data-authenticator.anypoint.mulesoft.com  
exchange-files.anypoint.mulesoft.com  
exchange2-asset-manager-kprod.s3.amazonaws.com  
runtime-manager.gov.anypoint.mulesoft.com  
gov.anypoint.mulesoft.com  
exchange-files.gov.anypoint.mulesoft.com  
exchange2-asset-manager-kgprod.s3.us-gov-west-1.amazonaws.com  
eu1.ingest.mulesoft.com  
eu1.anypoint.mulesoft.com  
runtime-manager.eu1.anypoint.mulesoft.com  
analytics-ingest.eu1.anypoint.mulesoft.com  
data-authenticator.eu1.anypoint.mulesoft.com  
exchange2-asset-manager-kprod-eu.s3.eu-central-1.amazonaws.com  
## Supported Operating Systems
Install the Anypoint Monitoring agent on a supported operating systems:
  * 3.x versions of the installer:
    * RedHat Enterprise Linux 9
    * Ubuntu 20.04
    * Windows Server 2019
  * 2.x versions of the installer:
    * CentOS 7
    * RedHat Enterprise Linux 7
    * Ubuntu 16.04 and 18.04
    * Windows Server 2016, Windows Server 2019


To install the agent, you must have root or administrative privileges.
## Supported Versions of Mule Runtime Engine
Run your Mule app on a supported Mule runtime version. For supported versions, see Mule Runtime Release Notes.
## Supported Versions of Mule Runtime Manager Agent
You must install a supported version of the Anypoint Runtime Manager agent in the Mule runtime engine running on the server. For more information, see Runtime Manager Agent.
  * 2.7.x
An Anypoint Monitoring agent from the version 3.x installer requires Mule Runtime Manager agent version 2.7.1 or higher.
  * 2.6.x
  * 2.5.x
  * 2.4.x
  * 2.3.0
  * 2.2.x
  * 2.1.9
  * 2.1.7
  * 2.1.6
  * 1.14.x
  * 1.13.0
  * 1.12.x
  * 1.11.x
  * 1.10.0


You must manage Mule deployments in Runtime Manager. You can add and check for your hybrid deployments from the Runtime Manager UI or the Runtime Manager REST API.
## Hardware Prerequisites
Ensure that the server has the minimum hardware requirements for Anypoint Monitoring, which are:
  * 2GHz CPU or 1 virtual CPU
  * 1GB RAM
  * 4GB disk space
  * Intel x86 or x64 compatible CPU architecture


## Install the Anypoint Monitoring Agent
  1. Meet the prerequisites in Before You Begin:
Mule runtime engine must be installed and running.
  2. Follow the steps in Managing CloudHub and Hybrid Settings to download the _latest version_ of the agent’s `am.zip` file and install the agent from **Settings** , in Anypoint Monitoring.
     * When you extract the `am.zip` file, a folder called `am` is created in the location where you extracted the ZIP.
     * The installation script asks if you want Anypoint Monitoring installed as a service.
       * If you don’t want to have it preset as a service by the installer, select **No**.
       * You can call the installation script with the `-x` argument to skip this prompt. If you skip the prompt, Anypoint Monitoring is not installed as a service.
     * You cannot install the Anypoint Monitoring agent on a server that has an existing filebeat instance running.
     * When you install the Anypoint Monitoring agent on a hybrid server in a server group, use Runtime Manager to restart the server to enable Anypoint Monitoring to begin collecting metrics.


## Proxy Settings
If you proxy your monitoring data as it leaves the data center, you must configure the Anypoint Monitoring agent to communicate through the proxy.
The configuration and supported protocol depends on the version of the Anypoint Monitoring agent’s installer:
  * 3.x: Supports the HTTP/HTTPS protocol and does not support SOCKS5
  * 2.x: Supports the SOCKS5 protocol and does not support HTTP/HTTPS


See also, Supported Operating Systems.
### Configure Proxy Communication for Monitoring Agents from Version 3.x Installers (HTTP/HTTPS Protocol)
If you proxy your monitoring data as it leaves the data center and you’re using an Anypoint Monitoring agent from a _version 3.x_ installer, the proxy must support the HTTP/HTTPS protocol, and you must configure the Anypoint Monitoring agent to communicate through the proxy.
To configure the agent for an HTTP or HTTPS proxy:
  1. Set up Runtime Manager with a proxy. See Create a Server in Runtime Manager.
  2. Install the Anypoint Monitoring agent and use the `-p` option. For example, on a Linux server, at the command line, enter:
```
$ ./install -p https://user:password@https-server:1080
```

For a user or password of the proxy configuration that contains URI-reserved characters, you must percent-encode the characters according to the encoding rules of RFC 3986.


### Configure Proxy Communication for Monitoring Agents from Version 2.x Installers (SOCKS5 Protocol)
If you proxy your monitoring data as it leaves the data center and you’re using an Anypoint Monitoring agent from a _version 3.x_ installer, the proxy must support the SOCKS5 protocol, and you must configure the Anypoint Monitoring agent to communicate through the proxy.
To configure the agent for the SOCKS5 proxy:
  1. Set up Runtime Manager with a proxy. See Create a Server in Runtime Manager.
  2. Install the Anypoint Monitoring agent and use the `-p` option. For example, on a Windows server, at the command line, enter:
```
<location-of-am-folder\bin>powershell -file install.ps1 -p socks5://user:password@socks5-server:1080
```

For a Linux server, enter:
```
$ ./install -p socks5://user:password@socks5-server:1080
```

For a user or password of the proxy configuration that contains URI-reserved characters, you must percent-encode the characters according to the encoding rules of RFC 3986.
For a user or password of the proxy configuration that contains URI-reserved characters, you must percent-encode the characters according to the encoding rules of RFC 3986.


### Configure Proxy Communication for Monitoring Agents from Version 2.x Installers (SOCKS5 Protocol)
If you proxy your monitoring data as it leaves the data center and you’re using an Anypoint Monitoring agent from a _version 3.x_ installer, the proxy must support the SOCKS5 protocol, and you must configure the Anypoint Monitoring agent to communicate through the proxy.
To configure the agent for the SOCKS5 proxy:
  1. Set up Runtime Manager with a proxy. See Create a Server in Runtime Manager.
  2. Install the Anypoint Monitoring agent and use the `-p` option. For example, on a Windows server, at the command line, enter:
```
<location-of-am-folder\bin>powershell -file install.ps1 -p socks5://user:password@socks5-server:1080
```

For a Linux server, enter:
```
$ ./install -p socks5://user:password@socks5-server:1080
```

For a user or password of the proxy configuration that contains URI-reserved characters, you must percent-encode the characters according to the encoding rules of RFC 3986.


## Verify the Anypoint Monitoring Agent is Running
  1. Wait 10 minutes for the configuration to propagate.
  2. Log in to Anypoint Platform with a user account that has permission to access Anypoint Monitoring content.
  3. Navigate to **Anypoint Monitoring**.
  4. On the **Built-in Dashboards** page, select an app that is deployed on the server where you installed the Anypoint Monitoring agent.
  5. Check whether the CPU Utilization and Memory Utilization charts are displaying data. If you see data in the charts, the Anypoint Monitoring agent is installed and running. If you do not see data in the charts, verify that Anypoint Monitoring is able to connect to the endpoint.
If necessary, you can add the endpoint to an allowlist for outbound firewall rules so your server allows connections. The endpoints to add to the allowlist are listed in Before You Begin.   
---  
  6. Log into the server on which the Anypoint Monitoring agent is running.
  7. Open the log file for the data-transfer agent:
     * 3.x log file path: `./am/logs`
If this agent is running as a service on Linux, the log file path is `/var/log/otel-collector`.
     * 2.x log file path: `/var/log/filebeat`
If this agent is running as a service on Linux, the log file path is /var/log/filebeat`.
  8. Look for entries that mention connection failures or connection retries.


## Update the Anypoint Monitoring Agent
When you upgrade the Runtime Manager agent, you must reinstall the Anypoint Monitoring agent.
If you need to update the Anypoint Monitoring agent at some point in the future, follow these steps:
  1. Stop Mule.
  2. In the `am` folder, run the following command: `./bin/uninstall`
  3. Delete the `am` folder.
  4. Follow the instructions for installing the Anypoint Monitoring agent on-premises.
  5. Restart Mule.


If you add or remove a Mule runtime from the cluster or a server group, you must reinstall the Anypoint Monitoring agent.   
---  
## Performance Impact
Enabling monitoring can impact CPU utilization and memory utilization depending on your specific application configuration. Before enabling monitoring in your production environment, test any performance impact in your pre-production environment to ensure optimal performance and sizing.
## Troubleshooting
  * If you find connection failure or retry entries, ensure that port 8443 (Anypoint Monitoring agents from installer version 3.x and later only) and 443 (all Anypoint Monitoring agents) in your outbound firewall are open.
  * If you don’t find connection failure or retry entries, contact the support team.


## Disable Log Forwarding for Hybrid Runtime Servers
Availability: Advanced and Titanium tiers only  
---  
You can disable log forwarding to Anypoint Monitoring if you have AM installer v.2.5.10 or later. To disable this feature, you must configure a JVM property on each hybrid server for which you want to disable log forwarding. You cannot disable log forwarding per application.
To disable log forwarding:
  1. Upgrade the Anypoint Monitoring agent to a version that supports disabling log forwarding (am-2.5.10.0.zip or later).
  2. Stop Mule.
  3. Add the following line in the `MULE_HOME/conf/wrapper.conf` file (where is the next available property index): `wrapper.java.additional.<N>=-Danypoint.platform.monitoring.logs_forwarding.enabled=false`
  4. Restart Mule.


## See Also
  * Anypoint Monitoring Settings
  * Download and Install Mule 4
  * Installing and Configuring Mule Runtime Manager Agent