Viewing and Configuring Logs in Runtime Fabric | MuleSoft Documentation
# Viewing and Configuring Logs in Runtime Fabric
Runtime Fabric generates log files for the following:
  * Deployed Mule applications
  * Deployed API proxies
  * Runtime Fabric services
  * Kubernetes services


## Log Levels
Runtime Fabric enables you to specify the severity level of messages that are written to the log file.
Runtime Fabric does not support custom logging categories.
Table 1. Runtime Fabric Log Levels Value | Description | Command  
---|---|---  
All Priorities |  List all messages.  
List only error messages, such when an exception occurs. |  priority:ERROR  
List only fatal messages when an application fails. |  priority:FATAL  
List informative messages. |  priority:INFO  
List messages about application and worker startup. |  priority:SYSTEM  
List messages about console events such as setting the object store. |  priority:CONSOLE  
List warning messages. |  priority:WARN  
List debugging messages. |  priority:DEBUG  
Log levels are specified per Mule application or API proxy during deployment.
## Change the Application Log Level
To change the log level for an application, use Anypoint Runtime Manager to configure the `logging.level.<PACKAGE>=LEVEL` property.
  1. In Runtime Manager, navigate to an application’s **Settings** page.
  2. Click the **Properties** tab.
  3. Select **Table View**.
  4. In the **New Key** field, add the `logging.level.<PACKAGE>` property.
  5. In the **New Value** field, add the desired log level.
For example, if you add the key `logging.level.org.mule.service.http.impl.service.HttpMessageLogger` and a value of `DEBUG`, `DEBUG` level log information is emitted from `HttpMessageLogger`.


## View Logs from an Application
Ops Center shows a stream of logs generated by applications and services running on Runtime Fabric on VMs / Bare Metal. This is useful when log forwarding is not configured.
  1. From Ops Center, navigate to **Logging**.
  2. Select **Kubernetes**.
  3. Select the **Pods** tab.
  4. In the drop-down list adjacent to the search input area, select the ID of the environment where the application is deployed.
  5. Locate the pod name that starts with the name of your application.
  6. Select that pod name and then select **Logs**.


The **Logs** tab is displayed with a filter applied to your application.
To view the latest logs, select **Refresh**.   
---  
### Filters
There are two levels of filters for drilling down in the logs:
  * The `_Containers_`filter filters by container name.
  * The filter filters by pod name.
This is useful for specifying application names that are followed by a wildcard (`%`).


## Forward Logs to External Services
Runtime Fabric on VMs / Bare Metal supports log forwarding to:
  * Anypoint Monitoring
  * Third-party logging solutions using log forwarding output plugins:
    * Azure Log Analytics
    * Elasticsearch
    * Graylog Extended Log Format (GELF)
    * Splunk
    * CloudWatch
    * Syslog
    * Datadog
  * External services, using the `rsyslog` client service


### Limitations for External Log Forwarding
  * Using a Log4j appender and external log forwarding at the same time might lead to duplicate log collection. Use one or the other.
  * Only one external log forwarding configuration is supported at any one time. However, you can enable Anypoint Platform log forwarding and an external log forwarding service at the same time.
  * External log forwarding does not currently support proxies that are configured for a Runtime Fabric cluster. You must explicitly configure firewall rules to allow the outbound IP address and port of the external log forwarding service.
  * External log forwarding does not currently join multiline log entries.
  * Runtime Fabric supports only one-way TLS (client to server) verification.
  * External log forwarding follows Syslog Protocol RFC5424, and therefore hostnames, application names, and process IDs are not automatically mapped to log entries. These fields are mapped when you forward logs to other external services.


### Enable Log Forwarding
  1. Install and configure the applicable third-party logging solution. Verify that you have the required resources allocated. Refer to the documentation for your third-party logging solution for details.
  2. From Anypoint Platform, select **Runtime Manager**.
  3. Select **Runtime Fabrics** in the sidebar navigation.
  4. Select the applicable Runtime Fabric based on the name used during installation.
  5. Select the **Logs** tab.
  6. Select the **Forward application logs to Anypoint Monitoring (Requires Titanium)** checkbox if you have a Titanium subscription and you want to forward logs to Anypoint Monitoring.
  7. Select the **Forward application logs to an external service** checkbox to forward logs using a log forwarding output plugin.
    1. In the **Connects to** drop-down list, select the applicable third-party log forwarding solution.
    2. Enter configuration information for your third-party logging solution, as shown in the following sections:
       * Azure Log Analytics Configuration Parameters
       * Elasticsearch Configuration Parameters
       * GELF Configuration Parameters
       * Splunk Configuration Parameters
       * Syslog Configuration Parameters
       * Datadog Configuration Parameters
  8. Select the log level:
     * **INFO** : Informative messages
     * **DEBUG** : Debugging messages
     * **WARNING** : Warning messages
     * **ERROR** : Error messages, such as when an exception occurs
     * **FATAL** : Fatal messages for when an application fails
  9. Optionally, set the log level for specific a Java package class by selecting the log level, entering the `package.name`, and pressing the Enter key.
  10. Select **Apply changes**.
  11. To verify successful log forwarding, manually check the logs using the external log forwarding service.


#### Azure Log Analytics Configuration Parameters
Table 2. Azure Log Analytics Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Customer ID |  Specifies the customer ID or workspace ID string.  
Specifies the primary or secondary connected sources client authentication key.  
Specifies the event type name. |  `runtime_fabric`  
Example output plugin configuration:
```
[OUTPUT]
  Name    azure
  Customer_ID id
  Shared_Key key
```
#### Azure Log Analytics Configuration Parameters
Table 2. Azure Log Analytics Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Customer ID |  Specifies the customer ID or workspace ID string.  
Specifies the primary or secondary connected sources client authentication key.  
Specifies the event type name. |  `runtime_fabric`  
Example output plugin configuration:
```
[OUTPUT]
  Name    azure
  Customer_ID id
  Shared_Key key
```

#### Elasticsearch Configuration Parameters
Runtime Fabric 1.10.26 and later uses Fluent Bit 1.8.3 for external log forwarding. If you forward logs to Elasticsearch, you must set the configuration parameter `Generate_ID` to `On`. By default, Fluent Bit sets this parameter to `Off`.  
---  
Table 3. Elasticsearch Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Specifies the IP address or hostname of the target Elasticsearch instance.  
Specifies the TCP port of the target Elasticsearch instance.  
Specifies the index name. |  runtime_fabric  
Elasticsearch accepts new data on HTTP query path `/_bulk`. But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such a path on the output plugin side. It adds a path prefix in the indexing HTTP POST URI. |  Empty string  
Buffer Size |  Specifies the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where reading the full response is needed. Note that the size of the response grows depending on the number of records inserted. To specify an unlimited amount of memory, set this value to `False`. Otherwise, set the value according to the Unit Size specification.  
Specifies an optional username credential for Elastic X-Pack access.  
HTTP Passwd |  Specifies a password for the user defined in `HTTP User`.  
Specifies the type name. |  runtime_fabric  
Logstash Format |  Enables Logstash format compatibility. This option takes a Boolean value: `True`, `False`, `On`, or `Off`.  
Logstash Prefix |  When `Logstash Format` is enabled, the `Index` name is composed of a prefix and the date. For example, if `Logstash Prefix` is set to `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string that is appended belongs to the date when the data is generated.  
Logstash DateFormat |  Specifies the time format (based on `strftime`) that is used to generate the second part of the `Index` name.  
When `Logstash Format` is enabled, each record is assigned a new timestamp field. The `Time Key` property defines the name of that field.  
Time Key Format |  When `Logstash Format` is enabled, this property defines the format of the timestamp. |  %Y-%m-%dT%H:%M:%S  
Include Tag Key |  When enabled, the tag name is appended to the record.  
When `Include Tag Key` is enabled, this property defines the key name for the tag.  
Generate ID |  When enabled, generates the `_id` value for outgoing records. This prevents duplicate records when retrying ES.  
Replace Dots |  When enabled, replaces field name dots ('.') with an underscore ('_'), which is required by Elasticsearch 2.0 through 2.3.  
Trace Output |  When enabled, prints the Elasticsearch API calls to stdout (for diagnosis).  
Trace Error |  When enabled, prints the Elasticsearch API calls to stdout when Elasticsearch returns an error.  
Current Time Index |  Specifies using the current time for index generation instead of message record information.  
Logstash Prefix Key |  Prefixes keys with this string.  
Enables or disables TLS support.  
CA Path Certificate File |  Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.  
Example output plugin configuration:
```
[OUTPUT]
  Name es
  Host 192.168.2.3
  Port 9200
  Index my_index
  Type my_type
```

#### GELF Configuration Parameters
Table 4. GELF Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Specifies the pattern to match for log tags to be output by this plugin.  
Specifies the IP address or hostname of the target Graylog server.  
Specifies the port on which your Graylog GELF input is listening.  
Specifies the protocol to use (`tls`, `tcp`, or `udp`).  
Gelf Short_Message Key |  Specifies a short descriptive message (This must be set in GELF).  
Gelf Timestamp Key |  Specifies your log timestamp (This should be set in GELF).  
Gelf Host Key |  Specifies the key for the value that is used as the name of the host, source, or application that sent the message. (This must be set in GELF).  
Gelf Full Message Key |  Specifies the key to use as the long message, which can contain a backtrace. (This is optional in GELF.) |  full_message  
Gelf Level Key |  Specifies the key to be used as the log level. Its value must be a standard syslog level (between 0 and 7).  
Packet Size |  Specifies the size of packets to be sent if the transport protocol is set to `udp`.  
Specifies compression of your UDP packets if the transport protocol is set to `udp`.  
Enables or disables TLS support.  
CA Path Certificate File |  Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.  
Example output plugin configuration:
```
[OUTPUT]
  Name          gelf
  Match          kube.*
  Host          <your-graylog-server>
  Port          12201
  Mode          tcp
  Gelf_Short_Message_Key log
```

#### Splunk Configuration Parameters
Table 5. Splunk Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Specifies the IP address or hostname of the target Splunk service.  
Specifies the TCP port of the target Splunk service.  
Splunk Token |  Specifies the Authentication Token for the HTTP Event Collector interface.  
Specifies an optional username for basic authentication on the HTTP Event Collector.  
HTTP Passwd |  Specifies a password for the user defined in `HTTP User`.  
Enables or disables TLS support.  
CA Path Certificate File |  Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.  
Example output plugin configuration:
```
[OUTPUT]
  Name     splunk
  Host     127.0.0.1
  Port     8088
  Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
  Tls     On
```

#### CloudWatch Configuration Parameters
Table 6. CloudWatch Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Specifies the AWS region  
Aws Access Key Id |  Specifies the access key ID for the IAM user  
Aws Secret Access Key |  Specifies the AWS secret access key for the IAM user  
Log Group Name |  Specifies the name of the CloudWatch log group to which you want log records sent |  runtime_fabric_cloudwatch  
Log Stream Name |  Specifies the name of the CloudWatch log stream to which you want log records sent  
Log Stream Prefix |  Specifies the prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the `Log Stream Name` option.  
Specifies the ARN of an IAM role to assume (for cross account access).  
Auto Create Group |  Automatically creates the log group. Valid values are "On" or "Off" (case sensitive).  
Note: You must specify either `Log Stream Name` or `Log Stream Prefix`, but not both.
Example output plugin configuration:
```
[OUTPUT]
  Name          cloudwatch_logs
  region         us-east-1
  aws_access_key_id   <your_AWS_access_key_ID>
  aws_secret_access_key <your_AWS_secret_access_key>
  log_group_name     runtime_fabric_cloudwatch
  log_stream_prefix   my_stream
  auto_create_group   On
```

#### Syslog Configuration Parameters
Table 7. Syslog Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Host name or IP address of the remote Syslog server  
TCP or UDP port of the remote Syslog server  
The mode of the transport type  
Syslog format |  Syslog protocol format  
Syslog Maxsize |  Integer value that sets the maximum number of bytes per message |  1024 for rfc3164, 2048 for rfc5424  
Example output plugin configuration:
```
[OUTPUT]
  Name      syslog
  Host      127.0.0.1
  Port      514
  Mode      udp
  Syslog_Format rfc5424
  Syslog_Maxsize 2048
```
#### Syslog Configuration Parameters
Table 7. Syslog Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
Host name or IP address of the remote Syslog server  
TCP or UDP port of the remote Syslog server  
The mode of the transport type  
Syslog format |  Syslog protocol format  
Syslog Maxsize |  Integer value that sets the maximum number of bytes per message |  1024 for rfc3164, 2048 for rfc5424  
Example output plugin configuration:
```
[OUTPUT]
  Name      syslog
  Host      127.0.0.1
  Port      514
  Mode      udp
  Syslog_Format rfc5424
  Syslog_Maxsize 2048
```

#### Datadog Configuration Parameters
Table 8. Datadog Configuration Parameters Key | Description | Required | Default Value  
---|---|---|---  
The host name of the remote Datadog server.  
The Datadog API key.  
Compresses the payload in GZIP format. Datadog recommends setting this to `gzip`.  
Datadog Service |  The name of the service generating the logs. For example, `runtime_fabric`  
Datadog Source |  The type of service. For example, `postgres` or `nginx`  
Datadog Tags |  The tags assigned to the logs in Datadog.  
End-to-end secure communications protocol support. DataDog does not support CA certificate files.  
Example output plugin configuration:
```
[OUTPUT]
  Name    datadog
  Host    http-intake.logs.datadoghq.com
  apikey   <your-datadog-api-key>
  TLS     on
  compress  gzip
  dd_service <your-app-service>
  dd_source  <your-app-source>
  dd_tags   team:logs,foo:bar
```

#### Configure Log Forwarding
The following configuration options are available:
  * Mule Applications: Forward logs from Mule applications.
  * Internal load balancer and components: Forward logs from Runtime Fabric components and the internal load balancer.
  * Runtime Fabric appliance: Forward logs from monitoring and appliance services.


  * All options are enabled by default.
  * Disabling log forwarding for Mule applications is not supported. Log forwarding can be disabled per application during deployment.

  
---  
### Kubernetes Metadata
When forwarding logs to external services, additional metadata might be available in some providers. When performing searches, the user can leverage these metadata labels to help narrow down search results and focus on the logs only from containers that matters.
Table 9. Kubernetes Metadata Key | Description | Sample Value  
---|---|---  
container_name |  Specifies the name of the container.  
container_image |  Specifies the full name of the container image. |  000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0:v1.0.0  
container_hash |  Specifies the name of the container image and includes the immutable identifier (SHA256 value). |  000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0@sha256:866daf14a55da1db21c562ddc09487bbb8ffae1dd4cc80b4dc5c64e0ac9c124b  
Specifies the ID of the container in Docker. |  a54a0bf80400b6cd44cae53b9705f5b6330ce4e8efb85c92b3f9c1709829c6fa  
Specifies the IP of the node on which the pod is running. |  10.0.178.87  
namespace_name |  Specifies the Kubernetes namespace of the pod.  
Specifies the ID of the pod. |  118cff63-a5ac-11ea-8414-0e281fcb2861  
Specifies the name of the pod. |  testapp-5789f9774-6lxv7  
Specifies the app label for the application the pod is supporting.  
labels_environment |  Specifies the Anypoint environment ID of the environment in which the application is deployed. |  24166311-4e5e-4091-b417-42307747267e  
labels_organization |  Specifies the Anypoint organization ID of the organization in which the application is deployed. |  24166311-4e5e-4091-b417-42307747267e  
labels_rtf_mulesoft_com_id |  Specifies the ID of the application deployment on Anypoint Platform. |  194a32bf-bf7e-4c91-aa4e-44132e5aa76d  
labels_rtf_mulesoft_com_version |  Specifies the spec version ID of the application deployment on Anypoint Platform. |  194a32bf-bf7e-4c91-aa4e-44132e5aa76d  
labels_type |  Specifies the type of deployment that the pod is running. |  MuleApplication  
## Send a Test Message
You can send a test message to verify connectivity between Runtime Fabric on VMs / Bare Metal and the external log forwarding solution:
  1. In Runtime Manager, select **Runtime Fabrics** in the sidebar navigation.
  2. Select the applicable Runtime Fabric based on the name used during installation.
  3. Select the **Log Forwarding** tab.
  4. Select **Forward application logs to an external service**.
  5. Select a service and configure the fields as needed for your log forwarding solution.
  6. Save the changes and apply the configuration to the Runtime Fabric cluster.
  7. To test the configuration and reachability of the log forwarding solution:
    1. Trigger a test message for the selected service.
    2. Verify the text that is forwarded to log forwarding solution.


## Disable Anypoint Monitoring or External Log Forwarding per Application
  1. Navigate to Runtime Manager and select **Applications**.
  2. Select the application for which you want to disable log forwarding.
  3. Select the **Logs** tab.
  4. Deselect the applicable log forwarding options:
     * **Forward logs to Anypoint Monitoring**
     * **Forward logs to <third-party service>**
  5. Select **Deploy** to redeploy the application.


## Forward Logs to Other External Services
Anypoint Runtime Fabric on VMs / Bare Metal also enables you to forward application and cluster logs to external logging solutions other than the third-party solutions supported using log forwarding output plugins. The log forwarder (`rsyslog` client service) built into Runtime Fabric enables you to send log data to an `rsyslog` server over TCP or UDP for viewing, retention, and receiving alerts in a centralized destination. Logging solutions such as Logstash provide methods to receive log data from `rsyslog` clients.
To enable multiline logging, your log entries must begin with an open bracket, `[`.  
---  
  1. Using a terminal, open a shell/SSH connection to a controller VM.
  2. Create a file named `log-forwarder.yaml`.
  3. Add the following content to this file after customizing based on the table below:
```
kind: logforwarder
version: v2
metadata:
  name: log-forwarder
spec:
  address: 192.168.100.1:514
  protocol: udp
```

Use the following values as applicable to your environment:
Key | Description  
---|---  
Specifies the name of the log forwarding rule.  
Specifies the endpoint and port to forward the log data.  
Specifies the protocol to send the data to. Supported protocols are TCP or UDP.  
  4. Run the following command on the controller VM, referencing the file created earlier.


```
gravity resource create log-forwarder.yaml
```

Your logs are forwarded to your external logging solution.
## Troubleshoot Log Forwarding Issues
If you experience issues with log forwarding:
  * Verify that the log forwarding configuration specifies the correct host and port information.
  * Verify that the server side configuration, such as networking, is correct and that the server is accessible.
  * Verify log forwarding status:
    1. Navigate to Runtime Manager and select **Runtime Fabrics**.
    2. Select the name of your Runtime Fabric to open the management page.
    3. Select the **Health Details** tab.
    4. Verify the following information as applicable:
       * The status of **Forwarding Logs To External Provider** is `Healthy`.
       * The status **Forwarding Logs to Anypoint Monitoring** is `Healthy`.
  * Run the following commands inside the RTF cluster:
    * `kubectl -nrtf get pods -lapp=external-log-forwarder` for a list of all external log forwarding pods. Verify that these pods are in `Running` state.
    * `kubectl -nrtf logs -l app=external-log-forwarder` to view the logs of external log forwarding pods.


## See Also
  * Configure Alerting on Anypoint Runtime Fabric
  * Deploy a Mule Application to a Runtime Fabric