Configuring Flex Gateway Log Output for Third-Party Services | MuleSoft Documentation
# Configuring Flex Gateway Log Output for Third-Party Services
Anypoint Flex Gateway enables you to deliver runtime and access logs to any supported Fluent Bit v3.2.10 output type. You configure log output via a custom YAML file. Additionally, to view access logs, you must also apply a Message Logging policy.
The following configuration file examples demonstrate how to direct Flex Gateway logs to output types: `File`, `HTTP`, and `Splunk`. However, you can use any Fluent Bit v3.2.10 output type, such as `Azure Log Analytics`, `Amazon S3`, `Kafka`, and more. For a complete list of Fluent Bit output types, see the Fluent Bit documentation for your Fluent Bit version.
As well as being able to configure additional Fluent Bit log output types, Flex Gateway sends runtime logs to runtime manager and access logs to API Manager. To learn more, see:
  * Viewing Runtime Logs in Runtime Manager in Connected Mode
  * Configuring Message Logging for Flex Gateway in Connected Mode.


Configure Log Output for Flex Gateway as a Linux Service |  Configure Log Output for Flex Gateway in a Docker Container |  Configure Log Output for Flex Gateway in a Kubernetes Cluster  
---|---|---  
## Before You Begin
Before configuring Flex Gateway log output, complete the following tasks:
  * Download Flex Gateway
  * Register and Run Flex Gateway


Additionally, depending on your output, ensure that the service where you direct logs to is operational and that you have the necessary information:
  * Splunk:
    * A running Splunk service. You can test Splunk with a free Splunk Cloud trial.
    * Added an HTTP Event Collector (HEC) data input to the Cloud or Enterprise Splunk service. See the Splunk documentation about how to add a data input, and how to create an HEC token.
  * Dynatrace:
    * A running Dynatrace service. You can test Dynatrace with a free Dynatrace trial.
    * Environment ID. To find your Dynatrace environment ID, see Environment ID
    * Access token. To generate a Dynatrace access token, see Access Tokens. Ensure that the token has the `Ingest logs` token scope.


## Runtime Logs and Access Logs
Runtime and access logs both appear in the same log output. However, you can choose whether to include both logs in your log output. To use access logs, you must also apply a Message Logging policy.
### Runtime Logs
Runtime logs include the following information:
  * The startup and shutdown of Flex Gateway, Fluent Bit, and Envoy
  * Deployed APIs and policies
  * Possible misconfigurations of Flex Gateway or other errors


### Access Logs
Access logs describe incoming interaction with Envoy over a period of time and incoming requests and responses for specific APIs.
To enable access logs, you must apply a Message Logging policy to the API instances you want to monitor. To apply a Message Logging policy, see Configuring Access Logs.
## Logging Configuration File
The `logging` object configures the delivery of runtime and access logs enabled via the message logging policy. Logs are delivered to any supported Fluent Bit v3.2.10 output.
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: <value>
 namespace: <namespace name>
spec:
 logging:
  outputs:
  - name: <output-name>
   type: <output-type>
   parameters:
    <param-name>: <param-value>
  runtimeLogs:
   logLevel: <value>
   outputs: <value>
  accessLogs:
   outputs: <value>
```

Parameter | Required or Optional | Default Value | Description  
---|---|---|---  
`logging.outputs[#].name` | Required | The name of this output to later refer to in runtime and access logs configurations.  
`logging.outputs[#].type` | Required | An output type supported by Fluent Bit. For Fluent Bit output types, see Fluent Bit documentation for your Fluent Bit version.  
`logging.outputs[#].parameters` | Required | A map of parameters for the specific Fluent Bit output type. For Fluent Bit output type parameters, see Fluent Bit documentation for your Fluent Bit version.  
`logging.accessLogs.outputs` | Optional | A list of output names to redirect access logs to.  
`logging.runtimeLogs.logLevel` | Optional | A parameter specifying log detail. The supported `logLevel` types listed in decreasing verbosity are `debug`, `info`, `warn`, `error`, and `fatal`.  
`logging.runtimeLogs.outputs` | Optional | A list of output names to redirect runtime logs to.  
Leaving a value blank applies the default value to your configuration.
In addition to parameters above, Flex Gateway offers variables for logging output. When configured, the variables render as one of their respective outputs in the logs:
Variable | Description | Outputs  
---|---|---  
Date and time of the logged event | A specific time, for example, `17/11/2022-09:48:27AM`  
Flex Gateway service where the logged event occurred | `flex-gateway-agent`, `flex-gateway-envoy`, and `flex-gateway-fluent`  
`Loglevel` of the logged event | `debug`, `info`, `warn`, `error`, or `fatal`  
Log type | `runtimeLog` or `accessLog`  
To view configuration files with applied values, see the following configuration file examples:
  * File Configuration Example
  * HTTP Configuration Example
  * Splunk Configuration Example
  * Dynatrace Configuration Example


## Configure Log Output for Flex Gateway as a Linux Service
  1. Create a YAML configuration file in the Flex Gateway configuration directory:
```
sudo touch /usr/local/share/mulesoft/flex-gateway/conf.d/logs-config.yaml
```

  2. Update the file with your desired output configuration details as shown in the following examples:
     * File Configuration Example
     * HTTP Configuration Example
     * Splunk Configuration Example
     * Dynatrace Configuration Example
If no events appear in the logging output, check the Flex Gateway runtime logs for any configuration errors.
To view runtime logs, see Viewing Runtime Logs in Runtime Manager in Connected Mode.
For additional troubleshooting help, see Troubleshooting Self-Managed Gateway Logging Issues.
  3. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.


## Configure Log Output for Flex Gateway in a Docker Container
If you have already added a volume for a folder with your Flex Gateway configuration files, skip to the create and save a YAML file step.   
---  
  1. Press Ctrl+C to stop your Flex Gateway and any replicas.
  2. Create a folder named `app` in the directory with your Flex Gateway configuration files.
  3. Restart your Flex Gateway with an additional volume for the new `app` directory:
```
docker run --rm \
-v "$(pwd)":/usr/local/share/mulesoft/flex-gateway/conf.d \
-p 8080:8080 \
mulesoft/flex-gateway
```

Specify an optional name you want to assign to your Flex Replica by including the following: `-e FLEX_NAME=<name-for-flex-replica> \`.   
---  
  4. Create and save a YAML file with your desired output configuration details as shown in the following examples:
     * File Configuration Example
     * HTTP Configuration Example
     * Splunk Configuration Example
     * Dynatrace Configuration Example
If no events appear in the logging output, check the Flex Gateway runtime logs for any configuration errors.
To view runtime logs, see Viewing Runtime Logs in Runtime Manager in Connected Mode.
For additional troubleshooting help, see Troubleshooting Self-Managed Gateway Logging Issues.
  5. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.


## Configure Log Output for Flex Gateway in a Kubernetes Cluster
To configure logging for Flex Gateway:
  1. Create a new resource using a YAML configuration file with your desired output configuration details as shown in the following examples:
     * File Configuration Example
     * HTTP Configuration Example
     * Splunk Configuration Example
     * Dynatrace Configuration Example
If no events appear in the logging output, check the Flex Gateway runtime logs for any configuration errors.
To view runtime logs, see Viewing Runtime Logs in Runtime Manager in Connected Mode.
For additional troubleshooting help, see Troubleshooting Self-Managed Gateway Logging Issues.
  2. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.


## File Configuration Example
Configure Flex Gateway to send formatted runtime and access logs to a file called `log.txt` (Fluent Bit `File` output), in the `/var/log` directory. As an example, the following definition specifies a `spec.logging.outputs.type` value of `file`:
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: logging-config
spec:
 logging:
  outputs:
  - name: default
   type: file
   parameters:
    file: /var/log/log.txt # Docker: replace `/var/log` with the absolute path to the mounted configuration directory
    format: template
    template: |
     [{date}][{logger}][{level}][{kind}] {message}
  runtimeLogs:
   logLevel: info
   outputs:
   - default
  accessLogs:
   outputs:
   - default
```
## File Configuration Example
Configure Flex Gateway to send formatted runtime and access logs to a file called `log.txt` (Fluent Bit `File` output), in the `/var/log` directory. As an example, the following definition specifies a `spec.logging.outputs.type` value of `file`:
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: logging-config
spec:
 logging:
  outputs:
  - name: default
   type: file
   parameters:
    file: /var/log/log.txt # Docker: replace `/var/log` with the absolute path to the mounted configuration directory
    format: template
    template: |
     [{date}][{logger}][{level}][{kind}] {message}
  runtimeLogs:
   logLevel: info
   outputs:
   - default
  accessLogs:
   outputs:
   - default
```

For Docker, in `spec.logging.output.parameters.file`, replace `/var/log` with the absolute path to the mounted configuration directory.   
---  
## HTTP Configuration Example
You can configure Flex Gateway to send formatted runtime and access logs to a `HTTP` output. As an example, the following definition specifies a `spec.logging.outputs.type` value of `HTTP`:
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: logging-config
spec:
 logging:
  outputs:
  - name: default
   type: HTTP
   parameters:
    host: collectors.au.sumologic.com
    port: "443"
    URI: /receiver/v1/http/[PrivateKey]
    format: json_lines
    json_date_key: timestamp
    json_date_format: iso8601
  runtimeLogs:
   logLevel: info
   outputs:
   - default
  accessLogs:
   outputs:
   - default
```

## Splunk Configuration Example
Splunk begins ingesting logs soon after you apply this resource.
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: splunk-config
spec:
 logging:
  outputs:
  - name: splunk
   type: splunk
   parameters:
    host: <instance-name>.splunkcloud.com
    port: "8088"
    splunk_token: <Splunk's HEC token>
    tls: "on"
    tls.verify: "off"
  runtimeLogs:
   logLevel: info
   outputs: [splunk]
  accessLogs:
   outputs: [splunk]
```

## Dynatrace Configuration Example
You can configure Flex Gateway to send formatted runtime and access logs to a Dynatrace environment `HTTP` output. As an example, the following file contains the necessary configurations:
```
apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
 name: dynatrace-config
spec:
 logging:
  outputs:
  - name: dynatrace
   type: http
   parameters:
    host: <environmentID>.live.dynatrace.com
    Port: "443"
    allow_duplicated_headers: "false"
    header:
     - Content-Type application/json; charset=utf-8
     - Authorization Api-Token <accessToken>
    URI: /api/v2/logs/ingest
    Format: json
    json_date_key: timestamp
    json_date_format: iso8601
    tls: "on"
    tls.verify: "off"
  runtimeLogs:
   logLevel: info
   outputs:
    - dynatrace
  accessLogs:
   outputs:
    - dynatrace
```

For more information about configuration options, see the Fluent Bit documentation for your Fluent Bit version.
## Configure Access Logs
To configure access logs in your log output, you must apply a Message Logging policy to the APIs whose request and response data you want to monitor. If you have previously applied a Message Logging policy to an API instance, you do not need to configure another.
To configure a Message Logging policy, see Configuring Message Logging for Flex Gateway in Connected Mode.
Configuring a Message Logging policy makes access logs visible in both API Manager **Logs** and your third-party log services.
If your access logs are not visible, it is possible that a policy with a higher execution order is causing an error before the Message Logging policy can execute. To reorder policy execution, see Ordering Policies.
## Logging Event Examples
The following log output examples demonstrate possible outputs that may occur for the following events:
  * Graceful shutdowns
  * Service disconnections
  * 

### Graceful Shutdowns
```
[flex-gateway-agent][info] SignalHandler: Signal terminated received
[flex-gateway-agent][info] FilesystemWatcher(/usr/local/share/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] XdsService: Stopping with context done
[flex-gateway-agent][error] flex-gateway-envoy: Stopped with error context canceled
[flex-gateway-agent][info] FilesystemWatcher(/etc/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][info] Agent communication closed
```

### Service Disconnections
```
[flex-gateway-agent][error] WebSocket receiver got error: read tcp 172.17.0.2:60780->100.64.1.37:443: i/o timeout
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][warn] Dropped connection due to error read tcp 172.17.0.2:60780->100.64.1.37:443: i/o timeout
[flex-gateway-agent][info] Reconnecting after connection was dropped
[flex-gateway-agent][error] Error connecting to Websocket Server: dial tcp: lookup arm-mcm2-service.kstg.msap.io on 192.168.65.7:53: dial udp 192.168.65.7:53: connect: network is unreachable
```

### Envoy Crash
```
[flex-gateway-envoy][critical] Caught Segmentation fault, suspect faulting address 0xffff87e8fb89485b
[flex-gateway-envoy][critical] Backtrace (use tools/stack_decode.py to get line numbers):
[flex-gateway-envoy][critical] Envoy version: 36cbae1aeedd8c3b4eb88cd28268d21fe0905ce5/1.23.0/Clean/RELEASE/BoringSSL
[flex-gateway-envoy][critical] #0: __restore_rt [0x7ff79db2f140]
[flex-gateway-envoy][critical] #1: [0x557dcfad249b]
[flex-gateway-envoy][critical] #2: [0x557dcdf35237]
[flex-gateway-envoy][critical] #3: [0x557dce16d49f]
[flex-gateway-envoy][critical] #4: [0x557dce1a8ff5]
[flex-gateway-envoy][critical] #5: [0x557dce1a8e47]
[flex-gateway-envoy][critical] #6: ...
[flex-gateway-agent][error] flex-gateway-envoy: Stopped with error flex-gateway-envoy: command error: signal: segmentation fault
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][info] FilesystemWatcher(/etc/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] FilesystemWatcher(/usr/local/share/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] XdsService: Stopping with context done
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][info] Agent communication closed
```

## See Also
  * Declarative Configuration Reference Guide
  * Configuring Message Logging for Flex Gateway in Connected Mode
  * Disabling Flex Gateway Log Output to Anypoint Platform