Project: CockroachDB (App Definition and Development / Database)
File: logsinks.md

The supported log output sink types are documented below. <a name="output-to-files"> This sink type causes logging data to be captured into log files in a configurable logging directory. The configuration key under the `sinks` key in the YAML configuration is `file-groups`. Example configuration: // sinks: // file-groups: # file group configurations start here // health: # defines one group called "health" // channels: HEALTH Each generated log file is prefixed by the name of the process, followed by the name of the group, separated by a hyphen. For example, the group `health` will generate files named `cockroach-health.XXX.log`, assuming the process is named `cockroach`. (A user can influence the prefix by renaming the program executable.) The files are named so that a lexicographical sort of the directory contents presents the file in creation order. A symlink (e.g. `cockroach-health.log`) for each group points to the latest generated log file. Every new file group sink configured automatically inherits the configurations set in the `file-defaults` section. For example: // file-defaults: // redactable: false # default: disable redaction markers // dir: logs // sinks: // file-groups: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from file-defaults // # unless overridden here. // # // # Example override: // dir: health-logs # override the default 'logs' {{site.data.alerts.callout_success}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}} Type-specific configuration options: | Field | Description | |--|--| | `channels` | the list of logging channels that use this sink. See the section for details. | | `dir` | specifies the output directory for files generated by this sink. Inherited from `file-defaults.dir` if not specified. | | `max-file-size` | the approximate maximum size of individual files generated by this sink. If zero, there is no maximum size. Inherited from `file-defaults.max-file-size` if not specified. | | `max-group-size` | the approximate maximum combined size of all files to be preserved for this sink. An asynchronous garbage collection removes files that cause the file set to grow beyond this specified size. If zero, old files are not removed. Inherited from `file-defaults.max-group-size` if not specified. | | `file-permissions` | the "chmod-style" permissions the log files are created with as a 3-digit octal number. The executable bit must not be set. Defaults to 644 (readable by all, writable by owner). Inherited from `file-defaults.file-permissions` if not specified. | | `buffered-writes` | specifies whether to buffer log entries. Setting this to false flushes log writes upon every entry. Inherited from `file-defaults.buffered-writes` if not specified. | Configuration options shared across all sink types: | Field | Description | |--|--| | `filter` | specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. | | `format` | the entry format to use. | | `format-options` | additional options for the format. | | `redact` | whether to strip sensitive information before log events are emitted to this

| | `redactable` | whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. | | `exit-on-error` | whether the logging system should terminate the process if an error is encountered while writing to this sink. | | `auditable` | translated to tweaks to the other settings for this sink during validation. For example, it enables `exit-on-error` and changes the format of files from `crdb-v1` to `crdb-v1-count`. | | `buffering` | configures buffering for this log sink, or NONE to explicitly disable. See the section for details. | <a name="output-to-fluentd-compatible-log-collectors"> This sink type causes logging data to be sent over the network to a log collector that can ingest log data in a -compatible protocol. {{site.data.alerts.callout_danger}} TLS is not supported yet: the connection to the log collector is neither authenticated nor encrypted. Given that logging events may contain sensitive information, care should be taken to keep the log collector and the CockroachDB node close together on a private network, or connect them using a secure VPN. TLS support may be added at a later date. {{site.data.alerts.end}} At the time of this writing, a Fluent sink buffers at most one log entry and retries sending the event at most one time if a network error is encountered. This is just sufficient to tolerate a restart of the Fluentd collector after a configuration change under light logging activity. If the server is unavailable for too long, or if more than one error is encountered, an error is reported to the process's standard error output with a copy of the logging event and the logging event is dropped. The configuration key under the `sinks` key in the YAML configuration is `fluent-servers`. Example configuration: // sinks: // fluent-servers: # fluent configurations start here // health: # defines one sink called "health" // channels: HEALTH // address: 127.0.0.1:5170 Every new server sink configured automatically inherits the configurations set in the `fluent-defaults` section. For example: // fluent-defaults: // redactable: false # default: disable redaction markers // sinks: // fluent-servers: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from fluent-defaults // # unless overridden here. The default output format for Fluent sinks is `json-fluent-compact`. The `fluent` variants of the JSON formats include a `tag` field as required by the Fluentd protocol, which the non-`fluent` JSON do not include. {{site.data.alerts.callout_info}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}} Type-specific configuration options: | Field | Description | |--|--| | `channels` | the list of logging channels that use this sink. See the section for details. | | `net` | the protocol for the fluent server. Can be "tcp", "udp", "tcp4", etc. | | `address` | the network address of the fluent server. The host/address and port parts are separated with a colon. IPv6 numeric addresses should be included within square brackets, e.g.:

| Configuration options shared across all sink types: | Field | Description | |--|--| | `filter` | specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. | | `format` | the entry format to use. | | `format-options` | additional options for the format. | | `redact` | whether to strip sensitive information before log events are emitted to this sink. | | `redactable` | whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. | | `exit-on-error` | whether the logging system should terminate the process if an error is encountered while writing to this sink. | | `auditable` | translated to tweaks to the other settings for this sink during validation. For example, it enables `exit-on-error` and changes the format of files from `crdb-v1` to `crdb-v1-count`. | | `buffering` | configures buffering for this log sink, or NONE to explicitly disable. See the section for details. | <a name="output-to-http-servers."> This sink type causes logging data to be sent over the network as requests to an HTTP server. The configuration key under the `sinks` key in the YAML configuration is `http-servers`. Example configuration: // sinks: // http-servers: // health: // channels: HEALTH // address: http://127.0.0.1 Every new server sink configured automatically inherits the configuration set in the `http-defaults` section. For example: // http-defaults: // redactable: false # default: disable redaction markers // sinks: // http-servers: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from fluent-defaults // # unless overridden here. The default output format for HTTP sinks is `json-compact`. {{site.data.alerts.callout_info}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}} Type-specific configuration options: | Field | Description | |--|--| | `channels` | the list of logging channels that use this sink. See the section for details. | | `address` | the network address of the http server. The host/address and port parts are separated with a colon. IPv6 numeric addresses should be included within square brackets, e.g.: [::1]:1234. Inherited from `http-defaults.address` if not specified. | | `method` | the HTTP method to be used. POST and GET are supported; defaults to POST. Inherited from `http-defaults.method` if not specified. | | `unsafe-tls` | enables certificate authentication to be bypassed. Defaults to false. Inherited from `http-defaults.unsafe-tls` if not specified. | | `timeout` | the HTTP timeout. Defaults to 0 for no timeout. Inherited from `http-defaults.timeout` if not specified. | | `disable-keep-alives` | causes the logging sink to re-establish a new connection for every outgoing log message. This option is intended for testing only and can cause excessive network overhead in production systems. Inherited from `http-defaults.disable-keep-alives` if not specified. | | `headers` | a list of headers to attach to each HTTP request Inherited from `http-defaults.headers` if not

| | `file-based-headers` | a list of headers with filepaths whose contents are attached to each HTTP request Inherited from `http-defaults.file-based-headers` if not specified. | | `compression` | can be "none" or "gzip" to enable gzip compression. Set to "gzip" by default. Inherited from `http-defaults.compression` if not specified. | Configuration options shared across all sink types: | Field | Description | |--|--| | `filter` | specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. | | `format` | the entry format to use. | | `format-options` | additional options for the format. | | `redact` | whether to strip sensitive information before log events are emitted to this sink. | | `redactable` | whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. | | `exit-on-error` | whether the logging system should terminate the process if an error is encountered while writing to this sink. | | `auditable` | translated to tweaks to the other settings for this sink during validation. For example, it enables `exit-on-error` and changes the format of files from `crdb-v1` to `crdb-v1-count`. | | `buffering` | configures buffering for this log sink, or NONE to explicitly disable. See the section for details. | <a name="standard-error-stream"> The standard error output stream of the running `cockroach` process. The configuration key under the `sinks` key in the YAML configuration is `stderr`. Example configuration: // sinks: // stderr: # standard error sink configuration starts here // channels: DEV {{site.data.alerts.callout_info}} The server start-up messages are still emitted at the start of the standard error stream even when logging to `stderr` is enabled. This makes it generally difficult to automate integration of `stderr` with log analyzers. Generally, we recommend using or instead of `stderr` when integrating with automated monitoring software. {{site.data.alerts.end}} It is not possible to enable the `redactable` parameter on the `stderr` sink if `capture-stray-errors` (i.e., capturing stray error information to files) is disabled. This is because when `capture-stray-errors` is disabled, the process's standard error stream can contain an arbitrary interleaving of and stray errors. It is possible for stray error output to interfere with redaction markers and remove the guarantees that information outside of redaction markers does not contain sensitive information. For a similar reason, no guarantee of parsability of the output format is available when `capture-stray-errors` is disabled, since the standard error stream can then contain an arbitrary interleaving of non-formatted error data. Type-specific configuration options: | Field | Description | |--|--| | `channels` | the list of logging channels that use this sink. See the section for details. | | `no-color` | forces the omission of VT color codes in the output even when stderr is a terminal. This option is deprecated; its effects are equivalent to 'format-options: {colors:

| Configuration options shared across all sink types: | Field | Description | |--|--| | `filter` | specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. | | `format` | the entry format to use. | | `format-options` | additional options for the format. | | `redact` | whether to strip sensitive information before log events are emitted to this sink. | | `redactable` | whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. | | `exit-on-error` | whether the logging system should terminate the process if an error is encountered while writing to this sink. | | `auditable` | translated to tweaks to the other settings for this sink during validation. For example, it enables `exit-on-error` and changes the format of files from `crdb-v1` to `crdb-v1-count`. | | `buffering` | configures buffering for this log sink, or NONE to explicitly disable. See the section for details. | <a name="channel-format"> Each sink can select multiple channels. The names of selected channels can be specified as a YAML array or as a string. Additionally, severity filters can be applied separately for different groups of channels. Example configurations: channels: [OPS, HEALTH] channels: {INFO: [PERF], WARNING: [HEALTH, OPS]} channels: OPS channels: {INFO: PERF} channels: [ops, HeAlTh] channels: OPS,HEALTH channels: 'OPS, HEALTH' channels: OPS HEALTH channels: INFO: PERF WARNING: OPS HEALTH It is also possible to select all channels, using the "all" keyword. For example: channels: all channels: 'all' channels: [all] channels: ['all'] Likewise: channels: {INFO: all} etc. It is also possible to select all channels except for a subset, using the "all except" keyword prefix. This makes it possible to define sinks that capture "everything else". For example: channels: all except ops,health channels: all except [ops,health] channels: 'all except ops, health' channels: 'all except [ops, health]' Likewise: channels: {INFO: all except ops,health} etc. <a name="buffering-config"> Buffering may be configured with the following fields. It may also be explicitly set to "NONE" to disable buffering. Example configuration: // file-defaults: // dir: logs // buffering: // max-staleness: 20s // flush-trigger-size: 25KB // max-buffer-size: 10MB // sinks: // file-groups: // health: // channels: HEALTH // buffering: // max-staleness: 5s # Override max-staleness for this sink. // ops: // channels: OPS // buffering: NONE # Disable buffering for this sink. | Field | Description | |--|--| | `max-staleness` | the maximum time a log message will sit in the buffer before a flush is triggered. | | `flush-trigger-size` | the number of bytes that will trigger the buffer to flush. | | `max-buffer-size` | the limit on the size of the messages that are buffered. If this limit is exceeded, messages are dropped. The limit is expected to be higher than FlushTriggerSize. A buffer is flushed as soon as FlushTriggerSize is reached, and a new buffer is created once the flushing is started. Only one flushing operation is active at a time. | | `format` | describes how the buffer output should be formatted. Currently 2 options: newline: default option - separates buffer entries with newline char json-array: separates entries with ',' and wraps buffer contents in square brackets |