Project: Vector (Observability and Analysis / Observability)
File: instrumentation.md

This document specifies Vector's instrumentation for the development of Vector. The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in [RFC 2119]. - - - - Vector's telemetry drives various interfaces that operators depend on to manage mission critical Vector deployments. Therefore, Vector's telemetry should be high quality and treated as a first class feature in the development of Vector. This document strives to guide developers towards achieving this. Events and metrics are namespaces as one of: `Component` `Buffer` `Topology` Depending on where they are emitted. Vector implements an event-driven instrumentation pattern ([RFC 2064]) and event names MUST adhere to the following rules: MUST only contain ASCII alphanumeric and lowercase characters MUST be in [camelcase] format MUST follow the `<Namespace><Noun><Verb>[Error]` template `Namespace` - the the event belongs to `Noun` - the subject of the event (e.g., `Bytes`, `Events`) `Verb` - the past tense verb describing when the event occurred (e.g., `Received`, `Sent`, `Processes`) `[Error]` - if the event is an error it MUST end with `Error` Vector broadly follows the [Prometheus metric naming standards]: MUST only contain ASCII alphanumeric, lowercase, and underscore characters MUST be in [snakecase] format MUST follow the `<namespace><name><unit>_[total]` template `namespace` - the the event belongs to `name` - is one or more words that describes the measurement (e.g., `memory_rss`, `requests`) `unit` - MUST be a single [base unit] in plural form, if applicable (e.g., `seconds`, `bytes`) Counters MUST end with `total` (e.g., `diskwrittenbytestotal`, `httprequests_total`) SHOULD be broad in purpose and use tags to differentiate characteristics of the measurement (e.g., `hostcpuseconds_total{cpu="0",mode="idle"}`) For performance reasons, as demonstrated in [pull request #8383], instrumentation SHOULD be batched whenever possible: Telemetry SHOULD emit for entire event batches, not each individual event. [RFC 9480] describes Vector's batching strategy. Benchmarking SHOULD prove that batching produces performance benefits. [Issue 10658] could eliminate the need to batch for performance improvements. Instrumentation SHOULD be event-driven ([RFC 2064]), where individual events serve as the vehicle for internal telemetry, driving the emission of metrics and logs. This organizes Vector's telemetry, making it easier to manage and catalogue. Metrics and logs SHOULD NOT be emitted directly except for where it is otherwise impossible to emit Vector's events, such as in an external crate that cannot import Vector's events. An `<Name>Error` event MUST be emitted when an error occurs during the running of a component. If an error occurs that prevents the component from starting up an event does not need to be emitted as this will prevent Vector from starting and the metric is unlikely to be collected. An error should still be logged, however. Properties `error_code` - An error code for the failure, if applicable. SHOULD only be specified if it adds additional information beyond

The values for `error_code` for a given error event MUST be a bounded set with relatively low cardinality because it will be used as a metric tag. Examples would be syscall error code. Examples of values that should not be used are raw error messages from `serde` as these are highly variable depending on the input. Instead, these errors should be converted to an error code like `invalid_json`. `error_type` - The type of error condition. MUST be one of the types listed in the `error_type` enum list in the cue docs. If any of the above properties are implicit to the specific error type, they MAY be omitted from being represented explicitly in the event fields. However, they MUST still be included in the emitted logs and metrics, as specified below, as if they were present. `stage` - The stage at which the error occurred. This MUST be one of `receiving`, `processing`, or `sending`. Metrics MUST include the defined properties as tags. MUST increment `<namespace>errorstotal` metric. Logs MUST log a descriptive, user-friendly error message that sufficiently describes the error. MUST include the defined properties as key-value pairs. MUST log a message at the `error` level. SHOULD be rate limited to 10 seconds. Events MUST emit an [`EventsDropped`] event if the error results in dropping events. An `<Namespace>EventsDropped` event MUST be emitted when events are dropped. If events are dropped due to an error, then the error event should drive the emission of this event, meeting the below requirements. This event MUST NOT be emitted before events have been created in Vector. For example a source failing to decode incoming data would simply emit the `ComponentError` event but would not emit the `ComponentEventsDropped` event. You MUST NOT emit this event for operations that Vector will retry to prevent data loss. For example, a failed HTTP request that will be retried does not result in data loss if the retry succeeds. Note that this event is independent of any clients of Vector that may retry when end-to-end acknowledgements are enabled. From Vector's perspective, it has dropped the events, and it cannot know if the client will retry them. Properties `count` - The number of events dropped `intentional` - Distinguishes if the events were dropped intentionally. For example, events dropped in the `filter` transform are intentionally dropped, while events dropped due to an error in the `remap` transform are unintentionally dropped. `reason` - A short, user-friendly reason that describes why the events were dropped. Metrics MUST increment the `<namespace>discardedevents_total` counter by the number of events discarded. MUST only include the `intentional` property and component properties that are inherited implicitly (e.g. `component_type`). Logs MUST log a `Events dropped` message. MUST include the defined properties as key-value pairs. If `intentional` is `true`, MUST log at the `debug` level. If `intentional` is `false`, MUST log at the `error` level. SHOULD be rate limited to 10 seconds.