Project: Vector (Observability and Analysis / Observability)
File: architecting.md

title: Architecting your Deployment description: Best practices for deploying Vector into production environments. short: Architecting weight: 1 Well start with , discuss in your architecture, and end with how to your pipeline. {{< info >}} For simple deployment of these architectures, please see our . {{< /info >}} While Vector aims to help you reduce the number of tools necessary to build powerful observability pipelines, some tools may be better suited to the task at hand (i.e., the Datadog Agent for collecting data). Dont hesitate to stick with other tools while adding Vector to your observability stack. Vectors flexibility ensures that if the time comes to remove or replace certain tools from your stack, it can be done as seamlessly as possible. To offset the risk of deploying different agents, agents should shift as much responsibility to Vector as possible. Agents should be simple data forwarders, performing only the task they do best and delegating the rest to Vector. This strategy minimizes your dependence on agents and reduces the risk of data loss and service disruption. Finally, Vectors small footprint and shared-nothing architecture make it possible to deploy Vector anywhere in your infrastructure. To avoid creating a single point of failure (SPOF), you should deploy Vector close to your data and within your network boundaries. This not only makes your observability data pipeline more reliable but reduces the blast radius of any single Vector failure. Vector can deploy anywhere in your infrastructure, serving different roles. For example, it can deploy directly on your nodes as an or on its own nodes as an . Vector's agent role deploys Vector directly on each node for local data collection and processing: It can directly collect data from the node, indirectly through another agent, or both. Data processing can happen locally in the agent or remotely in an aggregator. {{< info >}} Deploying Vector as an agent is demonstrated in the . {{< /info >}} Vector's aggregator role deploys Vector on dedicated nodes to receive data from upstream agents or pull data from pub-sub services: Processing is centralized in your aggregators, on nodes optimized for performance. {{< info >}} Deploying Vector as an aggregator is demonstrated in the . {{< /info >}} Before we consider , its helpful to understand the design characteristics used to form them. This helps you choose a particular architecture and tweak it to fit your unique requirements. Deploying Vector raises several networking concerns, such as , , , and . When deploying Vector as an , we recommend deploying within your network boundaries (i.e., within each cluster or VPC) across accounts and regions, even if this means deploying multiple aggregators. This strategy avoids introducing a single point of failure, allows for easy and secure internal communication, and respects the boundaries maintained by your network administrator. Management of Vector should be centralized for operational simplicity. Even though Vector distributes physically across your infrastructure, it is conceptually unified via shared configuration. The same configuration can be delivered to all instances unifying them in behavior, acting as a single aggregator. Consider Datadog Observability Pipelines, Vectors hosted control plane, for simplified management of multiple aggregators. {{< warning >}} Avoid deploying a single monolith aggregator since it introduces a single point of failure. Vectors shared-nothing architecture eliminates coordination (i.e., master/leader nodes) that would require

Instead, deploy Vector within your network boundaries and treat it like a single aggregator via shared configuration. {{< /warning >}} We recommend following our . Restrict agent communication to your aggregators, and restrict aggregator communication to your configured sources and sinks. For HTTP proxying, we do not recommend one way or another. However, if you prefer to use a HTTP proxy, Vector offers a global `proxy` option making it easy to route all Vector HTTP traffic through a proxy. Discovery of your Vector aggregators and services should resolve through DNS or service discovery. This strategy facilitates routing and load balancing of your traffic, it is how your agents and load balancers discover your aggregators. {{< info >}} For proper separation of concerns, Vector itself does not resolve DNS queries and, instead, delegates this to a system-level resolver (i.e., ). {{< /info >}} When sending data to Vector, we recommend choosing a protocol that allows easy load-balancing and application-level delivery acknowledgement. We prefer HTTP and gRPC due to their ubiquitous nature and the amount of available tools and documentation to help operate HTTP/gRPC-based services effectively and efficiently. Choose the source that aligns with your protocol. Each Vector source implement different protocols. For example, `vector` source and sink use gRPC and make inter-Vector communication easy, and the `http` source allows you to receive data over HTTP. See the list of source for their respective protocols. Your pipeline begins with data collection. Your services and systems generate logs, metrics, and traces that must be collected and sent downstream to your destinations. Data collection is achieved with agents, and understanding which agents to use will ensure youre reliably collecting the right data. In line with , it is best to choose the agent that optimizes your engineering team's ability to monitor their systems. Sacrificing this to use Vector is not recommended since Vector can be deployed in between agents to reduce lock-in. Therefore, Vector should with the best agent for the job and others. Specific guidance is below. Vector should replace agents performing generic data forwarding functions, such as: Tailing and forwarding log files Collecting and forwarding service metrics without enrichment Collecting and forwarding service logs without enrichment Collecting and forwarding service traces without enrichment Notice that these functions collect and forward existing data, unchanged. There is nothing unique about these functions. Vector should replace these agents since Vector offers better performance and reliability. Vector should not replace agents that produce vendor-specific data that Vector cannot replicate. For example, the Datadog Agent powers . In this case, the Datadog Agent integrates with vendor-specific systems and produces vendor-specific data. Vector should not be involved with this process. Instead, the Datadog Agent should collect the data and send it directly to Datadog since the data is not one of . As another example, the Datadog Agent collects service metrics and enriches them with vendor-specific Datadog tags. In this case, the Datadog Agent should send the metrics directly to Datadog or route them through Vector. Vector should not replace the Datadog Agent because the data being produced is enriched in a vendor-specific way. More examples are below. | Agent | Action | Rationale | | | | | | Datadog Agent | Integrate | The Datadog Agent integrates with more systems, produces better data, and seamlessly integrates with the Datadog platform. | | File Beat | Replace | Vector can tail files with better performance, reliability, and data

| | Fluentbit | Replace | Vector can perform the same functions as Fluentbit with better performance and reliability. | | FluentD | Replace | Vector can perform the same functions as Fluentd with significantly better performance and reliability. | | Metric Beat | Integrate | Metric Beat integrates with more systems, produces better data, and seamlessly integrates with the Elastic platform. | | New Relic Agent | Integrate | The New Relic Agent integrates with more systems, produces better data, and seamlessly integrates with the New Relic platform. | | Open Telemetry Collector | Integrate | Vector's built-in OTel support is forthcoming. | | Splunk UF | Replace | Vector integrates deeply with Splunk with better performance, reliability, and data quality. | | Syslog | Replace | Vector implements the Syslog protocol with better performance, reliability, and data quality. | | Telegraf | Integrate | Telegraf integrates with more systems and produces better metrics data. | If you decide to integrate with an agent, you should configure Vector to receive data directly from the agent over the local network, routing data through Vector. Sources such as the `datadog_agent` source can be used to receive data from your agents. Alternatively, you can deploy Vector . This strategy is covered in more detail in the . When integrating with an agent, you should configure the agent to be a simple data forwarder, and route through Vector. This strategy reduces the risk of data loss and service disruption due to misbehaving agents by minimizing their responsibilities. {{< warning >}} Vector should only replace agents performing . Otherwise, Vector should with the agent. {{< /warning >}} If you decide to replace an agent, you should configure Vector to perform the same function as the agent you are replacing. Sources such as the `file` source, `journald` source, and `host_metrics` source should be used to collect and forward data. You can process data on the node or on your aggregators. Data processing deals with everything in between your Vector sources and sinks. Its a non-trivial task for voluminous data such as observability data, and understanding and will go a long way in the performance and reliability of your pipeline. Primitive data types that supports should be processed by Vector (i.e., logs, metrics, and traces). Real-time, vendor-specific data, such as continuous profiling data, should not flow through Vector since it is not interoperable and typically does not benefit from processing. As stated in the , Vector can deploy anywhere in your infrastructure. It can deploy directly on your node as an agent for , or on separate nodes as an aggregator for . Where processing happens depends largely on your use cases and environment, which we cover in more detail below. With local processing, : This strategy performs processing on the same node that the data originated. It benefits from operational simplicity since it has direct access to your data and scales naturally with your infrastructure. We recommend this strategy for Simple environments that do not require . Use cases that do not need to hold onto data for long periods, such as fast, stateless processing and streaming delivery. Operators that can make node-level changes without a lot of friction. Since this is usually the exception, consider instead. {{< info >}} Local processing is demonstrated in the . {{< /info >}} For remote processing, : This strategy shifts processing off your nodes and onto remote aggregator

We recommend this strategy for environments that require (most environments). In addition, it is easier to set up since it does not require the infrastructure toil necessary when adding an agent. {{< info >}} Remote processing is demonstrated in the . {{< /info >}} Finally, you can combine the above strategies to get the best of both worlds, creating a unified observability data pipeline. We recommend evolving towards this strategy after starting with . {{< info >}} Unified processing is demonstrated in the . {{< /info >}} Buffering data is essential to a healthy pipeline, but and makes the difference. This section will cover the various ways you can buffer data using Vector and guide you towards sound pipeline buffering. Buffering should happen close to your destinations, and each destination should have its own isolated buffer. This design offers the following benefits: Each destination can configure its buffer to meet the sinks requirements. See the for more detail. Isolating buffers for each destination prevents one misbehaving destination from halting the entire pipeline until the buffer reaches the configured capacity. For these reasons, Vector couples buffers with sinks in its design. Finally, to follow , your agents should be simple data forwarders and shed load in the face of backpressure. This method reduces the risk of service disruption due to agents stealing resources. We recommend using Vector's built-in buffers. Vector offers reliable, robust buffering that simplifies operation and eliminates the need for complex external buffers like Kafka. When choosing a Vector buffer type, select the type optimal for the destinations purpose. For example, your system of record should use disk buffers for high durability, and your system of analysis should use memory buffers for low latency. Additionally, both buffers can overflow to another buffer to prevent back pressure form propagating to your clients. Routing data is the final piece in your pipeline design, and your aggregators should be responsible for sophisticated routing that sends data to the proper destination. We believe strongly in the power of choice, and you should use your aggregators to route data to the best system for your team(s). You should separate your system of record from your system of analysis. This strategy allows you to minimize cost without making trade offs that sacrifice their purpose. For example, your system of record can batch large amounts of data over time and compress it to minimize cost while ensuring high durability for all data. And your system of analysis can sample and clean data to reduce cost while keeping latency low for real-time analysis. Your system of record should optimize for durability while minimizing cost: Only write to your archive from the to reduce data loss due to node restarts and . Front the sink with a disk buffer. Enable on all sources. Set `batch.maxbytes` to 5MiB and `batch.timeoutsecs` to 5 minutes and enable `compression` (the default for archiving sinks, such as the `aws_s3` sink). Archive raw, unprocessed data to allow for data replay and reduce the risk of accidental data corruption during processing. Your system of analysis should optimize for analysis while reducing cost: Front the sink with a memory buffer Set `batch.timeoutsec` to 5 seconds (the default for analysis sinks, such as `datadoglogs`) Remove attributes not used for analysis via the `remap` transform Filter events not used for analysis Consider sampling logs with level `info` or lower to reduce their volume