Project: Vector (Observability and Analysis / Observability)
File: vector-remap-language.md

title: Vector Remap Language short: VRL description: A fast and safe way to transform observability data authors: ["binarylogic"] date: "2021-02-15" badges: type: announcement domains: ["remap"] tags: ["vector remap language", "vrl", "dsl", "expression oriented"] Vector Remap Language (VRL) is an language designed to work with observability data (logs and metrics) in a safe and performant manner. It features a , a rich set of tailored to observability use cases, and that set it far apart from other options. This release of Vector marks the official release of the language. <!--more--> VRL has been under intense but careful development for several months. We made it available in beta beginning with the release, but as of this release VRL is now generally available (GA). We're confident that it's a major step forward for Vector and its users and the observability space in general. Special thanks to and for spearheading the VRL project. *To jump right into the language, . Otherwise, read on for why we created the language.* I know what you may be thinking: another custom language? With so many out there, some of them poorly done, challenging to learn, and just plain unnecessary, you can be forgiven for being wary of another one on the pile. Rest assured that we decided to create VRL only when it became clear that a new language was the best way to improve upon our existing story for transforming data. But it's important to know at the outset that VRL is *not a programming language* in the fullest sense. It lacks a wide range of programming constructs that you find in all-purpose languages, such as loops, classes, modules, custom functions, and IO access. This simplicity is intentional. We're confident that Vector users will not chafe at the limitations of VRL and will in fact embrace them as guarantors of sound design. Up to the 0.12 release, Vector has provided two types of transforms: static transforms and runtime transforms. Static transforms do exactly one* particular thing and are configuration based. The now-deprecated `remove_fields` transform, for example, would remove the fields that you specify in your Vector configuration. Runtime transforms enable you to modify event data using full-blown language runtimes, such as . Both of these options have enabled our users to transform their data successfully, but they have severe limitations that we needed to address with VRL: Static transforms, while fast, are rigid, limited, and hard to manage. They leverage a configuration syntax for expressing data transformations. As a result, we've witnessed otherwise simple pipelines turn into hundreds of lines of configuration. Runtime transforms are very robust, but users pay a steep performance and safety cost when using them. The robustness of using full-blown programming languages makes it very easy to write slow and error-prone programs that are difficult to collaborate on across a team. The classic quadrant analysis looks something like this: {{< svg "/img/blog/vrl-quadrant-comparison.svg" >}} VRL eliminates this trade-off. Let's dig deeper on each of these points. It's commonplace for observability pipelines to offer a rigid list of static transforms that leverage a configuration syntax. Some call them "transforms", others call them "filters" or "functions". If you zoom in, they make more sense -- they perform a single task and make it difficult for users to do the wrong thing. If you zoom out, they start to look like poorly designed programming languages trying to be both a configuration and data transformation

{{< quote >}} If you zoom out, they start to look like poorly designed programming languages, trying to be both a configuration and data transformation language. {{< /quote >}} If you've worked with observability pipelines, it's not uncommon to have hundreds of lines of configuration for relatively simple pipelines. To demonstrate, let's transform a common-log (Apache) coming from Docker, a very common use case: ```json { "time": "2021-02-03T21:13:54.713161211Z", "stream": "stdout", "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574" } ``` Parsing this log, without VRL, looks like this: {{< tabs default="Vector" >}} {{< tab title="Vector" >}} ```yaml title="vector.yaml" transforms: parse_syslog: type: regex_parser inputs: parse_docker patterns: '^(?P<host>\S+) (?P<client>\S+) (?P<user>\S+) \[(?<timestamp>[\w:/]+\s[+\-]\d{4})\] "(?<method>\S+) (?<resource>.+?) (?<protocol>\S+)" (?<status>\d{3}) (?<bytes_out>\S+)$' field: log remove_log: type: remove_fields inputs: parse_syslog fields: time log coerce_fields: type: coercer inputs: remove_log types: timestamp: timestamp status: int bytes_out: int ``` {{< /tab >}} {{< tab title="Logstash" >}} ```text title="logstash.conf" filter { grok { match => { "message" => }( \"%{DATA:[agent]}\")?", "%{IPORHOST:[response_code]} -" ] } remove_field => "message" } mutate { remove_field => [ "time", "log" ] } date { match => [time]", "dd/MMM/YYYY:H:m:s Z" ] remove_field => "[time]" } mutate { coerce => { "" => "integer" "[response_code]" => "integer" } } } ``` {{< /tab >}} {{< tab title="Fluentd" >}} ```text title="fluentd.conf" <source> format apache2 tag apache.access </source> <parse> @type regexp expression /^(?<host>[^ ]) [^ ] (?<user>[^ ]) \[(?<time>[^\]])\] "(?<method>\S+)(?: +(?<path>[^ ]) +\S)?" (?<code>[^ ]) (?<size>[^ ])(?: "(?<referer>[^\"])" "(?<agent>[^\"])")?$/ time_format %d/%b/%Y:%H:%M:%S %z types code:integer,size:integer </parse> <filter apache.access> @type record_transformer remove_keys time,log </filter> ``` {{< /tab >}} {{< /tabs >}} Which results in this general output: ```json { "bytes_out": 20574, "host": "5.86.210.12", "method": "GET", "resource": "/embrace/supply-chains/dynamic/vertical", "protocol": "HTTP/1.0", "status": 201, "timestamp": "2021-02-03T23:13:55Z", "client": "-", "user": "zieme4647" } ``` As you can see, configuration languages become verbose for even simple pipelines, making them non-ideal for expressing data transformations. This is because they have competing concerns: the better a language is at configuration, the worse it is at data transformation. {{< quote >}} The better a language is at configuration, the worse it is at data transformation. {{< /quote >}} Instead of conflating these concerns, like Logstash, Fluentd, and others, Vector separates them, allowing you to choose your preferred configuration language (YAML, TOML or JSON) while offering a purpose-built language for data transformation (VRL). But is VRL really necessary? Couldn't you leverage Lua, JavaScript, or any other existing language? Runtime transforms enable you to modify event data using the full power of a programming language runtime, such as Lua or JavaScript. They're robust enough to handle even the thorniest use cases, but they have significant downsides that make them high-risk for critical infrastructure like observability pipelines: First, you pay a significant performance penalty. For example, on average, Lua is about 60% slower than Vector's Rust-based static transforms, and other runtimes like JavaScript are even slower. Second, there are severe security and safety risks that make them show stoppers for critical infrastructure like observability pipelines. Things like the lack of memory-safety, dynamic code evaluation, access to IO, unaudited dependency trees, lack of sandboxing, and all of the security risks that come with that language runtime. Finally, the extreme optionality presents easy foot-guns that manifest into real-world performance and reliability problems, making them difficult to manage. Runtime transforms acted as robust escape hatches for Vector users, unblocking exotic use cases and allowing us to learn more about the many ways people use

Still, we've seen the dark side of using them: Unexpected malformed data bringing pipelines down and waking operators up in the middle of the night. A poorly managed dependency tree introducing severe security risks, compromising their most sensitive data. Slow performance unable to keep up with data volume fluctuations. Complicated code leaving pipelines in an unmanageable state. Because of these risks we recommend static transforms if given the choice, but you shouldn't have to choose... Before I dive into the specifics of VRL, let's look at the Docker, common-log (Apache) example from above. If you skipped to this section we'll be parsing the following log: ```json { "time": "2021-02-03T21:13:54.713161211Z", "stream": "stdout", "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574" } ``` Which can be achieved with the following VRL program: ```coffee . = parsecommonlog!(.log) .total_bytes = del(.size) .internalrequest = ipcidr_contains("5.86.0.0/16", .host) ?? false ``` Resulting in this output: ```json { "host": "5.86.210.12", "internal_request": true, "user": "zieme4647", "timestamp": "2021-02-03T23:13:55Z", "message": "GET /embrace/supply-chains/dynamic/vertical HTTP/1.0", "method": "GET", "path": "/embrace/supply-chains/dynamic/vertical", "protocol": "HTTP/1.0", "total_bytes": 20574, "status": 201 } ``` As you can see, the above configuration is significantly easier to write, read, and manage. Also, it maintains optimal performance and doesn't present the problems that come with runtime transforms like Lua and JavaScript. You'll notice we threw in an extra field, `internal_request`, to demonstrate how VRL solves otherwise difficult tasks for static transforms. The syntax was originally inspired by [jq] (note the `.` based paths), but evolved into a simple multi-line language. For a deep dive into the language constructs, check out the . Here you will find a comprehensive list of , , and . VRL was designed to exploit two principles: safety and performance, while maintaining flexibility. This makes VRL ideal for always-up, performance-sensitive infrastructure like observability pipelines. To illustrate how we achieve this, below is a VRL feature matrix across these two principles: | Feature | Safety | Performance | |:--|::|:--:| | [Progressive type safety] | | | | [Fail safety] | | | | [Memory safety] | | | | [Ergonomic safety] | | | | [Vector/Rust native] | | | | [Stateless] | | | For more info on each click on the feature, but for the purposes of demonstrating how VRL is unique, let's touch on the first two: *progressive type safety and fail safety*. A unique design decision behind VRL is its implementation of type and fail safety. After seeing many users deal with pipeline instability due to runtime errors, we made type and fail safety cornerstones. This makes VRL programs infallible, ensuring that they will work as expected in production. Let's demonstrate with an example, using the same Docker, common-log (Apache) example from above. Given this log: ```json { "time":"2021-02-03T21:13:54.713161211Z", "stream": "stdout", "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574" } ``` We want to parse it into this result: ```json { "host": "5.86.210.12", "user": "zieme4647", "timestamp": "2021-02-03T23:13:55Z", "message": "GET /embrace/supply-chains/dynamic/vertical HTTP/1.0", "method": "GET", "path": "/embrace/supply-chains/dynamic/vertical", "protocol": "HTTP/1.0", "total_bytes": 20574, "status": 201 } ``` Someone new to VRL might write the following VRL program: ```coffee . = parsecommonlog(.log) .total_bytes = del(.size) ``` And they'll be greeted with this thoughtful error message at compile-time (Vector boot): ```rust error[E103]: unhandled fallible assignment :1:5 1 . = parsecommonlog(.log) ^^^^^^^^^^^^^^^^^^^^^^ this expression is fallible update the expression to be infallible or change this to an infallible assignment: ., err =

= see documentation about error handling at https://errors.vrl.dev/#handling = learn more about error code 103 at https://errors.vrl.dev/103 = see language documentation at https://vrl.dev ``` As you can see, VRL requires the user to handle any expression that can result in a runtime error. In our case, `.log` might not be a string, so we need to either specify the type of the `.log` field or handle the error in the event that `.log` is not a string. To resolve this error, the user must do one of three things: Handle the error ```coffee ., err = parsecommonlog(.log) if err != null { .malformed = true log("Failed to parse common-log: " + err, level: "error") } else { .total_bytes = del(.size) } ``` While subtle, this has valuable implications! If the event is malformed (`.log` is not a formatted common-log string), then we forgo log an error and add a `.malformed` field. This preserves the original data and makes it easy to the malformed data for inspection. Not handling this error is a very common mistake that would otherwise result in data loss and downtime. Raise the error and abort ```coffee . = parsecommonlog!(.log) .total_bytes = del(.size) ``` In some cases, malformed data is unacceptable and the program must be aborted. In this case VRL offers fallible function variants. This is a fancy way of saying that suffixing your function call with `!` will raise an error upon failure and abort the program. Vector itself will continue on to process the next event, but will also log an error message to alert operators. Again, this forces the users to decide how to handle errors instead of being surprised by them. Specify types ```coffee .log = to_string!(.log) ., err = parsecommonlog(.log) if err != null { log("Failed to parse common-log: " + err, level: "error") } else { .total_bytes = del(.size) } ``` Finally, a caveat of the two examples above is that it's hard to discern between a type error and a parsing error. Perhaps you want to abort on type errors and handle parsing errors? You can achieve this through *progressive type safety*. As a VRL program evaluates, it builds up type information about your fields. Once the type is known, subsequent usage of that field forgoes runtime type errors. Therefore, in the above example we know that the only possible way an error can occur is if parsing fails. We're just scratching the surface on the language. For more information, check out the VRL , , , and . The short answer is an emphatic yes. For the use cases that VRL can coverand that should be most use casesit should be seen as the preferred option beginning with this release. If you aren't sure if your use case is covered, don't ever hesitate to visit us . Although VRL should be considered ready for production use cases, there's more on the way in the next few releases: The initial lineup of VRL functions can ably cover many use cases but we do have more functions in the pipeline. Expect major new features for simple object traversal. A web playground that enables you to experiment with VRL in the browser. Improved type safety and ergonomics through event schemas. So stay tuned for much more work in this area. But for now, we strongly encourage to explore the and get in touch with us if you have any issues, comments, or suggestions.