Project: Vector (Observability and Analysis / Observability)
File: 2022-05-18-vrl-iteration-support.md

title: "VRL iteration support has arrived!" short: "VRL iteration" description: "Remap all your collections using iteration functions." authors: ["JeanMertz"] date: "2022-05-18" pr_numbers: [12317] release: "0.22.0" hideonrelease_notes: false badges: type: announcement domains: ["vrl", "remap"] tags: ["vector remap language", "vrl", "dsl", "iteration", "enumeration"] Since we launched Vector Remap Language (VRL) , the language has seen a massive growth in both usage, and feature richness. While we intentionally left the core language in a stable state since its launch, we have been steadily expanding the , increasing , adding , and [squashing many bugs][]. One of the last major pieces missing from the language and one requested by many of you is support for mapping collections. We needed to introduce this feature thoughtfully since mapping can be a common source of performance and reliability bugs (i.e., infinite loops). And our design accomplishes this by providing opinionated mapping functions that align closely with the use case. So, without further ado, we're happy to announce that as of Vector 0.22.0, mapping for collections is now available! To explain how collection mapping in VRL works, well dissect a simple example: ```json { "foo": true, "bar": { "bar": "" }, "baz": "", "qux": ["a", "b", "a", "c"] } ``` Lets assume we want to achieve the following results: Upcase all keys in the object. Change all empty string values to `null`. Count the frequency of elements in the `qux` array. Heres how we would solve these individual tasks: ```coffee . = map_keys(., recursive: true) -> |key| { upcase(key) } . = map_values(., recursive: true) -> |value| { if value == { null } else { value } } .qux_tally = {} foreach(.qux) -> |index, value| { tally = int(get!(.qux_tally, [value])) ?? 0 .quxtally = set!(.quxtally, [value], tally + 1) } ``` Running this VRL program results in the following output: ```json { "FOO": true, "BAR": { "BAR": null }, "BAZ": null, "QUX": ["a", "b", "a", "c"], "qux_tally": { "a": 2, "b": 1, "c": 1 } } ``` For a longer list of examples, see our and . As you can see, VRL iteration happens through regular function calls, combined with the new closure syntax. We chose this solution for a few reasons: Allows us to expand the standard library with more specialized iteration functions going forward. Prevents introducing accidental infinite recursions to your program. Provides enough flexibility for manipulating observability data, without adding overly complex special-purpose iteration syntax. The three functions we introduce in this release (`mapkeys`, `mapvalues`, and `for_each`) serve as a good starting point for any generic iteration logic, but we already have a list of special-purpose iteration functions available in the standard library, and plan to add more, as long as they fit the purpose of the language, remapping observability data. Take a look at our , and feel free to [file a request][] for any special purpose function that would make your program easier to maintain and/or more performant to run. For more details on the design of this feature, you can read our technical . We hope you are as happy with this enhancement to VRL as we are. Iteration support unlocks one of the last areas for which many of you had to fall back to the . You can now keep your remapping logic in VRL, with all the performance and runtime correctness guarantees VRL delivers. Please or tweet at us at to discuss this, or any other aspect of Vector and VRL.