Project: Vector (Observability and Analysis / Observability)
File: 2022-05-03-0-22-0-upgrade-guide.md

date: "2022-05-03" title: "0.22 Upgrade Guide" description: "An upgrade guide that addresses breaking changes in 0.22.0" authors: ["spencergilbert", "jszwedko"] pr_numbers: [12124] release: "0.22.0" hideonrelease_notes: false badges: type: breaking change Vector's 0.22.0 release includes breaking changes: and deprecations: We cover them below to help you upgrade quickly: The `gcpstackdrivermetrics` sink now matches the `gcpstackdriverlogs` configuration, and doesn't require an additional `labels` section to add labels to submitted metrics. Old configuration ```toml [sinks.mysinkid] type = "gcpstackdrivermetrics" inputs = [ "my-source-or-transform-id" ] credentials_path = "/path/to/credentials.json" project_id = "vector-123456" [sinks.mysinkid.resource] type = "global" [sinks.mysinkid.resource.labels] projectId = "vector-123456" instanceId = "Twilight" zone = "us-central1-a" ``` New configuration ```toml [sinks.mysinkid] type = "gcpstackdrivermetrics" inputs = [ "my-source-or-transform-id" ] credentials_path = "/path/to/credentials.json" project_id = "vector-123456" [sinks.mysinkid.resource] type = "global" projectId = "vector-123456" instanceId = "Twilight" zone = "us-central1-a" ``` For more information on the new syntax, you can review the documentation VRL strings can now be templated. It is now possible to insert the values of variables by inserting a placeholder with an embedded variable name into the string using `{{..}}`. For example in the following code: ```coffee beverage = "coffee" preference = "I love to drink {{ beverage }}!" assert!(preference == "I love to drink coffee!") ``` It should be noted that currently the placeholder must contain a simple variable name and that variable must resolve to a string. This will not work: ```coffee stars = 42 sky = "There are {{ stars }} in the sky." ``` Instead, the variable must be converted to a string first: ```coffee stars = to_string(42) sky = "There are {{ stars }} in the sky." ``` Also paths are currently not supported, so this will not work: ```coffee message = "The message is {{ .message }}." ``` Assign the field to a variable first: ```coffee message = .message message = "The message is {{ message }}." ``` If you wish to insert `{{` into the string, you can escape using `\{{` and `\}}`. You also still have the option to use raw strings (`s'...'`): ```coffee assert!("\{{ right here \}}" == s'{{ right here }}') ``` Values and keys containing whitespace and/or double quotes are now wrapped in double quotes with the original quotes escaped. This change brings `encode_logfmt` inline with the used by other libraries. Previously, only keys and values containing whitespace were wrapped in quotes. If using `encode_logfmt`, the previous behavior would result in messages similar to: ```text lvl=info msg={"some":"val"} ``` With this change, the message would be encoded as: ```text lvl=info msg="{\"some\":\"val\"}" ``` When the `remap` transform was introduced, several transforms were deprecated and removed from the documentation, despite still being available in Vector. Additionally the `merge` transform was replaced by `reduce`. In 0.23.0 we will be finally removing these transforms as part of some cleanup to our reference documentation. Transforms to be removed: `add_fields` `add_tags` `ansi_stripper` `awscloudwatchlogssubscriptionparser` `coercer` `concat` `grok_parser` `json_parser` `keyvalueparser` `logfmt_parser` `merge` `regex_parser` `remove_fields` `remove_tags` `rename_fields` `split` `tokenizer` See below for examples of how to replicate functionality using the `remap` transform. Before: ```toml [transforms.add_fields] type = "add_fields" inputs = ["some_input"] fields.parent.child2 = "value2" ``` After: ```toml [transforms.add_fields] type = "remap" inputs = ["some_input"] source = ''' .parent.child2 = "value2" ''' ``` Before: ```toml [transforms.add_tags] type = "add_tags" inputs = ["some_input"] tags.sometag = "somevalue" ``` After: ```toml [transforms.add_tags] type = "remap" inputs = ["some_input"] source = ''' .tags.sometag = "somevalue" ''' ``` Before: ```toml [transforms.ansi_stripper] type = "ansi_stripper" inputs = ["some_input"] ``` After: ```toml [transforms.ansi_stripper] type = "remap" inputs = ["some_input"] droponerror = false source = ''' .message =

''' ``` Before: ```toml [transforms.awscloudwatchlogssubscriptionparser] type = "awscloudwatchlogssubscriptionparser" inputs = ["some_input"] ``` After: ```toml [transforms.awscloudwatchlogssubscriptionparser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= parseawscloudwatchlogsubscription_message!(.message) ''' ``` Before: ```toml [transforms.coercer] type = "coercer" inputs = ["some_input"] types.some_bool = "bool" types.some_float = "float" types.some_int = "int" types.some_string = "string" types.some_timestamp = "timestamp" ``` After: ```toml [transforms.coercer] type = "remap" inputs = ["some_input"] source = ''' .somebool = tobool!(.some_bool) .somefloat = tofloat!(.some_float) .someint = toint!(.some_int) .somestring = tostring!(.some_string) .sometimestamp = totimestamp!(.some_timestamp) ''' ``` Before: ```toml [transforms.concat] type = "concat" inputs = ["some_input"] items = ["month", "day", "year"] target = "date" joiner = "/" ``` After: ```toml [transforms.concat] type = "remap" inputs = ["some_input"] droponerror = false source = ''' .date = join!([.month, .day, .year], "/") ''' ``` Before: ```toml [transforms.grok_parser] type = "grok_parser" inputs = ["some_input"] pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" types.timestamp = "timestamp|%+" types.level = "string" types.message = "string" ``` After: ```toml [transforms.grok_parser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= parsegrok!(.message, "%{TIMESTAMPISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}") .timestamp = parse_timestamp!(.timestamp , format: "%+") ''' ``` Before: ```toml [transforms.json_parser] type = "json_parser" inputs = ["some_input"] ``` After: ```toml [transforms.json_parser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= object!(parse_json(.message)) ''' ``` Before: ```toml [transforms.keyvalueparser] type = "keyvalueparser" inputs = ["some_input"] ``` After: ```toml [transforms.keyvalueparser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= parsekeyvalue!(.message) ''' ``` Before: ```toml [transforms.logfmt_parser] type = "logfmt_parser" inputs = ["some_input"] ``` After: ```toml [transforms.logfmt_parser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= parse_logfmt!(.message) ''' ``` Before: ```toml [transforms.merge] type = "merge" inputs = ["some_input"] ``` After: ```toml [transforms.merge] type = "reduce" inputs = ["some_input"] startswhen = ".partial == true" merge_strategies.message = "concat" ``` Before: ```toml [transforms.regex_parser] type = "regex_parser" inputs = ["some_input"] patterns = ['^(?P<host>[\w\.]+) - (?P<user>[\w]+) (?P<bytesin>[\d]+) \[(?P<timestamp>.*)\] "(?P<method>[\w]+) (?P<path>.*)" (?P<status>[\d]+) (?P<bytesout>[\d]+)$'] types.bytes_in = "int" types.timestamp = "timestamp|%d/%m/%Y:%H:%M:%S %z" types.status = "int" types.bytes_out = "int" ``` After: ```toml [transforms.regex_parser] type = "remap" inputs = ["some_input"] droponerror = false source = ''' . |= parseregex!(.message, [#"^(?P<host>[\w\.]+) - (?P<user>[\w]+) (?P<bytesin>[\d]+) \[(?P<timestamp>.)\] "(?P<method>[\w]+) (?P<path>.)" (?P<status>[\d]+) (?P<bytes_out>[\d]+)$"#] .bytesin = toint!(.bytes_in) .sometimestamp = parsetimestamp!(.some_timestamp, "%d/%m/%Y:%H:%M:%S %z") .status = to_int!(.status) .bytesout = toint!(.bytes_out) ''' ``` Before: ```toml [transforms.remove_fields] type = "remove_fields" inputs = ["some_input"] fields = ["parent.child"] ``` After: ```toml [transforms.remove_fields] type = "remap" inputs = ["some_input"] source = ''' del(.parent.child) ''' ``` Before: ```toml [transforms.remove_tags] type = "remove_tags" inputs = ["some_input"] tags = ["some_tag"] ``` After: ```toml [transforms.remove_tags] type = "remap" inputs = ["some_input"] source = ''' del(.tags.some_tag) ''' ``` Before: ```toml [transforms.rename_fields] type = "rename_fields" inputs = ["some_input"] fields.newname = ["oldname"] ``` After: ```toml [transforms.rename_fields] type = "remap" inputs = ["some_input"] source = ''' .newname = del(.oldname) ''' ``` Before: ```toml [transforms.split] type = "split" inputs = ["some_input"] fieldnames = ["remoteaddr", "user_id", "timestamp", "message", "status", "bytes"] types.status = "int" types.bytes = "int" ``` After: ```toml [transforms.split] type = "remap" inputs = ["some_input"] droponerror = false source = ''' values = split(.message) .remote_addr = values[0] .user_id = values[1] .timestamp = values[2] .message = values[3] .status = to_int!(values[4]) .bytes = to_int!(values[5]) ''' ``` Before: ```toml [transforms.tokenizer] type = "tokenizer" inputs = ["some_input"] fieldnames = ["remoteaddr", "ident", "user_id", "timestamp", "message", "status", "bytes"] .types.status = "int" .types.bytes = "int" ``` After: ```toml [transforms.tokenizer] type = "remap" inputs = ["some_input"] droponerror = false source = ''' values = parse_tokens!(.message) .remote_addr = values[0] .user_id = values[1] .timestamp = values[2] .message = values[3] .status = to_int!(values[4]) .bytes = to_int!(values[5]) ''' ```