Project: Tremor (App Definition and Development / Streaming & Messaging)
File: CHANGELOG.md

add collect postprocessor add support for access tokens for aws connectors add support for brotli compression fix reconnect issue in HTTP client fix issue where post and preprocessors that hold state might not finalize correctly kafka schema registry codec Extract connectors in their own crate and aws and gcp into another remove schema_registry preprocessor remove defunct schema_registry support for avro codec kafka schema registry integration Expand the `for` comprehension to include initial values and folding (using a limited set of possible operations for now) Add `+` for `record + record` hygienic errors for strings in path segements axiom stdlib module gcl now supports the `timestamp` metadata overwrite basic avro codec Fix bug in GBQ connector where batches were not sent All google connectors now require `token` to be either set to `{"file": "<path to json>"}` or `{"json": {...}}` Fix parsing with `std::datetime::formats::RFC3339` to allow for `Z` (zulu) timezones. Add configuration option `pathstyleaccess` to `s3streamer` and `s3reader` connectors. Replace `async-std` runtime wint `tokio` Update to rust toolchain 1.68 Warning types: now warnings can specify what they are about (performance, behaviour, etc.) New warnings for consistency, behaviour and performance Functions now can create warnings Allow grouping of use with `use std::{m1, m2}` Add `tremor` codec for tremor to tremor communication Optimized buffer handling for json and binflux codec Update HTTP connector to allow configured for mime codecs Fix google pub sub default api endpoint Handle `routing` and `pipeline` information for each bulk item in the `elastic` connector all* std library constants are now uppercase Remove `default` in `match of` and `fn of` and standardize on `case _` `textual-length-prefix` pre and postprocessor are now named `textual-length-prefixed` for consistency in naming `json-sorted` codec is now part of the `json` codec with `mode: sorted` Added `state` type window where both tick and event can be handled in script form Added new config `socketoptions` to UDP, TCP and Websocket connectors to enable use of socket option `SOREUSEPORT` Added the `chunk` postprocessor for creating payloads as close to a configurable `max_bytes` as possible Arithmetic expressions now error on overflows Rust based erlang nif for tremor-script EQC testing Fix `udp_client` connector connecting to IPv6 hosts. Fix `json` codec to deduplicate duplicate keys by using the last key in the JSON record. Fix scoping for named scripts, they no longer polute/share the outer scope. Move `nodelay` config of `tcpserver` and `wsserver` to `socketoptions.TCP_NODELAY`. The HTTP connectors now work with the `mime_mapping` instead of `codec`. Added `std::datetime` library for parsing, formatting and working with datetimes implement script enhancement RFC (initial state and port based scripts) Fix bug in local variable resolving when using expression-paths Improve dogstatsd performance Improve performance of `std::array::flatten` by roughly 30% Fix windows based on `interval` not sending the correct event id when timing out The Google BigQuery connector will now split requests that are longer than the allowed limit (10MB) The GoogleBigQuery connector now accepts `table_id` metadata, to allow setting target table per event. Fix pipeline DAG creation when pipeline has unused nodes (e.g. a stream, operator or script) Fix possible hangs in `http_client`, `kv` and `dns` connectors when no pipelines is connected to their `out` port Fix `gbq_writer` swallow errors for requests that have been successfully sent to gbq, but

Fix systemd startup script Add `dogstatsd` codec for Datadog DogStasD implementation Fix output of `dns_client` being empty for any lookup. Remove `$elastic._type` field from `elastic` connector response events Add clear error messages for pipelines for console in, pipeline out, pipeline in Added a new port in ConnectInput and weaved them together Added a new port in ConnectOutput and weaved them together Fixed linking pipelines to pipelines Fixed error reporting in ConnectOutput and ConnectInput functions Fixed kv test cases Change `gcsstreamer` and `s3streamer` to only ack events upon finished uploads. `cb` connector config `path` changed to `paths`. Introduce breaking change in `kafka_consumer` configuration, by adding `mode`. Update grok to 0.2, optional patterns are now omitted no longer "" Unify to `url` instead of `endpoint` for google and s3 connectors so they're in line with every other connector Unify to `urls` for `elastic` connector Unify to `path` for `bench`, `kv` and `wal` connector so it's in line with file Rename `s3writer` to `s3streamer` Add `gcl_writer` Google Cloud Platform Cloud Logging connector for writing log entries, with in-flight concurrency support. Add reverse functions and tests for arrays and strings, add sort test for arrays Add a ClickHouse connector Add a GCS streamer connector Fix `tcpserver`, `wsserver` and `unixsocketserver` to properly close sockets when a client disconnects. Fix bug in the runtime, swallowing some event acknowledgements and fails for batched events from multiple streams Fix custom metrics for `kafkaconsumer`, `kafkaproducer` not being reported in the requested interval Fix `metrics` connector not emitting all metrics it did receive. Fix invalid scope handling for `flow` definition default arguments Fix `elastic` sink not acking events handled successfully if no source is connected. Fix `kafka_consumer` possibly committing earlier offsets, thus replaying events that have already been handled. Fix off-by-one error in `kafka_consumer` committing offsets, thus replaying the last committed event. Allow `kafka_consumer` connector to reconnect upon more error conditions and avoid stalls. Include flow alias in pipeline and connector aliases reported via metrics events and logging in order to deduplicate entries Fix startup startup script for systemd installations Avoid hangs in `elastic` connector when not connected as source via its `out` or `err` port. Default udp client bind ip to `0.0.0.0:0` instead of `127.0.0.1:0` Add compression level for `lz4`, `xz2` and `zstd` Don't fail unrecoverably when the connection to the server is lost in Google PubSub consumer. Update bert operators to work with tremor 0.12 Added the `gpubsub_publisher` connector Added the `gpubsub_consumer` connector Added new metadata options to `elastic` connector: `version`, `versiontype`, `retryonconflict`, `ifprimaryterm`, `ifseq_no` Fix race condition leading to quiescence timeout when shutting down Tremor Fix `timeout` config unit mismatch in `qos::backpressure` and `qos::percentile` operators. Changed to nanoseconds precision. Update WAL operator to fix issue with overwriting first entries. Fix misbehaving stdio connector when stdio is a pipe Fix Docker entrypoint script argument ordering Fix id encoding for discord connector Add full TLS support for `elastic` connector (including client certificate auth) Add several auth methods for `elastic` and `HTTP` connector (Basic Auth, Api-Key, Bearer Token, ...) Add support for specifying client certificates with `tls` config for `httpclient` and `tcpclient` connectors. Add `tremor new` to create new template projects. Support `elastic.raw_payload` for `update` Add support for `chunked` requests and responses for the HTTP connectors Add the `gbq` connector for Google BigQuery Add support for `tuple patterns` inside of record patterns. Refactor visitors to seperate walker and visitor and visit all

Add support for modular subqueries in Trickle Add `checktopicmetadata` configuration flag to kafka source to bypass topic metadata fetch port `json!` improvements to `literal!` print names of failing tests Add `checktopicmetadata` configuration flag to kafka source to bypass topic metadata fetch port `json!` improvements to `literal!` Add support for `troy` deployment language. The language adds the `pipeline`, `connector`, `flow`, `links`, `connect`, `to`, `deploy` and `config` reserved keywords. These are now reserved in the scripting and query languages and must now be escaped when they appear in event data. This is a breaking change in the query and script language dialects. Add support for `FLOAT4` and `FLOAT8` serialization/deserialization to postgres connectors Add `std::path::try_default` fn Experimental ARM support Add `default => {...}` and `default "key" => "value"` to patch Add `zstd` pre- and post-processors Remove `rental` from `Event` Put event raw payload into `Arc` to improve cloning perf Remove `rental` from the entire runtime Restructure operators and select to avoid transmutation Fix tremor-cli to use it's own binary when possible Restructure functions to avoid transmutation Update blackhole to print events / s Improve soundness and documentation of SRS code. Add support for concatenating arrays Allow gcp headers to be included in `rest` offramp with `auth: gcp` Add env source Remove need for emitemtpywindow by automatic eviction for groups where no window holds data Ensure all windows of a group are alligned and can not go out of sync Box apropriately in the rest sink replace macros with functions in gcp code Add -q flag and clarify -v flag for unit tests Add win::cardinality function Backpressure now allows lossless (no discard) circuit breaker behaviour Allow functions, constants and expressions as roots for path lookups Add the `unix-socket` source Enabale automatic benchmakrs Add the `csv` codec Add the `permissions` option for setting the file mode for the `unix-socket` source Tests can be run without their suite. Add the `std::size` module to convert sizes Add custom function calls to constant folding Integration test names are added as a tag so they can be run by name integration tests low log stdout/stderr for before and after the `tremor api` sub command in the cli has been removed. the `-` is no longer a valid part of identifiers. binaries now use `_` to sepoerate type names as `-` is no longer a identifier. changed naming for `record` object to avoid keywords like `select` and `merge`. New names are `record.extract` and `record.combine`. command separators are now unified, both `patch`, `match` and `for` now use `;` the same way the rest of the language does in all definitional statements `args` now specifies interface arguments that are overwritable in the correspanding `create` statement, while `with` specifies non-overwritable configuration in both `define` and `create` statements - this unifies the use of `with` and `args` between trickle and troy file connector no longer splits by lines - it now requires a preconnector define for both troy and trickle now follow the same principle of `define <type> <alias> from <source>` `wal` is no longer an operator but a connector for the elastic connector indexes have not to be set on the batch not the individual event so one batch can only be to a single

metronome interval is now in nanoseconds (as all other timings) Most connectors require a specified codec now instead of using JSON as a default `merge` no longer treats `null` in the spec as a delete option but rather as a normal value Combine all compression and decompression pre/postprocessors. Fix detection of `*.troy` files in entrypoint.sh causing duplicate configs to be loaded when using Kubernetes. Fix a one-off error in the `bench` connector leading to it producing one event too much Avoid acking events that failed while preprocessing or decoding using a codec. Remove acknowledging events when dropped or sent to a dead end (e.g. unconnected port) as this was causing confusing and unwanted behaviour with error events move EXIT connector to debug connectors to avoid shutting down tremor Fix bug in grok extractor that would never return good matches Properly terminate after `tremor run ...` Fix the `bench` connector to actually stop after the given amount of events when `iters` is configured. http headers to allow strings and arrays fix the use of `args` in the with part of a `create` insode of a `flow`. fix silent swallowing of unknown fields in connector definition. Make otel severity_number optional: #1248 Don't allow duplicate stream names: #1212 Fix memory safety issue when using `merge` or `patch` with `state` as target and reassigning the resulting value to `state` Fix delayed event re-execution in case of errors in a branched pipeline Skip instead of fail EQC on out of repo PRs Ensure patch keys are strings to move runtime errors into the compiletime Fix issue with the token `"` being presented as a tick in errors Fix `heredocstart` and `heredocend` showing up in error messages instead of `"""` Fix some errors in otel pb <-> json translation Fix windowed queries emitting events with `null` metadata on tick Fix sorting for artefacts Fix issue where the test framework would generate reports without being asked for it Remove the need for eviction_period for time based windows Remove dead code and unneeded allows in otel and gcp code Fix test and suite names not being printed Fix badly nested structure in unit tests Fix bug in the unit testing framework that would ignore all tags Fix the illogical structure of suites that required a doubly nested record Fix `-v` flag Fix issue with double counting of unit test stats Fix issue with wrong script snippets being shown for unit tests Fix argument order in test cases Fix GCS go-auth token refresh Fix `create script` syntax for aliased scripts with overridden `params` Add benchmark names to benchmark tags Kafka onramp: Remove failing metadata fetch in order to verify topic existance. Instead detect subscription errors and stop the onramp in that case. Unix offramp: Add the missing StartStream message tremor-script: Add more details about Unicode in the documentation of the `string` module Fix `hdr` and `dds` aggregation function losing events when aggregating > 8192 events Ensure merge can only happen on objects Unix onramp: Add the missing StartStream message to handle multiple connections correctly Fix windowed queries emitting events with `null` metadata on tick Kafka onramp: Remove failing metadata fetch in order to verify topic existance. Instead detect subscription errors and stop the onramp in that case. Fix `hdr` and `dds` aggregation function losing events when aggregating > 8192 events Make otel severity_number optional: #1248 Add the `unix-socket` onramp Upgrade dependency on librdkafka to 1.6.1 . Fix possible memory unsafety issue when using patch or merge on `state`

Fix artefact sorting in tremor startup script Update to clap 3, this forced some breaking changes: `tremor server run -f file1 file2` now is `tremor server run file1 file2` `tremor test -i i1 i2 -e e1 e2` is now `tremor test -i i1 -i i2 -e e1 -e e2` Add tests for patch feature in tremor-script Add onramp for SSE (Server Sent Events) Add raw flag in tremor-cli dbg Fix the Release archive and package build process Replace preprocess from tremor-cli dbg with --preprocess option Add tests for merge feature in tremor-script Add support for receiving TLS encrypted data via TCP onramp. Add support for sending TLS encrypted data via TCP offramp. Remove a number of transmutes Catalog remaining transmutes with related tickets Bump snmalloc-rs to 0.2.26 to ensure builds succeed on systems with libc < 2.25 Add `op` key to KV offramp responses in order to differentiate responses by the command that triggered them Change format of KV offramp responses to a more unified structure. Add `KnownKey::map_*` functions to directly work on the `Value::Object`s inner `HashMap`, if available. Add `HEALTHCHECK` to Dockerfiles Improve printing for dot files Scan sub directories of ${CFG_DIR} for config files. Add offramp and onramp for with the package. KV offramp sends error responses for each failed command Ensure binding to first bind linked offramps to pipelines, then pipelines to offramps/onramps/pipeline, then onramps to pipelines to ensure events only start flowing when all downstreams are connected. Fix empty record pattern to only match records. Fix ws offramp not reconnecting after connection loss. Run tests in tremor-cli bin Switch operations of `tremor dbg lex` and `tremor dbg preprocess` as they did the job of the other. Fix heredoc preprocessing, which was messing up error reporting Fix false positives in cycle detection Include `cncf::otel` stdlib sources in deb package Add `/usr/local/share/tremor` to default `TREMOR_PATH` also for all packages as a well-known directory for custom tremor-script libraries and modules. Record the partition number assigned during rebalancing when running Kafka. Fix bug in HDR histogram implementation when using emit without reset. Fix bug in mean that invalid values would be counted as part of the total number of values. Avoid possible contraflow cycles via `system::metrics` pipeline, if a pipeline is connected to an output port of `system::metrics` pipeline. Add `tremor_value::structurize` convenience fn Change `qos::wal` operator to only require one of `maxelements` or `maxbytes` (using both is still possible). Add DNS sink Add syslog codec. Add `$udp.host` and `$udp.port` to allow controling udp packet destinations on a per event basis. Deprecate `udp.dst_` config, introduce `udp.bind.` config instead. Allow insights/contraflow events to traverse through multiple connected pipelines Add GCP Cloud Storage linked sink connector. Add textual-length-prefix pre and postprocessor. Add GCP Pubsub sink and source connector. Inform docker builders that lack of resources may crash builds Fix CI runners to work with caching Fix dependencies for `tremor-value` and `tremor-common` Fix docker entrypoint not forwarding arguments to tremor binary, unless `--` is used before them Fix match default clause only executing the last statement in the

Kafka back to async with a timeout on waiting Fix `qos::wal` operator never cleaning up the last event from its storage Fix `generic::batch` operator swallowing the `transactional` status of an Event Fix windowed select queries not tracking the `transactional` status of an Event Fix windowed select queries not tracking the Events that constitute an outgoing aggregated event Avoid several offramps to swallow fail insights. Send correlation metadata for send error events in elastic sink Record the partition number assigned during rebalancing when running Kafka. CNCF OpenTelemetry source, sink and `cncf::otel` tremor-script library adds log, trace, metrics OpenTelemetry support Fixes module path function resolution for `tremor-cli doc` tool to use unified path resolution Removed the vsn.sh script which checks if the lockfile is up to date and replaces it with the --locked flag Allow using '\_' as seperators in numeric literals Refactor kafka metadata variables to be under a single record `$kafka`. Add support for Kafka message headers, available through the `$kafka.headers` metadata variable. Add the `cb` offramp for testing upstream circuit breaker behaviour Add the `kafka` onramp config `retryfailedevents` to acoid retrying failed events, and `polling_interval` to control how often kafka is polled for new messages if none were available previously Add `kv` connector with the supported operations `put`, `get`, `delete`, `scan`, `cas`. Add discord badge to README.md. Handle signals and terminate properly on Ctrl+C in docker Update to rust 1.50.0 Emit error events to the `err` port on exceeding concurrent requests limit for `rest` and `elastic` offramps. Add the `maxgroups` and `emitempty_windows` settings on window definitions Restrict event and event metadata references in the `SELECT` clause of a windowed select statement Improve default visibility of tremor info logs from packages Include the request info of a response for a linked rest offramp, available through the `$response.request` metadata variable. Emit warnings when a window with `emitemptywindows` without guards is used. Extend match to handle top-level `~` extractors in match Extend match to allow more assign expressions (i.e. `case a = ~ glob|snot*| =>` or `case v = _ =>`) Match pipeline improvements: tree search for `==`, grouping based on shared keys, re-ordering of exclusive case statements Optimize glob matches of the form `glob|snot|` or `glob|badger|` to cheaper prefix and suffix checks Remove warnings for match w/o default if a `` case or a `v = ` case exists Add `--exprs-only` to `dbg ast` to not show metadata Add Delete and Update to ES sink Add more metadata to Kafka source Update to simd-json 0.4 Add tests covering basic operations and string interpolation for tremor-script Add offramp and onramp for . Add a stdin onramp. Add tests covering arrays and records for tremor-script Support for non caching UDP sink Fix `kafka` onramp hanging with no message in the queue, leading to delayed offset commits Fail the `kafka` onramp if any of the configured topics could not be subscribed to Tremor no longer requires a home dir for operations that do not need a config Add performance section to PR template. Fix markdown for discord link. Fix lalrpop builds when extra folders exist. Refactor operator metrics collection to eliminate clones in most

Fix systemd spec file to load tremor files in `/etc/tremor/config` Do not crash when we can not execute a config file Sort the artefacts while running the benchmarks so that the benchmark run is more deterministic Remove the benchpipepassthrough_csv benchmark Fix a bug in the `test bench` command that was giving false negatives when the benchmarks were failing Fix time based windows to not emit empty windows anymore and to not error if they contain event references Do not commit an empty topic-partition-list in the kafka onramp and improve logging for better debugging Fix kafka consumer offset lag of at least `1` continually by using offset + 1 when committing. Fix issue where binary not (`!`) was not getting lexed correctly Fix missing ack/fail insight events with offramps that dont support guaranteed delivery (e.g. udp, stdout) Fix wrong error message for misconfigured UDP sinks the `api-host` argument is renamed from `-h` to `-a` to resolve a conflict with `-h` being used for help Ensure blaster sends all events from the source Allow the use of const and custom functions using const in select queries Print hygenic errors when invalid `trickle` files are loaded in `server run -f ...` Ensure `elastic` offramp does not issue empty bulk requests. Avoid sending empty batches from the `batch` operator. Ensure `elastic` offramp includes event `payload` in every error response. Add discord connector to allow communicating with the discord API Update tremor-value to 0.2 to include binary changes and thus unbreak the 0.10 tremor-script crate String interpolation is now done via `#{..}` instead of `{}`. `{` no longer needs to be escaped, but `\#{` needs an escape for literal `#{`. Emit tumbling windows based on `size` immediately when they are full. Emit tumbling windows based on time `interval` also when no event comes in but the interval passed Elasticsearch offramp `elastic`: change the config value `endpoints` to `nodes` Default to thin-lto for all builds (prior this was only done in docker) Automatically generate rpms and tarballs for releases. Update rust to 1.49.0 Build deb packages Statically link openssl elastic sink now supports linked transports rewrite string interpolation to fix Extract simd_json::BorrowedValue into tremor specific tremor-value to allow extension of the type system. Introduce Binary datatype and binary semi literals `<< 1, 2, event, (3 + 7) >>` Introduce `base64` module for encoding and decoding binary to base64 encoded strings introduce `binary` module to work with binary data Terminate pipeline creation when a node already exists with the given name Fix visibility of pipeline metrics Fix panic upon usage of postgres ramps due to incompatbility with tokio and async-std runtime. Fix missing newlines when error line doesnt end with a newline Fix possible panic in random::integer and random::float when used with values <= 0. Add `prefix` and `raw` config options to `stdout` and `stderr` offramps Make use of postprocessors in `stdout`, `stderr` and `udp` sinks Allow to express minimal value of i64 as int literal Fix scientific float literals (e.g. `1.0e-5`) Output errors to stderr on `tremor run` invocation. Output more helpful errors for runtime errors (like accessing non-existing fields) Fix gelf preprocessor to accept valid (unchunked) gelf messages Fix memory access issue for large objects Add docker image, and action for training docker image Performance enhancements Standardize error responses for tremor api Remove empty event filtering from preprocessor logic Fix possible crashes from todo macro in tcp sink Fix linked offramp not shutting down after binding is deleted Fix slow kafka sink when queue.buffering.max.ms is set to > 0 Fix string and heredoc errors with and without interpolation Add hygienic error feedback for `tremor run`