Project: Vector (Observability and Analysis / Observability)
File: 2020-04-15-2341-wasm-plugins.md

Introduce plugin support to Vector, providing a way for users to add custom functionality to Vector. This RFC introduces WASM plugins specifically, but we envision the possibility of other implementations that may reuse ideas from this RFC. This RFC introduces Transforms and will be later amended with Source and Sink functionality. There is a plurality of reasons for us to wish for flexible, practical plugin support to Vector. These may also be regarded as 'Foreign Modules' or 'Extensions.' WASM is the most versatile and obvious choice for an implementation as it can be targeted by a number of languages and is intended to be OS, arch, and platform independent. Since the concept of a plugin is a abstract, this section will instead discuss a number of seemingly unrelated topics, all asking for different features. While plugin support does not resolve all these problems, it gives us or our users a path to a solution. We noted in discussions with several current and prospective users that custom protobuf support in the form of an `encoding` option or a transform. Protobufs in their `proto` form aren't usable by the prevailing Rust Protobuf libraries in a dynamic way. Both `prost` and `rust-protobuf` focus almost solely on compiling support for specific Protobufs into a Rust program. While a `serde-protobuf` crate exists and can do dynamic decoding, we found it to be an order of magnitude slower than the other two options, we also noted it lacked serialization support. While evaluating the possibility of adding serialization we noted that any implementation that operated dynamically would be quite slow. Notably, it would it would seriously impact overall event processing infrastructure if Vector was responsible for a slowdown. These organizations are choosing to use protobufs usually for speed and/or compatibility guarantees. We should enable and encourage these goals, our users should be empowered to capture their events, and not worried about processing load. So the question became: *How do we let users opt-in to custom protobuf support efficiently and without great pain?* Recently, we've been discussing the trade offs associated with different code modularization techniques. We currently make fairly heavy use of the Rust feature flag system. This is great because it enables us to build in multi-platform support as well as specialized (eg only 1 source) builds. We also noted builds with only 1 or two feature flags are significantly faster than builds with all features supported. One downside is our current strategy can require tests (See the `make check-features` task) to ensure their correctness. The other is that we sometimes need to spend a lot of time weaving through `cfg(feature = "..")` flags. Having some solution to cleanly package independent modules of functionality would allow us to have a core Vector which builds fast while keeping modularity features. Vector currently bundles a , and there is ongoing work on a . Currently, each language requires as separate runtime, and possibly separate integration subtleties like needing to call a . It would be highly convenient if we could focus our efforts on one consistent API for all languages, and spend this time saved providing a better, faster, safer experience. As Vector grows and supports more integrations, our binary size and build complexity will inevitable grow. In informal discussions we've previously noted the

This kind of architecture can help us simplify the codebase, reduce binary sizes, formalize our APIs, and open Vector up to a third-party module ecosystem. Terraform typically runs on CIs and developer machines, making portability valuable. For Terraform, users can follow the to write a Go-based provider, which they can then distribute as a portable binary. Notably: This means either distributing an unoptimized binary, distributing a lot of optimized binaries (per architecture), or requiring folks build their own optimized binaries. Terraform doesn't care much about speed, so unoptimized binaries are acceptable. Vector is in a slightly different position than Terraform though! Vector runs primarily in servers and even end user machines. We can't expect users to have build tooling installed on their servers (it's a security risk!) and we definitely can't expect it on end-user machines. Most people just aren't that interested in computers. Vector has different performance needs, too. While folks aren't generally wanting to execute terraform providers hundreds of thousands (or millions) of times per second, they are absolutely doing that with Vector. When we're processing the firehose of events originating from a modern infrastructure every millisecond counts. Vector needs a way to ship portable, optimizable plugins if we ever hope of making this a reality. We would like to avoid asking users to chain together a number of transforms to accomplish simple tasks. This is evidenced by issues like , , and . Having to, for example, use two transforms just to add 1 field and remove another is very verbose. We noted that the existing lua runtime was able to accomplish these tasks quite elegantly, however it was an order of magnitude slower than a native transform. Users shouldn't pay a high price just for a few lines saved in a configuration file. They shouldn't feel frustration when building these kinds of pipelines either. We've discussed various refinements to our pipelines such as and . Some of these discussions have been about improving our TOML syntax, others about new syntax entirely, and others about how we could let users write in the language of their choice. The path forward is unclear, but if we choose to adopt something new we must focus on providing a good user experience as well as performance. Having a fast, simple, portable compile target could make this an easier effort. We exist in an ecosystem, exists and we'd really love to be able to work in harmony somehow. While both tools process events, Vector focuses on acting as a host-level or container-level last/first mile router, Tremor focuses on servicing demanding workloads from it's position as an infrastructure-wide service. What if we could provide users of Tremor and Vector with some sort of familiar shared experience? What if we could share functionality? What kind of framework could satisfy both our needs? There are so many questions! We are in communication with them and this RFC will amended in the future to include cross compatibility details if any emerge. We have an existing `lua` runtime existing as transform, and there is currently an regarding an eventual `javascript` transform. Neither of these features reach the scope of this proposal, still, it is valuable to

While evaluating the `lua` transform to solve the Protobuf problem described in Motivations, we benchmarked an implementation of `add_fields` that was already performing at approximately the same performance as the `serde-protobuf` solution. We also noted that the existing `lua` v1 transform was quite simple. While this makes for nice usability, it means things like on-start initializations aren't possible. In `lua` has learnt many new tricks, and this WASM support shall match them. We also noted the capabilities and speed of WASM (particularly WASI) mean we could eventually support things like sockets and files. Indeed, it is possible we will be able to let Vector build and run Lua code as a WASM module in the future. Vector contains a WebAssembly (WASM) engine which allows you to write custom functionality for Vector. You might already be familiar with the transform, WebAssembly is a bit different. While it enables more functionality and possibly better speeds, it also introduces complication to your system. You'll need to learn a bit about Foreign Function Interfaces (FFI), a bit about memory management, and build your WASM artifact. Don't worry: We'll walk you through it. It's a new frontier, and you've got a friend in Vector. Using the WASM engine you can write your own transforms, sinks, codecs in any language that compiles down to a `.wasm` or `.wat` file. Vector can then compile this file down to an optimized, platform specific module which it can use. This allows Vector to work with internal or uncommon protocols or services, support new language transforms, or just write a special transform to do exactly what you want. With the WASM engine, Vector puts you in command. There is a trade-off though! You'll need to embolden yourself, as WASM is a fledgling technology. When it first loads a WASM file, Vector needs to spend some time building an optimized module before it can load it. You'll also be responsible for the safety and correctness of your code, if your module crashes on an event, Vector will re-initialize the module and try again on the next event. We try to follow the ideas of . Some examples of when a WASM plugin is a good fit: You need to support a specific protobuf type. You require a source or sink which is not currently supported by Vector. You want to write a complex transform in Rust. (Other languages will be library supported according to demand and resources) Some examples of where WASM plugin probably isn't right for you: You only need basic functionality already offered by Vector. Your desired language doesn't support compiling to WASM with WASI. You are using Vector on a non-Linux X86-64 bit compatible target. (, our Engine, only supports Linux at this current time. Work on other operating systems is ongoing!) You need a battle-proven system, WASM support in Vector is young! Vector plugins are `.wasm` files (WASI or not) exposing a predefined interface. We provide a `vector_wasm` crate for Rust projects located in `lib/vector-wasm` of the source tree, and we will be providing bindings for other languages as we see user demand. Plugins are sandboxed and can only interact with Vector over the Foreign Function Interface (FFI) which Vector exposes. For each instance of a module, Vector holds some context. For example, when a transform module processes an event, the context of the module contains an event, as well as the configuration used to create the

A guest module is not able to access the memory of the Vector instance, but Vector is free to modify the guest memory. For example, behind the scenes of a `get` call, the Guest must allocate some memory for Vector to write the result into. Due to the current semantics of the `vector::Event` type, passing data between WASM plugin and the Vector host involves serialization. This may change in the future, resulting in an increase major version of the `vector-wasm` crate. Regardless of whether of not the plugin uses WASI, or has access to a language specific binding, it can interact with Vector. Vector works by invoking the plugins public interface. First, as soon as the module loads, `register` is called once per module. In this call, the module can configure how Vector sees it and talks to it. During this time, the module can do one time setup it needs. (Eg. Opening a socket, a file). If using the Vector guest API this can be done via: ```rust use vector_wasm::{Registration, roles::Sink}; // TODO: This is unsafe -- Needs a fix. pub extern "C" fn init() { Registration::transform() .register() } ``` What happens next depends on which type of module it is. `Transform`: Modules of this role can be used as a transform `type`. ```rust use vector_wasm::hostcall; pub extern "C" fn process(data: u64, length: u64) -> usize { let data = unsafe { std::ptr::slicefromrawpartsmut(data as *mut u8, length as usize) .as_mut() .unwrap() }; let mut event: BTreeMap<String, Value> = serdejson::fromslice(data).unwrap(); event.insert("newfield".into(), "newvalue".into()); event.insert("newfield2".into(), "newvalue2".into()); hostcall::emit(serdejson::tovec(&event).unwrap()); 1 } ``` TODO: Sources, sinks, and codecs have no POC or formal specification. To create your first module, start with a working add the `wasm32-wasi` toolchain: ```bash rustup target add wasm32-wasi ``` Then, create a module: ```bash cargo init --lib echo ``` In your `Cargo.toml` fill in: ```toml [lib] crate-type = ["cdylib"] [dependencies] vector-wasm = { path = "/path/to/your/clone/of/vector/lib/vector-wasm"} serde = { version = "1", features = ["derive"] } ``` Next, in your `lib.rs` file: ```rust use serde_json::Value; use std::collections::BTreeMap; use vector_wasm::{hostcall, Registration}; pub extern "C" fn init() -> *mut Registration { &mut Registration::transform().set_wasi(true) as *mut Registration } pub extern "C" fn process(data: u64, length: u64) -> usize { let data = unsafe { std::ptr::slicefromrawpartsmut(data as *mut u8, length as usize) .as_mut() .unwrap() }; let mut event: BTreeMap<String, Value> = serdejson::fromslice(data).unwrap(); event.insert("newfield".into(), "newvalue".into()); event.insert("newfield2".into(), "newvalue2".into()); hostcall::emit(serdejson::tovec(&event).unwrap()); 1 } pub extern "C" fn shutdown() { (); } ``` Now to build it: ```bash cargo +nightly build --target wasm32-wasi --release ``` In your `target/wasm-wasi/release/` verify that the `echo.wasm` file exists. Next edit your Vector `config.toml`: ```toml data_dir = "/var/lib/vector/" [sources.source0] max_length = 102400 type = "stdin" [transforms.demo] inputs = ["source0"] type = "wasm" module = "target/wasm32-wasi/release/echo.wasm" [sinks.sink0] healthcheck = true inputs = ["demo"] type = "console" encoding.codec = "json" buffer.type = "memory" buffer.max_events = 500 buffer.when_full = "block" [[tests]] name = "demo-tester" [tests.input] insert_at = "demo" type = "log" [tests.input.log_fields] "message" = "foo" [[tests.outputs]] extract_from = "demo" [[tests.outputs.conditions]] "echo.equals" = "foo" ``` Now try `vector test config.toml` and Vector will go ahead and build an optimized `cache/echo.so` artifact then run it for a unit test. ```bash $ vector test config.toml Running config.toml tests test config.toml: demo-tester

passed ``` Congratulations, you're ready for a new frontier! Integrating a flexible, practical, simple WASM plugin support means we can: Expose one common interface to any existing and future languages supported by WASM. Allow large-scale users to use custom encodings. Support and advertise a plugin system. Empower us to modularize Vector's components and increase maintainability. Share a common plugin format with Tremor. In the future, support something like AssemblyScript as a first class language. WASM is still very young. We will be early adopters and will pay a high adoption price. Additionally, we may find ourselves using unstable functionality of some crates, including a non-trivial amount of `unsafe`, which may lead to unforeseen consequences. Our binary size will bloom. We currently (in unoptimized POC implementations) see our binary size grow by over 60 MB. The Lucet team is aware of this issue, and in some preliminary discussions this binary size increase is unexpected and we expect it to improve. If this does not improve, we can consider adopting `wasmtime`, lucet's sister JIT implementation which has a smaller binary footprint. Our team will need to diagnose, support, and evolve an arguably rather complex API for WASM modules to work correctly. This will mean we'll have to think of our system very abstractly, and opens many questions around API stability and compatibility. This feature is advanced and may invoke user confusion. Novice or casual users may not fully grasp the subtleties and limitations of WASM. We must practice caution around the user experience of this feature to ensure novices and advanced users can understand what is happening in a Vector pipeline utilizing WASM modules. Since this is a broad reaching feature with a number of green sky ideas, we should discuss these questions: We should support plugins so broadly? As sinks? For this we should consider how we handle batching, partitioning, etc, more. Temporary Conclusion:* Held off on deciding. Some interest. As sources? This requires thinking about how we can do event sourcing. Temporary Conclusion:* Held off on deciding. Some interest. What about the idea of codecs being separate? Temporary Conclusion:* Held off on deciding. Some interest. This RFC makes some provisions for a future Event refactor, is it possible that this might happen? Temporary Conclusion:* Held off on deciding. Some interest. Our original discussions included the idea of a blessed compiler and a UX similar to our existing `lua` or `javascript` transform. Indeed, our initial implementation includes just a transform, as it's by far the easiest place to introduce this feature. During investigation of the initial POC we determined that our most desirable "blessed" language would be AssemblyScript, unfortunately, it's currently not able to run inside Lucet. Work is ongoing and we expect it to be possible soon. Temporary Conclusion: AssemblyScript and other languages we evaluated did not provide a way to package themselves as WASM modules, making it hard to integrate with them. We are planning to revisit this at a later date when the ecosystem is more mature. We should consider if we are satisfied with the idea of hostcalls being used for `get` and other API. We could also let the host call the Guest allocate function and then pass it a C String pointer to let it work on. This, however, requires serializing and deserializing the entire Event each time, which is a huge performance bottleneck. We could also consider adopting a new strategy for our `event` type. However, that work is outside the scope of this

The current structure of the `Event` type is already discussed in . We should also consider if we want to change how we handle codecs, since it is likely that WASM module use cases will include wanting to add codec support to already existing sources. Temporary Conclusion: We will adopt hostcalls for the time being to allow for a minimal POC. We would like to investigate these choices later, and we will make a decision before stabilizing WASM modules. We will need to expose some Vector guest API to users. This will require ABI compatibility with Vector, so it makes sense to keep it in the same repository (and maybe the same crate?) as Vector. We can consider either making Vector able to be built by WASI targets (more feature flags) or using a workspace and having the guest API as a workspace member we publish. A couple good questions to consider: Do we one day want to support a cross platform `cargo install vector` option for installing a Vector binary? Should a user import `vector::...` to use the Vector guest API in their Rust module? Vector's internal API is largely undocumented and kind of a mess. Should we hide/clean the irrelevant stuff? Conclusion: We are including a workspace member `vector-wasm` in the `vector-wasm` directory. This may be renamed and/or published in the future. How can we let users see what's happening in WASM plugins? Can we use tracing somehow? Lucet supports tracing, perhaps we could hook in somehow? Conclusion: Preliminary results show we may be able to allow guests to write with the current tracing subscriber. There is ongoing work to support more platforms in Lucet, here are how things stand today: Platform support: [x] Linux (x86_64) [ ] Linux (ARMv7) (Likely never) [ ] Linux (Aarch64) (Probably coming, ARM contributing) [ ] Mac (x86_64) https://github.com/bytecodealliance/lucet/pull/437 [ ] Windows (x86_64) https://github.com/bytecodealliance/wasmtime/pull/1216 https://github.com/bytecodealliance/lucet/issues/442 https://github.com/bytecodealliance/lucet/pull/437 [ ] FreeBSD (x86_64) https://github.com/bytecodealliance/lucet/pull/419 https://github.com/bytecodealliance/lucet/pull/437 Temporary Conclusion: The Lucet team desires to support other platforms in the near future. Since the lion's share of Vector usage is on X86_64 Linux, and this platform is already supported, we decided to adopt Lucet for now. We could consider adopting `wasmtime` in the future if lucet does not eventually reach it in terms of platform parity. Incremental steps that execute this change. v0.1: (Done) This draft forms the groundwork for this RFC, and demos a POC of how a theoretical user could use this for a protobuf decoding transform, and permitting the first `wasm` transform test to pass. v0.2: (Done) The POC of the initial transform has it's v1 v0.3: (Done) We have benchmarks of the initial transform POC. v0.4: (Done) Compile artifact caching is implemented, so Vector doesn't unnecessarily recompile wasm modules. v0.5: Guest API expanded to support majority of Event API. v0.6: We talk to a couple known users who would be interested in this feature and let them test it out. v0.7: This RFC is amended to include Sinks and Sources information. v0.8: This RFC is amended to include information about possible codec changes. v0.9: Source and Sink implementation POCs made. v0.10: Final APIs specced and tested. Source, Sink, Transform POCs are in tree and running as tests/benches. v1.0: This RFC is complete and we announce with the above guide. Note: This can be filled out during the review process.