Project: Redpanda (App Definition and Development / Streaming & Messaging)
File: 20190928_record_batch_header_vtype.md

Feature Name: add `vtype` to `model::recordbatch_header` Status: in-progress Start Date: 2019-09-28 Authors: alex RFC PR: (PR # after acceptance of initial draft) Issue: (one or more # from the issue tracker) Add an internal type to differentiate between different subsystem batches. ```patch @@ -141,10 +141,24 @@ private: std::bitset<16> _attributes; }; +using recordbatchtype = namedtype<uint8t, struct modelrecordbatch_type>; struct recordbatchheader { // Size of the batch minus this field. uint32t sizebytes; offset base_offset; recordbatchtype type; int32_t crc; recordbatchattributes attrs; int32t lastoffset_delta; modified src/v/raft/types.h @@ -37,40 +37,32 @@ struct group_configuration { std::vector<model::broker> learners; }; -class entry { +/// \brief a collection of record_batch. In other words +/// and array of array. This is done because the majority of +/// batches will come from the Kafka API which is already batched +/// Main constraint is that all records and batches must be of the same type +class entry final { public: using type = int32_t; // well known types static constexpr type type_unknown = 0; static constexpr type type_data = 1; static constexpr type type_configuration = 2; explicit entry(type t, model::recordbatchreader r) explicit entry(model::recordbatchtype t, model::recordbatchreader r) : _t(t) , _rdr(std::move(r)) { } virtual ~entry() = default; virtual type entry_type() const { model::recordbatchtype entry_type() const { return _t; } virtual model::recordbatchreader& reader() { model::recordbatchreader& reader() { return _rdr; } virtual future<> on_replicated() { return makereadyfuture<>(); } private: type _t; model::recordbatchtype _t; model::recordbatchreader _rdr; }; struct appendentriesrequest { model::nodeid nodeid; protocol_metadata meta; std::vector<std::unique_ptr<entry>> entries; std::vector<entry> entries; }; ``` First observation The detection whether or not the record is internal should actually happen at the batch level. The reason is `record_batch` is our atomic unit for reads and writes. Thus, the next modifications follow: add a `model::recordbatchtype` above the CRC line de-virtualize `raft::entry` type We want other subsystems `raft` and `cluster` in particular to take advantange of our `storage::log` and related infrastructure to persist changes to disk. It allows us to have one code base for persisting records and reading them from disk. We will be extending the `model::recordbatchheader` with a `model::recordbatchtype` which is just a wrapper around a `uint8_t` By default `0` will be used to detect errors while either parsing data from the Kafka layer or from the RPC layer. The wire format will increase by 1 byte on disk The Kafka produce API will have to set this byte explicitly to `1` Raft configurations will have to set this byte to `2` Kafka fetch API will have to skip all records that are not equal to `1` We want to leverage the existing infrastructure for persisting and reading data from

Largely, each new component that wishes to extend the type, should reserve their integer in the `model` submodule to ensure no collisions at runtime. Note that we `explicitly` chose an integer to have no inter-module coupling. Reading and writing will have no modification other than reading+writing the extra byte which should be a very small change. The main dependency is that the `kafka::produce` API will have to set the byte. Changing the `model::recordbatchheader` is a very low level feature. It is modifying the disk layout and therefore the readers. We expect that most features will never actually extend this. Storage Raft Cluster Kafka We expect most features will actually interact with modules at the interface level and not at the disk format level. That is, you will wrap your puts as a Kafka produce API format, or you will interact w/ the global cluster via the `cluster::controller` module and so on. The main note worth highlighting is that by setting the default to 0, it allows us to detect if the proper subsystems actually set the field or simply left it alone. Note that leaving it alone is a bug. It must be set by each subsystem. We expect to have tooling integrated with this feature so system administrators can actually inspect and see if we have a bug by not having set the `model::recordbatchtype` in the actual on-disk format. Why should we not do this? Adds complexity to the parsing protocol Adds complexity to every module by allowing every module to set the `model::recordbatchtype` Why is this design the best in the space of possible designs? This proposal allows us to leverage the same on-disk format as the `storage::log` without the need for keeping secondary metadata. In other words, if we were to not use this design, we would have to have some way of knowing whether the batch on disk is publicly visible to the Kafka layer or not. The complexities of maintaining alternative metadata is a possible source of inconsistencies and a major source of complexity. What other designs have been considered and what is the rationale for not choosing them? Setting a single bit in the `model::recordbatchattributes` was considered and discarded: It is actually taken into account with the Kafka CRC Which meant that we would have to re-checksum all payloads We would have no way of detecting `unset` bits. This is a powerful property of this design. By requiring each module to set this byte we can detect if a new extension to the code base simply forgot to set the `model::recordbatchtype` and inform early during testing. This property could not have been achieved with a single bit. Missing integration with tooling. What and which tool should work w/ the log replay functions to help us debug bad log segments