Project: Vector (Observability and Analysis / Observability)
File: steps.md

As this PR introduces the change that moves the default `disk` buffer implementation from disk v1 to disk v2, we similarly add the logic to automatically migrate users' disk buffers from v1 to v2 when Vector starts. We do so to ensure that users do not upgrade Vector and have their old buffers fall by the wayside unbeknownst to them, as it is likely they may unintentionally upgrade such that they do not see any warning/error logs that we emit to indicate that they must first drain their old buffer before upgrading, and so on. Additionally, we did not change the on-disk format for `disk_v2` between 0.20.0 and now, so we should also be able to create a `disk_v2` buffer with 0.20.0 and continue using it as a `disk` buffer with a binary built from this PR. Start by grabbing a Vector binary for 0.20.0, and build one from the PR branch. We'll use a simple configuration that will read records from stdin and attempt to send them to an HTTP sink. For the HTTP sink, we'll have one configuration that uses an invalid port and another, an identical version, which has the correct port. Since Vector will retry "connection refused" errors, and retry them infinitely, the messages will never be acknowledged, which ensures they remain in the buffer. For the purposes of verifying that the same data that went in is still present after migrating the disk v1 buffer, etc, we can use `dummyhttp` (a Rust project for serving up a simplistic HTTP endpoint that can be configured to respond a certain way) to listen on the port and inspect the HTTP request made by the sink. For generating logs so we can fill up our disk buffers to their limit, we'll use which is a handy tool for generating fake logs, and a lot of them very quickly. Ensure that records written to a v1 disk buffer from a version of Vector without this change can be read back after being migrated: Start `dummyhttp` listening on the relevant port. Run the 0.20.0 binary, using the "wrong" configuration, with a clean data directory. The `five-lines-first` file should be piped to STDIN. Stop

Run the PR binary, using the "wrong" configuration. The `five-lines-second` file should be piped to STDIN. Observe that the old buffer has been migrated, and that five records have been migrated: the entries from `five-lines-first`. Stop Vector. Run the PR binary, using the "right" configuration, and ensure that it reads the records from the buffer and sends them to the HTTP sink. The `five-lines-second` file should be piped to STDIN. Overall, there should be fifteen records: all five from the run using Vector 0.20.0, five from the run of the PR binary with the wrong configuration, and five from this run. Ensure that records written to a v2 disk buffer from a version of Vector without this change can be read back after being migrated: Start `dummyhttp` listening on the relevant port. Run the 0.20.0 binary, using the "wrong disk v2" configuration, with a clean data directory. The `five-lines-first` file should be piped to STDIN. Stop Vector. Run the PR binary, using the "wrong" configuration. The `five-lines-second` file should be piped to STDIN. Stop Vector. Run the PR binary, using the "right" configuration, and ensure that it reads the records from the buffer and sends them to the HTTP sink. The `five-lines-second` file should be piped to STDIN. Overall, there should be fifteen records: all five from the run using Vector 0.20.0, five from the run of the PR binary with the wrong configuration, and five from this run. Ensure that records written to a v1 disk buffer from a version of Vector without this change can be migrated over when the new v2 buffer ends up exceeding the configured maximum buffer size: Run the 0.20.0 binary, using the "wrong big buffer" configuration, with a clean data directory. `flog` should be piped to STDIN. Stop Vector once the maximum buffer size has been reached. Observe that the old buffer data directory is the only one that exists. Run the PR binary, using the "wrong big buffer" configuration. Do not feed any input to STDIN. Vector should immediately exit after reporting that the old buffer has been migrated. Observe that the old buffer directory is now gone, and the new one should exist. Likewise, the new one should be the same size or larger than the old one.