I have some ideas.

First, I have been thinking more about how packets should "ideally" be divided up into buffers, and learning more about the constraints imposed on us by the virtio_net driver in Linux guests running under KVM.

Current theory:
1. The first iovec in a packet should be just big enough to contain all of the packet headers (~128 bytes).
2. The remaining iovecs should be "pretty big" e.g. 4KB.

Then we should feel free to rewrite the first small iovec to our heart's content for hacking headers etc but try to reuse the remaining iovecs verbatim to avoid hacking/copying excessive amounts of data.

In practice the most common packets are probably:
1. Control packets (ICMP, TCP SYN, etc) that fit in one iovec.
2. ~1500-byte data packets that are split into two iovecs (one small iovec and one big).
3. ~9000-byte jumbo packets that are split into two or three iovecs (one small iovec and some big).
4. ~64KB "TSO" packets that will be segmented in hardware (one small iovec and many big).

This scheme would let us do header-hacking on all such packets with a bounded cost in terms of data copies.

The scheme *should* also work reasonably well for Virtio to KVM. I believe that for sending to Virtio it's okay for us to control the sizes of the iovecs but they have to start on the 4KB-boundry that we are given by the VM (so to decapsulate we need to rewrite the whole header rather than simply shorten the prefix in memory).

The scheme *should* also work well for Intel hardware which has a "header split" feature that can split the packet headers into a separate buffer from the payload i.e. create this iovec layout on packets arriving from the network.

Does that make any sense? This is stuff I've been working out this past couple of weeks while hacking vhost_user support for KVM.