Remove unnecessary e2e header fields

The whole e2e header will be moving to XNI soon. But before then I
want to make the header as small as possible.