Trim ByteBuf when we always end up with extra bytes, so that ByteBuf does not grow too large