Alter vector file loader to support passing read mode

This will be used in the X.509 PR since we're loading binary files there.