Prefix shouldn't be interpreted as Base64 data

 * Things worked as is because the prefix will always be Base64 safe
   chars, but that doesn't mean it should be interpreted as such.

 * This is a backwards incompatible change.

 * Bug introduced in f13519ff2db986c4d474f0aac50d984453783cb5.