Make sure we always have unk in the vocabulary, otherwise we get off-by-one indexes during decoding