Before revision:  Experiments on 10 language pairs show that removing the embedding matrix consistently improves the performance of byte-to-byte models , often outperforms character-to-character models , and sometimes even produces better translations than standard subword models .
After revision:  Experiments on 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models , often outperforms character-to-character models , and sometimes even produces better translations than standard subword models .