Before revision:  Our model parallelly masks and predicts each target token , and extracts high-quality alignments without any supervised loss. In addition, we introduce leaky attention to alleviate the problem of unexpected high attention weights on special tokens  . Experiments on four language pairs show that our model significantly outperforms all existing unsupervised neural baselines and obtains new state-of-the-art results.
After revision:  Our model parallelly masks and predicts each target token and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates the problem of unexpected high attention weights on special tokens . Experiments on four language pairs show that our model significantly outperforms all existing unsupervised neural baselines and obtains new state-of-the-art results.