Question: Which modifications do they make to well-established Seq2seq architectures?
Answer: Replacing attention mechanism to query-key attention, and adding a loss to make the attention mask as diagonal as possible