Non-autoregressive Neural Machine Translation Based on Latent-Variable Models
潜在変数モデルに基づいた⾮⾃⼰回帰型ニューラル機械翻訳

Raphael Shu (朱 中元)
東京大学

1

Neural Machine Translation•NMT is conditional probabilisitic modeltraining

MT Model<latexit sha1_base64="CCH2vR6QIal1EdvHOChQnrp0h18=">AAAB7XicdVDJSgNBEK2JW4xb1KOXxiDEyzAjouYW9OIxglkkGUJPpydp7ekeunuEMOYfvHhQxKv/482/sbMIrg8KHu9VUVUvTDjTxvPendzc/MLiUn65sLK6tr5R3NxqaJkqQutEcqlaIdaUM0HrhhlOW4miOA45bYY3Z2O/eUuVZlJcmmFCgxj3BYsYwcZKjaR8ddfa7xZLnlvx/MqRj34T3/UmKMEMtW7xrdOTJI2pMIRjrdu+l5ggw8owwumo0Ek1TTC5wX3atlTgmOogm1w7QntW6aFIKlvCoIn6dSLDsdbDOLSdMTYD/dMbi3957dREJ0HGRJIaKsh0UZRyZCQav456TFFi+NASTBSztyIywAoTYwMq2BA+P0X/k8aB63uuf3FYqp7O4sjDDuxCGXw4hiqcQw3qQOAa7uERnhzpPDjPzsu0NefMZrbhG5zXD0v4jvI=</latexit>p(Y |X)

Parallel corpus•Translation in NMT-Find an output to maximize the probability

そこのどんつきまでガッと行ったら右やで

X=<latexit sha1_base64="2t6NzztG2ZHCrv2BMVpB+owQakU=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqBeh6MVjFfsBbSib7aZdutmE3YlQQv+BFw+KePUfefPfuG1z0NYHA4/3ZpiZFyRSGHTdb6ewtr6xuVXcLu3s7u0flA+PWiZONeNNFstYdwJquBSKN1Gg5J1EcxoFkreD8e3Mbz9xbUSsHnGScD+iQyVCwSha6aFz3S9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+6ZScWWVAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uwys/EypJkSu2WBSmkmBMZm+TgdCcoZxYQpkW9lbCRlRThjackg3BW355lbQuqp5b9e4vK/WbPI4inMApnIMHNajDHTSgCQxCeIZXeHPGzovz7nwsWgtOPnMMf+B8/gA3eI0j</latexit>

what is argmax p(Y|X=そこの...) ?

Go straight and turn rightY<latexit sha1_base64="T3f5yAtJWEWlpP50X1BS1VdMuC0=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ie0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHst7M0nQj+hQ8pAzaqzUeOiXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHp0X5935WLQWnHzmGP7A+fwBtz2M3Q==</latexit><latexit2

Autoregressive modeling•

Most of current application-level NMT models are based on
autoregressive modeling-

Breaks down p(Y|X) to word probabilities using chain rule
In each step, the model predicts the next wordp(my dog stole my cheese | mi perro robó mi queso)= p(my|mi perro robó mi queso)p(dog| my,mi perro robó mi queso)p(stole | my dog,mi perro robó mi queso)p(my| my dog stole,
mi perro robó mi queso)p(cheese| my dog stole my, mi perro robó mi queso)3

Obtain Translations•Approximating the global argmax with search algorithms-Greedy search and beam search<latexit sha1_base64="M48whkjq/zyemuEPBCvixCzsXcQ=">AAAB73icbVBNS8NAEJ31s9avqkcvi0Wol5KIoMeiF48VbBtoQ9lsN+3SzSbuboQQ+ye8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IBFcG8f5Riura+sbm6Wt8vbO7t5+5eCwreNUUdaisYiVFxDNBJesZbgRzEsUI1EgWCcY30z9ziNTmsfy3mQJ8yMylDzklBgreUkt67tP3lm/UnXqzgx4mbgFqUKBZr/y1RvENI2YNFQQrbuukxg/J8pwKtik3Es1SwgdkyHrWipJxLSfz+6d4FOrDHAYK1vS4Jn6eyInkdZZFNjOiJiRXvSm4n9eNzXhlZ9zmaSGSTpfFKYCmxhPn8cDrhg1IrOEUMXtrZiOiCLU2IjKNgR38eVl0j6vu07dvbuoNq6LOEpwDCdQAxcuoQG30IQWUBDwDK/whh7QC3pHH/PWFVTMHMEfoM8fND6PaA==</latexit>p(y1 |X)<latexit sha1_base64="+S5DbcDzIOCDYoWEwbmGAB53SPk=">AAAB83icbVBNSwMxEJ2tX7V+VT16CRahgpTdIuix6MVjBfsB7bJk07QNzWZDkhWWtX/DiwdFvPpnvPlvTNs9aOuDgcd7M8zMCyVn2rjut1NYW9/Y3Cpul3Z29/YPyodHbR0nitAWiXmsuiHWlDNBW4YZTrtSURyFnHbCye3M7zxSpVksHkwqqR/hkWBDRrCxUl9W06D+lAbeRfc8KFfcmjsHWiVeTiqQoxmUv/qDmCQRFYZwrHXPc6XxM6wMI5xOS/1EU4nJBI9oz1KBI6r9bH7zFJ1ZZYCGsbIlDJqrvycyHGmdRqHtjLAZ62VvJv7n9RIzvPYzJmRiqCCLRcOEIxOjWQBowBQlhqeWYKKYvRWRMVaYGBtTyYbgLb+8Str1mufWvPvLSuMmj6MIJ3AKVfDgChpwB01oAQEJz/AKb07ivDjvzseiteDkM8fwB87nD6tHkMY=</latexit>p(y2 |y1 , X) p(y3 |y1 y2 , X) p(y4 |y1 y2 y3 , X)<latexit sha1_base64="ZaWg5IlVkjHqyJSO6gscQ35qYIc=">AAAB+HicbVBNS8NAEJ34WetHox69LBahgpSkCnosevFYwX5AG8Jmu2mXbjZhdyPE2l/ixYMiXv0p3vw3btsctPXBwOO9GWbmBQlnSjvOt7Wyura+sVnYKm7v7O6V7P2DlopTSWiTxDyWnQArypmgTc00p51EUhwFnLaD0c3Ubz9QqVgs7nWWUC/CA8FCRrA2km+Xkkrmnz9lvpv5tbPOqW+XnaozA1ombk7KkKPh21+9fkzSiApNOFaq6zqJ9sZYakY4nRR7qaIJJiM8oF1DBY6o8sazwyfoxCh9FMbSlNBopv6eGONIqSwKTGeE9VAtelPxP6+b6vDKGzORpJoKMl8UphzpGE1TQH0mKdE8MwQTycytiAyxxESbrIomBHfx5WXSqlVdp+reXZTr13kcBTiCY6iAC5dQh1toQBMIpPAMr/BmPVov1rv1MW9dsfKZQ/gD6/MHNLeSIA==</latexit><latexit sha1_base64="yUwTdEzI/lIt1+XkS7EAv5/gGqM=">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBahgpSkFnRZdOOygn1AG8JkOmmHTh7MTMQQ+ytuXCji1h9x5984bbPQ1gMXDufcy733eDFnUlnWt1FYW9/Y3Cpul3Z29/YPzMNyR0aJILRNIh6Jnocl5SykbcUUp71YUBx4nHa9yc3M7z5QIVkU3qs0pk6ARyHzGcFKS65Zjqup23hKXTt166l7cd47c82KVbPmQKvEzkkFcrRc82swjEgS0FARjqXs21asnAwLxQin09IgkTTGZIJHtK9piAMqnWx++xSdamWI/EjoChWaq78nMhxImQae7gywGstlbyb+5/UT5V85GQvjRNGQLBb5CUcqQrMg0JAJShRPNcFEMH0rImMsMFE6rpIOwV5+eZV06jXbqtl3jUrzOo+jCMdwAlWw4RKacAstaAOBR3iGV3gzpsaL8W58LFoLRj5zBH9gfP4AS32TSg==</latexit>

argm

myax




NNaxargm

NNaxargm

mi perro robó mi queso

NNNN

dogstoleProblems-

Low parallelizability-

Worse if the model is bigger and deeper-

Require search algorithm to approximate the argmax
4Non-autoregressive Machine Translation•

Predict all output tokens in one
forward pass•

Can be fully parallelized on GPUAT

•

•

NAT

CMLM (conditional masked language model) [Ghazvinine et al., 2018]•

Predict the sequence and mask tokens withlow confidence•Perform such token refinement by multipleiterations

Drawback of token-based refinement models•Token prediction is time-consuming

CMLM Decoding5Latent-variable Non-autoregressive MT•

Core Idea:-•

my dog stole my cheese

Capture translation decisions with
continuous latent variables

Each source token is assigned with
one latent variable•

Each latent variable is a lowdimensional vector•Finding the best setting of latentvariables with high-speed inference

Decoder

8z1z2z3z4z5

Encoder / Inference

mi perro robó mi queso

6x

Objective function•<latexit sha1_base64="f2yzimwbR/Dgjzp6tZ360fHRqNI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit><latexitz

Similar to VAE, we train our model with ELBO
(evidence lower bound)<latexit sha1_base64="HDzXchlsPlmuEyZZ/9zFJ+iVC6I=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N0IN/QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB6UGM/g==</latexit><latexit

lyy<latexit sha1_base64="l29WxoUb9DEbvmhLG7jHtZ0OU24=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68diC/YA2lM120q7dbMLuRgihv8CLB0W8+pO8+W/ctjlo64OBx3szzMwLEsG1cd1vZ219Y3Nru7RT3t3bPzisHB23dZwqhi0Wi1h1A6pRcIktw43AbqKQRoHATjC5m/mdJ1Sax/LBZAn6ER1JHnJGjZWa2aBSdWvuHGSVeAWpQoHGoPLVH8YsjVAaJqjWPc9NjJ9TZTgTOC33U40JZRM6wp6lkkao/Xx+6JScW2VIwljZkobM1d8TOY20zqLAdkbUjPWyNxP/83qpCW/8nMskNSjZYlGYCmJiMvuaDLlCZkRmCWWK21sJG1NFmbHZlG0I3vLLq6R9WfPcmte8qtZvizhKcApncAEeXEMd7qEBLWCA8Ayv8OY8Oi/Ou/OxaF1zipkT+APn8wfnvYz9</latexit><latexit sha1_base64="Qbfk9kTFNBY8ERcAxGtI+JdPwPY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7EyGE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST25nfeeLaiFg9YpZwP6IjJULBKFrpQQ6yQbXm1t05yCrxClKDAs1B9as/jFkacYVMUmN6npugn1ONgkk+rfRTwxPKJnTEe5YqGnHj5/NTp+TMKkMSxtqWQjJXf0/kNDImiwLbGVEcm2VvJv7n9VIMr/1cqCRFrthiUZhKgjGZ/U2GQnOGMrOEMi3srYSNqaYMbToVG4K3/PIqaV/UPbfu3V/WGjdFHGU4gVM4Bw+uoAF30IQWMBjBM7zCmyOdF+fd+Vi0lpxi5hj+wPn8AWaIjdw=</latexit>

log p(Y |X)<latexit sha1_base64="je7hjeDlZoRCzZffv7n7WzvdYAo=">AAACsHicbVFri9QwFE3rax0fO+pHvwQHlxbr0KqoqwjLyILggrvg7HamKSXNZDpx04dJKgyd/jz/gN/8Nyadsj7WC4GTc8+59+YmrTiTyvd/WvaVq9eu39i5Obh1+87d3eG9+6eyrAWhU1LyUoQplpSzgk4VU5yGlaA4Tzk9S8/fm/zZNyokK4vPal3ROMdZwZaMYKWpZPgd8TKDlTPbhC7cQxmFKMdqJfLm8GjyyQm92VukVlRhD1Ur5qEypxl2W4jQYO8d3IrTtDlsk2YOkWQ5/JoYpVZMWBZB9EaLuhbJto7p5M09nszc35y+GdY1cuOL4dOLOT4etU5f1JlrlfZtNtrZTWIYF7rJcOSP/S7gZRD0YAT6OE6GP9CiJHVOC0U4ljIK/ErFDRaKEU7bAaolrTA5xxmNNCxwTmXcdOtu4WPNLOCyFPoUCnbsn44G51Ku81QrzRvkvzlD/i8X1Wr5Om5YUdWKFmTbaFlzqEpo/g4umKBE8bUGmAimZ4VkhQUmSv/woFvCvomXF0++DMJn4+D5eH8cnLwYHUz6heyAh+ARcEAAXoED8AEcgykg1hPrxJpbke3bp3Zi463UtnrPA/BX2F9+AeWyzbg=</latexit>

ELBO(X, Y; ✓, , !)hi= EZ⇠q log p✓ (Y |X, Z, lY )p✓ (lY |X, Z)
decoder•length predictor

KL(q (Z|X, Y )||p! (Z|X))
posterior

prior

Training:ˆ ˆ, !
✓,ˆ = argmax ELBO(X, Y ; ✓, , !)<latexit sha1_base64="KC7woEjkKGD9ZFp+1YwBQmw5PJc=">AAACa3icbVHRatRAFJ3EqnXVutoHpfowuBQqlCUpBQURSkXoQ8EKbruyWZab2buboTOZMHMjLiEvfqJv/kFf+g9OdlOprReGnDn3nJk7J2mhpKMo+h2Ed9bu3ru//qDz8NHjjSfdp89OnSmtwIEwythhCg6VzHFAkhQOC4ugU4Vn6fnHpn/2Ha2TJv9KiwLHGua5nEkB5KlJ92eSAfGEMiTY5atNkckraDTOgX/giQbKTFEtv1ZXYOcaftR1oqSW5CbV3xMa88pW8yv1p+PDz/XOcJd/e9/qrsneTLq9qB8ti98GcQt6rK2TSfdXMjWi1JiTUODcKI4KGvuZSAqFdScpHRYgzmGOIw9z0OjG1TKrmm97ZspnxvqVE1+y1x0VaOcWOvXKZnp3s9eQ/+uNSpq9G1cyL0rCXKwumpWKk+FN8HwqLQpSCw9AWOln5SIDC4L87+n4EOKbT74NTvf6cdSPv+z3Dg7bONbZS/aa7bCYvWUH7IidsAET7CLYCJ4HL4LLcDPcCl+tpGHQejbZPxVu/wHGKLpZ</latexit>

✓, ,!

7

Model architecture
decoder•

Latent NAT parameterizes fourdistributions•

Reuse Transformer modules•

Length Transform: adjust |x|vectors to |y| vectors(skip the details here)

length predictorpriorposterior8

Translation with Latent NAT•X

After model training we get:Prior•XZX
Z

Y

Decoder

XYPosterior

Z

Naive inference (decoding)Priorp(Z|X)
XZDecoderYmean of priorlatent space

9

Problem of naive inference•

Problem of naive inference-the center of a Gaussian may not produce the best results
犬が私の チーズが 犬に

私のチーズを盗んだ

犬が私の盗まれた

盗まれた

犬に

mean of prior

ワンちゃんが俺のチーズを盗んだ

latent space10Better inference approaches (1)•Delta inference (Shu et al., AAAI 2020)•Latent-variable updating for maximizing approximated ELBO•

ELBO is improved after iterations with rapid convergence

updated Z
mean of prior

XZDecoderXYPosteriorZ

latent space11

Better inference approaches (2)•Energy-based inference (Jason et al., EMNLP 2020)••

Build an energy model E(Z)High-quality latent vectors get low energy<latexit sha1_base64="8Gwrn49E/+Mpa25zjCzeEkE3qsA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBahXkKiovZWFMFjBWuLbSib7bZdursJuxuhhP4FLx70IF79P978NyZpELU+GHi8N8PMPD/kTBvH+bQKC4tLyyvF1dLa+sbmVnl7504HkSK0SQIeqLaPNeVM0qZhhtN2qCgWPqctf3yZ+q0HqjQL5K2ZhNQTeCjZgBFsUumqen/YK1cc28mA5ombkwrkaPTKH91+QCJBpSEca91xndB4MVaGEU6npW6kaYjJGA9pJ6ESC6q9OLt1ig4SpY8GgUpKGpSpPydiLLSeCD/pFNiM9F8vFf/zOpEZnHsxk2FkqCSzRYOIIxOg9HHUZ4oSwycJwUSx5FZERlhhYpJ4SlkItRSn3y/Pk/aR7R7bNdu9OanUL/JAirAH+1AFF86gDtfQgCYQGMEjPMOLJawn69V6m7UWrHxmF37Bev8CUUmONg==</latexit>•Update latent variables with theenergy gradientZt+1 = Zt<latexit sha1_base64="9UfWajid/xZmO7PONqBek5oTmSU=">AAACFXicbVDJSgNBEO1xjXGLevTSGAQXDDMqmhyEgAgeIxgNJmGo6XRMY0/P0F0jhCH/4MVf8eJBD+JV8Obf2FkQtwdFPd6rorteEEth0HU/nLHxicmp6cxMdnZufmExt7R8YaJEM15lkYx0LQDDpVC8igIlr8WaQxhIfhncHPf9y1uujYjUOXZj3gzhWom2YIBW8nNbV36K216PHtErH+kObYCMO0AbCgIJfmrFHj3ZsG3Tz+XdgjsA/Uu8EcmTESp+7r3RilgScoVMgjF1z42xmYJGwSTvZRuJ4TGwG7jmdUsVhNw008FNPbpulRZtR9qWQjpQv2+kEBrTDQM7GQJ2zG+vL/7n1RNsF5upUHGCXLHhQ+1EUoxoPyDaEpozlF1LgGlh/0pZBzQwtDFmByGU+jj4Ovkvqe0WvL1CqeCd7efLxVEgGbJK1sgG8cghKZNTUiFVwsgdeSBP5Nm5dx6dF+d1ODrmjHZWyA84b58PPZzE</latexit>

mean of priorenergy field↵rZt E(Zt )Energy-driven gradient descent12Better inference approaches (2)•Energy-based inference (Jason et al., EMNLP 2020)••

Build an energy model E(Z)High-quality latent vectors get low energy<latexit sha1_base64="8Gwrn49E/+Mpa25zjCzeEkE3qsA=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBahXkKiovZWFMFjBWuLbSib7bZdursJuxuhhP4FLx70IF79P978NyZpELU+GHi8N8PMPD/kTBvH+bQKC4tLyyvF1dLa+sbmVnl7504HkSK0SQIeqLaPNeVM0qZhhtN2qCgWPqctf3yZ+q0HqjQL5K2ZhNQTeCjZgBFsUumqen/YK1cc28mA5ombkwrkaPTKH91+QCJBpSEca91xndB4MVaGEU6npW6kaYjJGA9pJ6ESC6q9OLt1ig4SpY8GgUpKGpSpPydiLLSeCD/pFNiM9F8vFf/zOpEZnHsxk2FkqCSzRYOIIxOg9HHUZ4oSwycJwUSx5FZERlhhYpJ4SlkItRSn3y/Pk/aR7R7bNdu9OanUL/JAirAH+1AFF86gDtfQgCYQGMEjPMOLJawn69V6m7UWrHxmF37Bev8CUUmONg==</latexit>•Update latent variables with theenergy gradientZt+1 = Zt<latexit sha1_base64="9UfWajid/xZmO7PONqBek5oTmSU=">AAACFXicbVDJSgNBEO1xjXGLevTSGAQXDDMqmhyEgAgeIxgNJmGo6XRMY0/P0F0jhCH/4MVf8eJBD+JV8Obf2FkQtwdFPd6rorteEEth0HU/nLHxicmp6cxMdnZufmExt7R8YaJEM15lkYx0LQDDpVC8igIlr8WaQxhIfhncHPf9y1uujYjUOXZj3gzhWom2YIBW8nNbV36K216PHtErH+kObYCMO0AbCgIJfmrFHj3ZsG3Tz+XdgjsA/Uu8EcmTESp+7r3RilgScoVMgjF1z42xmYJGwSTvZRuJ4TGwG7jmdUsVhNw008FNPbpulRZtR9qWQjpQv2+kEBrTDQM7GQJ2zG+vL/7n1RNsF5upUHGCXLHhQ+1EUoxoPyDaEpozlF1LgGlh/0pZBzQwtDFmByGU+jj4Ovkvqe0WvL1CqeCd7efLxVEgGbJK1sgG8cghKZNTUiFVwsgdeSBP5Nm5dx6dF+d1ODrmjHZWyA84b58PPZzE</latexit>↵rZt E(Zt )updated Z
mean of priorenergy field

Energy-driven gradient descent13

Experiments

14

Experiment settings•

Dataset:WMT'14 English -> German- IWSLT'16 Romanian -> English- IWSLT'16 German -> English
Evaluation- Translation quality: BLEU
- Translation speed: averaged decoding time for one sentence-•15

Experiment results on machine translation
WMT'14 En-De

WMT'16 Ro -> En

IWSLT'16 De -> En

BLEU Speed
(4)

BLEU

Speed

31.5

1x

31.5

1x

BLEUSpeed

Transformer baseline, beam = 328.31x

Transformer baseline, beam = 1

27.5

1.1x

30.9

1.1x

31.1

1.1x

Latent NAT (Naive Inference)

25.7

15x

28.4

34x

27.0

19x+ Delta Inference

26.1

6.3x

29.0

19x

28.3

11x+ Energy Inference (w/ approximation)

26.3

10x

29.1

24x

28.8

13x+ Score Inference + Latent Search

27.4

6.2x

30.4

15x

30.2

6.3x

•

Latent Search: parallel decoding by sampling multiple latentvariable16

Translation speed related to computational capacity

17

Conclusion
XPriorZDecoderYFast Inference•We show a novel sequence generation framework-sequence prediction problem is solved by latent-variable
inference in the continuous space

Continuous setting and low dimensionality enable us toupdating efficiently• Fit for on-device computing•18

Thanks

19